How do I improve the performance of a closed loop NARNET network?

3 次查看（过去 30 天）
Gary
Gary 2016-8-15
评论： Greg Heath 2017-9-7
I want to predict the next 10 steps for a time series. The single time series (T), has multiple values per time period.
The design criteria, are 1. Forecast accuracy, (next prediction point is more accurate than the 10th prediction point) 2. NN Execution speed. (Training speed is a secondary).
I used the 'Workflow for Neural Network Design' as a template, and the 'Narnet Tutorial On Multi Step Predictions' as a working model.
The code is below. (I specify some default parameters, so that I know what to set them back to when testing different network configurations).
The only relevant (i.e. multi value) dataset I could find for testing is 'oil_dataset'.
The code & results are below:
%%1. Collect & Prepare the data
clear all; clc; plt = 0;
PredictSteps = 10;
T = oil_dataset;
N = length(T);
%%2. Create the Neural Network
FD = 2; % Optimize this later with nncorr
H = 10; % Optimize this later with trial/error loops
feedbackDelays = 1:FD; % Default = 1:2
hiddenLayerSize = H; % Default = 10
trainFcn = 'trainbr'; % Default = 'trainlm'
net = narnet(feedbackDelays,hiddenLayerSize,'open',trainFcn);
%%3. Configure the network
% Choose Feedback Pre/Post-Processing Functions
net.input.processFcns = {'removeconstantrows','mapminmax'};
% Setup dynamic weights & biases
net.adaptFcn = 'adaptwb'; % Default = 'adaptwb'
% Setup Division of Data for Training, Validation, Testing
net.divideFcn = 'divideblock'; % Divide data into blocks
net.divideMode = 'time'; % Divide up every value
net.divideParam.trainRatio = 70/100;
net.divideParam.valRatio = 15/100;
net.divideParam.testRatio = 15/100;
net.trainParam.epochs = 1000; % Default = 1,000
% Choose a Performance Function
net.performFcn = 'mse'; % Mean squared error
% For MSE, Normalize errors of multiple outputs in different ranges
net.performParam.normalization = 'standard';
% Choose Plot Functions
net.plotFcns = {'plotperform','plottrainstate','plotresponse', ...
'ploterrcorr','plotinerrcorr','plotwb','ploterrhist','plotfit', ...
'plotregression',};
%%4. Initialize weights & biases
rng('default');
% In this case set 'weighted Error Weights'
ind = 1:N;
ew = 0.99.^(N-ind);
% Plot the Error Weights
plt = plt + 1;
figure(plt), hold on
plot(ew)
title('Error Weights');
ew = con2seq(ew);
%%5.a) Train the network
[Xo,Xio,Aio,To,EWo] = preparets(net,{},{},T, ew);
% Plot original data
to = cell2mat(To);
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, to)
legend('Upper', 'Lower')
title('INITIAL TRAINING DATA');
% Train the NN
[net,tr,Yo,Eo,Afo,Xfo] = train(net,Xo,To,Xio,Aio,EWo);
%%5.b) Plot the OPENLOOP traing results
yo = cell2mat(Yo);
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, to, 'b')
plot(FD+1:N, yo, 'r--')
legend('TARGET', 'OUTPUT') % Fix the legend colors, need to make legend for each line
title('OPENLOOP NARNET RESULTS');
%%5.c) Validate the trained 'Open' Network
[Yo, Xfo, Afo] = net(Xo,Xio,Aio);
Eo = gsubtract(To,Yo);
OpenLoopMSEperformance = mse(net, To, Yo); % 728.0917
OpenLoopPERFORMperformance = perform(net,To,Yo, EWo); % 0.0063
% NMSEopen = mse((Eo)/var(To,1)); % Error: Undefined function 'sum' for input arguments of type 'cell'
view(net);
%%5.d) Close the Network
[netc, Xic,Aic] = closeloop(net,Xio,Aio);
netc.name = [net.name ' - Closed Loop'];
view(netc);
[Xc,Xic,Aic,Tc,EWc] = preparets(netc,{},{},T,ew); % should we use EWo instead of ew?
[Yc,Xfc,Afc] = netc(Xc,Xic,Aic);
%%5.e) Plot the CLOSELOOP results
tc = cell2mat(Tc); % tc isequal to
yc = cell2mat(Yc);
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, tc, 'b')
plot(FD+1:N, yc, 'r--')
legend('TARGET', 'OUTPUT') % Fix the legend colors
title('CLOSELOOP NARNET RESULTS')
%%5.f) Validate the closed network
Ec = gsubtract(Tc,Yc);
CloseLoopMSEperformance = mse(netc, Tc, Yc); % 9.8945e+04
CloseLoopPERFORMperformance = perform(netc,Tc,Yc,EWc); % 0.5823
% NMSEclosed = mse((Ec)/var(Tc,1)); % Error: Undefined function 'sum' for input arguments of type 'cell'
view(netc);
%%5.g) ReTrain the 'Closed' network for better accuracy
% 'preparets' performed in section 5.d)
[netc2, trc, Yc2, Ec2, Xfc2, Afc2] = train(netc, Xc, Tc, Xic, Aic, EWc);
[Yc3,Xfc3, Afc3] = netc2(Xc,Xfc2,Afc2);
%%5.h) Plot the retrained CLOSELOOP results
yc3 = cell2mat(Yc3);
% tc = to % tc isequal to
plt = plt + 1;
figure(plt), hold on
plot(FD+1:N, tc, 'b')
plot(FD+1:N, yc3, 'g--')
legend('TARGET', 'OUTPUT') % Fix the legend colors
title('TRAINED CLOSELOOP NARNET RESULTS');
%%6. Validate the trained 'Closed' Network
Ec2 = gsubtract(Tc,Yc3);
TrainedClosedLoopMSEperformance = mse(netc2, Tc, Yc3); % 6.3859e+04
TrainedClosedLoopPERFORMperformance = perform(netc2,Tc,Yc3,EWc); % 0.2147
% NMSEs = mse((Ec2)/var(Tc,1)); % Error: Undefined function 'sum' for input arguments of type 'cell'
view(netc2);
%%7.a) Use the network (for multi-step prediction)
Xc3 = cell(1, PredictSteps); % Empty cells for prediction
[Yc4,Xfc4,Afc4] = netc2(Xc3,Xfc3,Afc3);
%%7.b) Plot the predicted results
yc4 = cell2mat(Yc4);
plt = plt+1; figure(plt), hold on
plot(FD+1:N, tc)
plot(FD+1:N, yc3)
plot(N+1:N+PredictSteps, yc4, '+')
legend('TARGET', 'OUTPUT' , 'TARGETLESS PREDICTION')
title('NARNET PREDICTION RESULTS');
%%Save the NN and the final input and layer delay states, Xfc and Afc
Xis = Xfc4;
Ais = Afc4;
save('NARNETfcn.mat', 'Xis', 'Ais'); % Ready for use next time
As you can see, once the narnet is closed, the performance is bizarre.
I ran the 'Narnet Tutorial On Multistep Ahead Predictions' on oil_dataset and produced similar results.
1. What am I doing wrong?
2. How do I correct the error calculating NMSEs?
Thank you in anticipation.
  2 个评论
Greg Heath
Greg Heath 2016-8-16
1. Why did you use TRAINBR instead of the default TRAINLM?
2. Cannot have valratio = 0.15 with TRAINBR. Must use valratio = 0!
3. Why mse instead of msereg??? Combining TRAINBR with MSE is unusual.
4. Why need EW? Have you compare w and w/o?
5. Xfo, Afo erroneously in reverse order in training equation
6. What is the performance function? perform(neto,To,Yo, EWo). Similarly for netc
7. NMSEopen = mse(Eo)/mean(var(to',1)) . Similarly for NMSEclosed
Hope this helps.
Greg
Gary
Gary 2016-8-16
编辑：Gary 2016-8-16
Greg,
Thank you for your prompt response, yes it helps enormously. If I may, I would like to answer your questions in a slightly different order, for reasons which I hope will become clear:
1. I used TRAINBR to improve NN generalization and avoid overfitting. (As per the documentation, but I haven't mastered the 'early stopping' yet though!)
3. Originally I considered 'msereg' but 'help msereg' stated "Obsoleted in R2010b NNET7.0. Last used in 2010a NNET6.0.4.". I have since updated "net.performFcn = msereg".
2. I removed all the NET.performParam's. (As per 'help msereg' - Set NET.performFcn to 'msereg'. This will automatically set NET.performParam to msereg's default performance parameters.)
4. I used Error Weights for prediction accuracy, i.e. because I wanted the 'predictions' to reflect the more recent history, while remaining generalized without overfitting. At this point I have not tested without them, because I still have to optimize the feedbackDelays & hiddenLayerSize, so this was part of my 'tuning' plan.
5. Thank you for identifying that error.
7. Thank you for that clarification
6. Here are the results:
MSEo = 728.0917 MSEREGo = 655.5660 PERFORMo = 0.0063 NMSEo = 0.0231
MSEc = 9.9845e+04 MSEREGc = 8.9051e+04 PERFORMc = 0.5823 NMSEc = 3.1392
MSEr = 6.3859e+04 MSEREGr = 5.4462e+03 PERFORMr = 0.2147 NMSEr = 2.0260 (r = retrained)
The current performance levels are just not usable. What do you recommend as the way forward? Should I work on the feedbackDelay and hiddenLayerSize parameters to improve the NN performance, or am I missing something else?
Thanks and regards Gary

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Greg Heath
Greg Heath 2016-8-17
编辑：Greg Heath 2017-8-31
Greg,
1. I used TRAINBR to improve NN generalization and avoid overfitting. (As per the documentation, but I haven't mastered the 'early stopping' yet though!)
GEH1 = It is better to train with defaults first.
Early Stopping is a default. Training stops when
the error on the validation set increases for
6(default) consecutive epochs.
Simple as that. Dealing with TRAINBR is somewhat complicated because of various idiosyncracies of the MATLAB code.
3. Originally I considered 'msereg' but 'help msereg' stated "Obsoleted in R2010b NNET7.0. Last used in 2010a NNET6.0.4.". I have since updated "net.performFcn = msereg".
GEH2 = msereg is a TRAINBR default. Therefore you
needed to do nothing.
2. I removed all the NET.performParam's. (As per 'help msereg' - Set NET.performFcn to 'msereg'. This will automatically set NET.performParam to msereg's default performance parameters.)
GEH3 = Not clear. However probably
irrelevant because of GEH2.
4. I used Error Weights for prediction accuracy, i.e. because I wanted the 'predictions' to reflect the more recent history, while remaining generalized without overfitting. At this point I have not tested without them, because I still have to optimize the feedbackDelays & hiddenLayerSize, so this was part of my 'tuning' plan.
GEH4: It is best to begin with all defaults.
Then make "improvements" one at a time.
5. Thank you for identifying that error.
7. Thank you for that clarification
6. Here are the results:
MSEo = 728.0917 MSEREGo = 655.5660 PERFORMo = 0.0063 NMSEo = 0.0231
MSEc = 9.9845e+04 MSEREGc = 8.9051e+04 PERFORMc = 0.5823 NMSEc = 3.1392
MSEr = 6.3859e+04 MSEREGr = 5.4462e+03 PERFORMr = 0.2147 NMSEr = 2.0260 (r = retrained)
The current performance levels are just not usable. What do you recommend as the way forward? Should I work on the feedbackDelay and hiddenLayerSize parameters to improve the NN performance, or am I missing something else?
Thanks and regards Gary
I can only recommend what I do.
1. use all defaults; use an outer for loop to vary
H and an inner loop to vary random
initial weights
and random data-divisions
2. use DIVIDEBLOCK; repeat double loop search
3. Obtain significant lags
4. Choose a sensible subset of siglags then try
to minimize H subject to the constraint
NMSEo <= 0.005 (or 0.001)
using the doule loop search
5. It is often difficult to obtain a low NMSEc with closeloop if NMSEo is any higher.
6. If none of the CL nets is successful, try training them further otherwise go back and lower the NMSEo goal.
Hope this helps.
Thank you for formally accepting my answer
Greg
  11 个评论
显示 9更早的评论
Gary
Gary 2017-9-7
Greg,
Many thanks for the update. I'm pretty comfortable with the NN training, (thanks to your posted tutorials, comments etc.) what I'm looking for guidance on, is the methodology for using autocorrelation and cross-correlation to support the FD = 1:35
While there is a -ve correlation peak at lag 98, I still can't reconcile that with your earlier calculations for FD = 1:35
I was looking for +ve correlations because I understood a -ve correlation to be an "inverse" relationship between the two data series, i.e. the opposite to what I'm looking for.
Would you suggest I look at "smoothing" the data to reduce the "noise", before looking for correlation?
Thank you in advance
Gary
Greg Heath
Greg Heath 2017-9-7
Look for significant ABSOLUTE values.
1. Negative correlations are just as effective as positive correlations. The only difference is that the sign of some of the NN weights will be reversed.
2. Significant absolute values are all absolute values above a specified threshold. Not just the peak values.
3. I consider signal correlation coefficients significant if their absolute values are not exceeded by the absolute values of noise/noise correlation coefficients 95% of the time.
Hope this helps.
Greg

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Modeling and Prediction with NARX and Time-Delay Networks 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by