Questions about the regularization (Modified Performance Function) of neural network

7 次查看（过去 30 天）
Wen Yui
Wen Yui 2018-8-21
评论： Wen Yui 2018-8-29
Hello, everyone. I tried to find out the best regularization ratio for a very simple problem from Matlab, using the function trainbgf for a shallow neural network. Then I plotted a validation curve. The problem is that the curve didn't make any sense. I just followed the contents from the official document as follows:
Here are my codes.
*******************************************
regularization_term = [0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1];
m = size(regularization_term,2);
[x,t] = simplefit_dataset;
x_train = x(1:70);
t_train = t(1:70);
x_test = x(71:94);
t_test = t(71:94);
trainPerformance = zeros(50,11);
testPerformance = zeros(50,11);
for j = 1:50
for i = 1:m
net = feedforwardnet(10,'trainbfg');
net.divideFcn = '';
net.trainParam.epochs = 300;
net.trainParam.goal = 1e-5;
net.performParam.regularization = regularization_term(i);
net = train(net,x_train,t_train);
y_train = net(x_train);
trainPerformance(j,i) = sqrt(perform(net,t_train,y_train));
y_test = net(x_test);
testPerformance(j,i) = sqrt(perform(net,t_test,y_test));
end
end
plot(regularization_term, mean(trainPerformance),regularization_term,mean(testPerformance))
legend('trainperformance-RMSE','testperformacne-RMSE','best')
xlabel('Regularization Ratio')
ylabel('RMSE')
************************************************
Here is the learning curve I plotted.
I think that the RMSE of the training data should increase as the regularization ratio increases and the RMSE of the test data should decrease at first and at a certain point start to increase as the regularization ratio increases. I'm not sure where I made a mistake, can anyone give me advice? Thank you in advance!
  2 个评论
Wen Yui
Wen Yui 2018-8-24
编辑：Wen Yui 2018-8-24
Thanks for your reply. Here is the link.
https://ww2.mathworks.cn/help/nnet/ug/improve-neural-network-generalization-and-avoid-overfitting.html
It is very wired that as the regularization ratio gets close to 1, both the training error and the test error decrease to almost 0. It don't make any sense for me. Thank you for your advice in advance!

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Greg Heath
Greg Heath 2018-8-24
Oh! … O.K.
The simplefit_dataset is smooth with 4 interior local extrema. Therefore, you probably only need H = 4 hidden nodes.
More than H = 4 hidden nodes can be considered overfitting. So, if you use the default H = 10, you will have an overfit net and should implement a mitigation to
PREVENT OVERTRAINING AN OVERFIT NET.
The most common mitigations are
1. DO NOT OVERFIT
A. No. of unknown weights <= No. of training equations:
Nw <= Ntrneq
AND/OR
B. Minimize weighted sum of SQUARED ERRORS AND SQUARED WEIGHTS
MSE + gamma * MSW
2. DO NOT OVERTRAIN:
Use a validation subset to implement EARLY STOPPING
Hope this helps.
Greg
  7 个评论
显示 5更早的评论
Greg Heath
Greg Heath 2018-8-27
I understood perfectly what you were doing.
My concern was/is for the less experienced readers.
Greg

请先登录，再进行评论。

更多回答（1 个）

Greg Heath
Greg Heath 2018-8-24
% When I search the command line window for info on "regularization" :
>> help regularization
regularization not found.
Use the Help browser search field to search the documentation, or type "help help" for help command options, such as help for methods.
>> doc regularization
% No answer!
>> lookfor regularization
plotbr - Plot network performance for Bayesian regularization training.
trainbr - Bayesian Regularization backpropagation.
msereg - Mean squared error with regularization performance function.
msnereg - Mean squared normalized error with regularization performance function.
lasso - Perform lasso regularization for linear regression on tall arrays.
lassoglm - Perform lasso or elastic net regularization for a generalized linear model.
lasso - Perform lasso or elastic net regularization for linear regression.
% Therefore, it seems that for general
%(i.e., nonlinear) applications
%
% USE TRAINBR WITH MSEREG
%
% I will let you do the rest. First see the results of the following commands
>> help trainbr
>> doc trainbr
Hope this helps.
THANK YOU FOR FORMALLY ACCEPTING MY ANSWER
Greg
  2 个评论
Wen Yui
Wen Yui 2018-8-24
Thanks for your reply sincerely! I know trainbr and used it much. However, in my present study, I want to plot a validation curve to see how the regularization term affects the training error and the test error. Actually, in my codes (net = feedforwardnet(10,'trainbfg')) you can see I used the function ''trainbfg'' whose cost function could be msereg. So I can use msereg to change the regularization term. The problem is if I used the regularization term 1 I can get the training error and the test error to 0. It is incorrect with no doubt. So I an wondering I may make a mistake in my codes or I misunderstand the meaning of the regularization term. You can find the contents attached here from the link I gave you in the former comment. I know you have answered many questions on this forum and admire your efforts. I hope you can continue to give some advice if you have spare time. Anyway, Thanks very much!
Wen Yui
Wen Yui 2018-8-24
I regard the function trainbr as a function which can automatically determine the regularization term for us. It would be very useful in many cases. But I want to figure out how the regularization term affects the results. So I think trainbr does not help in my present study.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Regression 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by