Hi Rudolf,
First of all, there are some bugs in your code:
y=sim(net,x);
mu=mean(y-t); %.^2
the default cost function is MSE (Mean Square Error), so the correct code should be:
y=sim(net,x); e=y-t;
mu=mean(e.^2); % or use mu=perform(net,y,t)
Mean of errors: 0.00053693 8.3758e-10 0.13737 0.33484 0.14557
Stdev of errors: 0.0018166 3.8184e-09 0.18903 0.44031 0.29932
Secondly, if your goal is to compare the performance of optimization solvers, use only training data to calculate MSE,
not the entire data set.
y=sim(net,xx); e=y-tt; % use training set to compute MSE
mu=mean(e.^2);
"trainlm works well, trainbr works very well, but trainbfg, traincgf and trainrp do not work at all. What is the reason for this? "
About this question, there are two main reasons:
Cost function
Training neural nets for surface fitting task is just solving a nonlinear least square problem, however, the cost functions posed by neural nets are extremely complex (non-convex and extremely large condition number), without curvature information (Hessian), it's hard for gradient descent (GD)-based method converges to the minimum, in theory, trainlm and trainbfg shold outperform other solvers. (other solvers are GD-Based methods, except for trainbr, I don't know its algorithm)
Optimization algorithm
trainlm is a "hybrid" method that combines Gauss-Newton (G-N) method and GD. GD-based methods are usaully more robust than Newton-type methods (but extremely slow in terms of iterations compare to Newton-type methods ). Although G-N / Quasi-Newton methods (BFGS) are much faster (quadratic / superlinear convergence), they are sensitive to initial point, if your initial is not "good enough" , you may get stuck by local min and saddle point, I guess that's the main reason why BFGS fail. In the early stage of LM, the algorithm usaully perform more like GD, so it kind of using GD to provide a better "initial point", than performing G-N to achieve fast convergence.
So if your network is not too large, for most cases using "trainlm" will be the best choice.
(However, LM is not feasible for larger neural nets, because it requires to solving a linear system at each iteration.)
So, is there exist some robust and fast method for trainning medium-size neural nets?
(medium-size networks are very common in scientific ML applications)
Some papers provide useful suggestions. that is, train the net using Stochastic Gradient Descent-based methods ( SGD) first, then use BFGS or L-BFGS solver, this will be faster than pure LM. In my experience, two-stage optimization usaully very robust and works quite well, check out the following function approximation example, it's much more complex than simplefit_dataset.
SSE: 0.14
MAE: 0.003
clear; clc; close all;
% To run this script, download the pack
% at file exchange: https://tinyurl.com/wre9r5uk
%% Generate data
n=50;
x=linspace(-2,2,n);
y=linspace(-2,2,n);
n1=numel(x); n2=n1;
count=0;
for j=1:n2
for k=1:n1
count=count+1;
data(1,count)=x(j);
data(2,count)=y(k);
u=x(j)^2+y(k)^2;
label(1,count)=(log(1+(x(k)-4/3)^2+3*(x(k)+y(j)-x(k)^3)^2));
label(2,count)=exp(-u/2).*cos(2*u);
end
end
%% Network Structure Set Up
InputDimension=2; OutputDimension=2;
LayerStruct=[InputDimension,10,10,10,OutputDimension];
NN.Cost='SSE';
NN.labelAutoScaling='on';
NN=Initialization(LayerStruct,NN);
%% First Order Solver Set Up
option.Solver='ADAM'; % ADAM is the state-of-the-art SGD-based solver.
option.s0=1e-3; % step size
option.MaxIteration=250;
option.BatchSize=1000;
NN=OptimizationSolver(data,label,NN,option);
%% Quasi-Newton Solver Set Up
option.Solver='BFGS';
option.MaxIteration=500;
NN=OptimizationSolver(data,label,NN,option);
%% Validation
Prediction=NN.Evaluate(data);
Error=label-Prediction;
Report=FittingReport(data,label,NN);
%% Visualization
figure
slice=2;
scatter3(data(1,:),data(2,:),label(slice,:),'black')
hold on
[X,Y]=meshgrid(x,y);
n1=numel(x); n2=numel(y);
surf(X,Y,reshape(Prediction(slice,:),n1,n2))
title('Neural Network Fit')
legend('data','Fitting')
