NN accuracy on test set low

2014 3 12
4 个回答
回答已采纳
更新时间：2014 3 20
11 次查看（30 天）

0 个投票

I have implemented a neural network in Matlab R2013a for character recognition. I have used trainbr function for nn training. 80% samples were used for training and the rest for testing. When i plot the confusion matrix, i get 100% accuracy on the training set. But for the test set the accuracy is very low(around 60%). What could be possibly wrong?

请先登录，再回答此问题。

更多回答（3 个）

Greg Heath
Greg Heath 2014-3-13

2 个投票

Insufficient info:
How many characters?
How many examples for each character?
What are the dimensions of the input and target matrices?
Are the summary statistics of the training and test subsets sufficiently similar?
How many input, hidden and output nodes?
What values of hidden nodes did you try ?
How many random weight initializations for each value ?
Although trainbr should mitigate the effect of using more hidden nodes than are needed, you still need many trials to establish sufficient confidence intervals.
Hope this helps.
Thank you for formally accepting my answer
Greg

6 个评论
显示 4更早的评论 隐藏 4更早的评论

Anitha
Anitha 2014-3-13
I am using 13 characters. There are a total of 234 characters in the database. 18 for each class.
The dimension of input matrix is 18 x 234.
The dimension of target matrix is 13 x 234
Number of input nodes=18 Number of hidden nodes=30( I have tried 10,15,20 also.But without crossvalidation got better result with 30 hidden nodes) Number of output nodes=13
Greg Heath
Greg Heath 2014-3-14
Sorry, that does not make sense to me. Consider the following
13 characters of the alphabet A-to-M
234 examples, 18 for each character
All characters are columnized 8x5 images
size(input) = [ 40 234]
size(target) = [ 13 234] % columns of eye(13)
Where did I go wrong?
Anitha
Anitha 2014-3-15
I have extracted 18 features for each of the characters. So
size(input)=[18 234] and since there are 13 characters,
size(target)=[13 234]
Greg Heath
Greg Heath 2014-3-15
[ I N ] = [ 18 234]
[ O N ] = [ 13 234]
The default trn/val/tst split for trainbr is 0.8/0.0/0.2. The resulting number of training equations is
Ntrn = N - round(0.2*N) % 187
Ntrneq = Ntrn*O % 2431
With H=30 hidden nodes, the number of unknown weights is
Nw = (I+1)*H+(H+1)*O % 973
The ratio is
r = Ntrneq/Nw % ~2.5
Which should be ok for trainbr.
I suggest making multiple designs (20?) in a loop with different mixes of training examples, testing examples and initial weights. For examples of multiple designs in a loop, search using
greg Ntrials
Post your code if you still have problems.
Anitha
Anitha 2014-3-16
编辑：Anitha 2014-3-16
I have done the code as you said with multiple designs.Now i am getting good results. After running 10 trials, i get best test set performance in trial 8. Now i want to train my final neural network with the parameters of this trial 8. But the problem is that, I have saved the rng state but when I use the same state for traing my net again, I get different results. This is my code.
clear all, close all, clc, plt=0;
tic
load inputdata
load targetdata
x=input;
t=target;
hiddenLayerSize = 30;
net = patternnet(hiddenLayerSize);
net.inputs{1}.processFcns = {'removeconstantrows','mapminmax'};
net.outputs{2}.processFcns = {'removeconstantrows','mapminmax'};
net.trainFcn='trainbr';
net.trainParam.epochs=100;
rng(4151941)
Ntrials=10
for i = 1:Ntrials
s{i} = rng;
[ net tr y e] = train(net,x,t);
mseval(i) = tr.best_tperf; % Best mseval over all epochs of ith run
tstind = tr.testInd;
ytst = y(:,tstind);
ttst = t(:,tstind);
plt=plt+1,figure(plt)
plotconfusion(ttst,ytst)
title([ ' TEST SET CONFUSION MATRIX. TRIAL = ', num2str(i)] )
hold off
end
[ minmseval ibest ] = min(mseval);
rng = s{ibest}; % For repeating the best design
bestnet = configure(net,x,t);
bestIW0 = bestnet.IW
bestb0 = bestnet.b
bestLW0 = bestnet.LW
[ bestnet tr y e] = train(net,x,t)
tstind = tr.testInd;
ytst = y(:,tstind);
ttst = t(:,tstind);
plt=plt+1,figure(plt)
plotconfusion(ttst,ytst)
title([ ' OVERALL TEST SET CONFUSION MATRIX= '] )
hold off
save net
view(bestnet)
Greg Heath
Greg Heath 2014-3-19
Two mistakes
1. No configure statement in the loop
2. Used net instead of bestnet in the last train statement

请先登录，再进行评论。

Greg Heath
Greg Heath 2014-3-16

1 个投票

1. Not necessary to specify default process functions.
2. How did you know my birthdate is 4151941 ??
3. You are reusing the same net for each trial without using CONFIGURE.
Therefore, the initial weights of each trial are the final weights of the last trial.
I suspect that if the design results are not monotonically better it is because TRAIN is
using a new trn/tst division.
4. Use configure after the RNG initialization.
5. An alternate approach is to CONTINUALLY save one or all of
a. the best current RNG state
a. the best current net
b. the best current Wb = getwb(net)
6. I think you should do all three at the same time and compare results
Hope this helps.
Thank you for formally accepting my answer
Greg

2 个评论
显示 无 隐藏 无

Anitha
Anitha 2014-3-17
Hi Greg,
I used an example from your previous post. That is why i set the rng as 4151941. I have modified the code as you said with configure after initialization. I got the best result in trial 4. But when i try to train the final network with these parameters, i still get different results. The weights and bias are all different as well. The confusion matrix is also different. What shall i do?
clear all, close all, clc, plt=0;
tic
load inputdata
load targetdata
x=input;
t=target;
hiddenLayerSize = 30;
net = patternnet(hiddenLayerSize);
net.trainFcn='trainbr';
net.trainParam.epochs=100;
rng(4151941);
Ntrials=15
for i = 1:Ntrials
s{i} = rng;
net = configure(net,x,t);
netIW0{i} = net.IW
netb0{i} = net.b
netLW0{i} = net.LW
[ net tr y e] = train(net,x,t);
% Best mseval over all epochs of ith run
tstind = tr.testInd;
ytst = y(:,tstind);
ttst = t(:,tstind);
%mseval(i) = mse(net,ttst,ytst)
mseval(i)=tr.best_tperf;
plt=plt+1,figure(plt)
plotconfusion(ttst,ytst)
title([ ' TEST SET CONFUSION MATRIX. TRIAL = ', num2str(i)] )
hold off
end
[ minmseval ibest ] = min(mseval);
rng=s{ibest}; % For repeating the best design
bestnet = configure(net,x,t);
bestIW0 = bestnet.IW
bestb0 = bestnet.b
bestLW0 = bestnet.LW
[ bestnet tr y e] = train(net,x,t);
msetst=tr.best_tperf;
tstind = tr.testInd;
ytst = y(:,tstind);
ttst = t(:,tstind);
fWb=getwb(bestnet);
%msetst= mse(bestnet,ttst,ytst)
plt=plt+1,figure(plt)
plotconfusion(ttst,ytst)
title([ ' OVERALL TEST SET CONFUSION MATRIX= '] )
hold off
save bestnet
view(bestnet)
Greg Heath
Greg Heath 2014-3-19
The second train statement contains net instead of bestnet

请先登录，再进行评论。

类别

在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

产品

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by