In evaluating a neural net, should NMSE be based only on test subset of data?

3 次查看（过去 30 天）
KAE
KAE 2019-5-15
评论： Greg Heath 2019-5-27
In answers like this, Greg Heath suggests using the normalized mean square error, NMSE, to compare the performance of different neural networks and pick the best one.
I have been calculating NMSE from all samples in the training set t and prediction y,
[net tr y e ] = train(net,x,t); % Train network
vart1 = var(t',1);
% MSE for a naive constant output model
% that always outputs average of target data
MSE00 = mean(vart1);
NMSE = mse(t-y)/MSE00; % Normalize
That includes the training samples, and so may favor models that fit the training data well but not new data. In order to choose the most robust model, should I calculate NMSE from the test samples only?
iTest = tr.testInd; % Index to the samples that were set aside for testing
NMSE_test_only = mse(t(:,iTest)-y(:,iTest))/MSE00; % Only use test samples

请先登录，再回答此问题。

采纳的回答

Greg Heath
Greg Heath 2019-5-19
For serious work I calulate FOUR values of NMSE:
1.70% Training
2.15% Validation
3.15% Test
4.100% All
for 10 (typically) random data divisions & initial weights and try to use as few hidden nodes as possible.
Hope this helps
Greg
  2 个评论
KAE
KAE 2019-5-20
编辑：KAE 2019-5-21
Once you have those 4 values of NMCE, do you pick the 'best' number of neurons (or whatever network feature you are optimizing) based on the net which has the highest test NMSE, averaged over the 10 trials?
Greg Heath
Greg Heath 2019-5-27
Typically, I try to minimize the number of hidden nodes subject to the constraint NMSEtrn <= 0.01 . I then rank those nets according to NMSEval and NMSEtst.
Details can be found in my NEWSGROUP and ANSWERS posts.
Greg

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Modeling and Prediction with NARX and Time-Delay Networks 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by