How to assign my defined training set, validation set and test set for training a Neural net in NeuralNetwork toolbox and not relying on GUI functionalities ?

Question

Srikar 2016-4-27

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/281222-how-to-assign-my-defined-training-set-validation-set-and-test-set-for-training-a-neural-net-in-neur

评论： SayedPedram Hosseini 2020-5-4

To find an optimal number of hidden neurons and layers in my code using feedforward net, I use cross validation technique and cvpartition function to split data. Now my aim is to use this split data of train,Val and test for generating a good neural network, may I know how would it be possible to bypass the options of toolbox and force set our datasets. As I have already normalised the dataset. My code looks like the following

output = voltage_norm;
input = data_KC_ip_norm1;
numInst = length(output); %size of dataset
%******************************************************
%# Split training/testing
idx                     = randperm(numInst);
numTrain                = round(numInst*0.80); 
numTest                 = numInst - numTrain;
trainData               = input(idx(1:numTrain),:);  
testData                = input(idx(numTrain+1:end),:);
trainLabel              = output(idx(1:numTrain)); 
testLabel               = output(idx(numTrain+1:end));
%*********************************************************
param.hset = 3:12;     %Number of hidden neurons
%considered structure to include further parameters that help in getting optimized network      
%*********************************************************
cv = cvpartition(trainLabel,'kfold',5);
Rval = zeros(length(param.hset));
rmse_val = zeros(length(param.hset));
uint64 tStart_cvtrain;uint64 tElapsed_cv;
tStart_cvtrain = zeros(length(param.hset));
tElapsed_cv = zeros(length(param.hset));
tic;
for i = 1:cv.NumTestSets
    trIdx = cv.training(i);
    teIdx = cv.test(i);
    %learnData=trainData(trIdx,:);valData = trainData(teIdx,:);
    %learnLabel = trainLabel(trIdx,:);valLabel = trainLabel(teIdx,:);
  for k = 1:length(param.hset)
            tStart_cvtrain(j,k) = tic;
      param.h = param.hset(k);
            % build model on Learning data
              hiddenLayerSize = param.h; 
                net = feedforwardnet(hiddenLayerSize);
                net.trainParam.showWindow = false;
                net.trainParam.showCommandLine = true ;
                %trainInd,valInd,testInd
                net.divideParam.trainInd = trIdx;
                net.divideParam.valInd = teIdx;
                net.divideParam.testInd = idx(numTrain+1:end);
                  [net,tr] = train(net,input,output);
                  perf = perform(net,t,y);
                   % predict on the validation data
      y_hat = net(valData);
                Rval(j,k) = Rval(j,k) + mean((y_hat-valLabel).^2);
                  tElapsed_cv(j,k) = toc(uint64(tStart_cvtrain(j,k)));
      end
  end
  averageTime_CV = toc/cv.NumTestSets;
Rval = Rval ./ (cv.NumTestSets);
[v1, i1] = min(Rval);
optparam = param;
optparam.h = param.hset( i1);

Also i would like to know how to choose best option number of layers, and if also anyone could help me if there are any other parameters that could fetch me optimized results. Help of any kind shall be greatly appreciated.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Greg Heath 2016-4-27

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/281222-how-to-assign-my-defined-training-set-validation-set-and-test-set-for-training-a-neural-net-in-neur#answer_219603

在 MATLAB Online 中打开

I have devised an easy way to minimize the number of hidden nodes and layers (to prevent over-fitting) subject to additional constraints on overtraining and maximum training error.

However, instead of strict cross-validation, data-division as well as initial weights is random.

I have posted zillions of examples in the NEWSGROUP and on ANSWERS.

Search using

Ntrials

or

Ntrials Greg.

I have also posted on strict cross-validation. However, since it was only one or two posts, I will find the references.

Hope this will be of some help.

Greg

3 个评论
显示 1更早的评论隐藏 1更早的评论

Srikar 2016-4-28

Dear Greg,

Thanks for the reply, yes I have checked the tutorials, but I am not clear exactly the necessity of MSE biased and unbiased, could you lend some more explanation over this. Sorry I am quite new to this area, hence finding difficulty in understanding.

Thanks again Professor.

Greg Heath 2016-4-28

Whenever you use a data subset to create a function that will be used on similar, but different data, using that function on training data, will in general, yield an optimistically biased result of performance on nontraining data.

That is EXACTLY why validation and test subsets are used.

If you do not have validation and test subsets, the training data estimate is not reliable unless the number of equations is considerable larger than the number of unknown weights.

To mitigate the bias, the number of degrees of freedom, Ndof = Ntrn-Nw, replaces Ntrn, the number of input/target training examples when estimating the average performance.

https://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)

Hope this helps.

Greg

请先登录，再进行评论。

Answer 2

Greg Heath 2016-4-28

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/281222-how-to-assign-my-defined-training-set-validation-set-and-test-set-for-training-a-neural-net-in-neur#answer_219885

Whenever you use a data subset to create a function that will be used on similar, but different data, using that function on training data, will in general, yield an optimistically biased result of performance on nontraining data.

That is EXACTLY why validation and test subsets are used.

If you do not have validation and test subsets, the training data estimate is not reliable unless the number of equations is considerable larger than the number of unknown weights.

To mitigate the bias, the number of degrees of freedom, Ndof = Ntrn-Nw, replaces Ntrn, the number of input/target training examples when estimating the average performance.

https://en.wikipedia.org/wiki/Degrees_of_freedom_(statistics)

Hope this helps.

Greg

2 个评论
显示无隐藏无

Greg Heath 2016-4-28

Sorry about posting my above comment as an answer.

My computer gave me no choice !?

SayedPedram Hosseini 2020-5-4

Hi Greg,

I have the same problem as written, but Newsreader is no longer available and therefore the Links are not working. I'd be grateful if you could send me, if you have saved them or you have some similar tutorial materials.

Regards,

请先登录，再进行评论。

How to assign my defined training set, validation set and test set for training a Neural net in NeuralNetwork toolbox and not relying on GUI functionalities ?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

回答（2 个）

3 个评论
显示 1更早的评论隐藏 1更早的评论

2 个评论
显示无隐藏无

另请参阅

类别

标签

Community Treasure Hunt

How to assign my defined training set, validation set and test set for training a Neural net in NeuralNetwork toolbox and not relying on GUI functionalities ?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

回答（2 个）

3 个评论 显示 1更早的评论隐藏 1更早的评论

2 个评论 显示 无隐藏 无

另请参阅

类别

标签

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

3 个评论
显示 1更早的评论隐藏 1更早的评论

2 个评论
显示无隐藏无