Classification Experiment Design - stopping training
显示 更早的评论
Hi,
I have a question (or several) regarding experiment design using matlab to conduct binary classification. I have a "design" set (for training and validation) and a separate test set to evaluate generalisation. My problem is when to stop training and apply the resulting net to my test set. From the nnet faq "Statisticians tend to be skeptical of stopped training because it appears to be statistically inefficient due to the use of the split-sample technique". So I use cross-validation by the following general method:
- for a given node value (H) I create a 100 random starting weight sets (S)
- for each (S) I randomly divide the design set into k equally sized, mutually exclusive subsets and train K nets using K(i) as the validation set and K-K(i) as the training set.
- Each net is trained to stop at mse_goal = 1e-6
- I evaluate validation error for each K(i) and retrain the relevant net to the number of epochs where validation error was lowest ?
- (do I need to do this or can I somehow select/return the net with the weights trained to this epoch from the [net tr] output) ?????
- I apply this net to my test set to evaluate generalisation
- the net from the set of S*K gives with the lowest generalisation error gives me the best trained net for my given H using my available data
Does this make sense ??
1 个评论
Greg Heath
2013-4-20
You failed to give 4 important values
1. N-size of data set
2. I-input dimensionality
3. O-output dimensionality
4. MSE00-mean target variance mean(var(target'))
采纳的回答
Greg Heath
2013-4-20
1. Initialize the RNG before the H loop and record the current RNG seed at the beginning of each inner loop. You can retrain any individual net by knowing the corresponding values of H, ntrial and SEED.
2. Train with k-2 nets Using MSEtrngoal = max(0,0.01*Ndof*MSE00a/Ntrneq).
3. Record MSEtrn and MSEtrna for the combined training set and BOTH MSEs of the nontraining subsets at the MSE minima of EACH nontraining set:
a. MSEk-1 at the minimum of MSEk is an unbiased estimate of generalization
error.
b. So is MSEk at the minimum of MSEk-1.
c. So is their average MSEtst = ( MSEk + MSEk-1)/2
4. Calculate the summary stats (min,median,mean,stdv,max)of R2trn, R2trna, R2tst and plot as a function of H.
Hope this helps.
Thank you for formally accepting my answer
Greg
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
