CNN - training, cross-validating and testing procedure -procedure and issues

Question

Venkat 2018-4-13

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/394553-cnn-training-cross-validating-and-testing-procedure-procedure-and-issues

评论： Venkat 2018-4-26

I know this is a fundamental question, but still I need to get a clear idea on this before I proceed further.

I have 8 subjects' data (images) with me: A,B,C,D,E,F,G,H. I have designed a CNN for a binary classification application. I train my CNN using 7 of the subjects (after balancing the minority class using oversampling technique). During training I use 80% of my data for training and 20% for cross-validation. I alter the hyper-parameters till I get a good cross-validation accuracy. Now that I am satisfied with my CNN's performance I test it with the left our subject.

Unfortunately, I get a low test-accuracy. After this what I did was, I went back to my CNN's hyper-parameters and modified my filter size and few other parameters and trained the CNN again with the same seven subjects till I got a good cross-validation accuracy and again tested it with the same left out subject as before. This time I got a good test accuracy.

I repeated this leave-one-out method(LOOM) training and testing process for the rest of the combination. fortunately for me, I got one set of hyper-parameters which worked for all LOOMs.

I was advised not to do this because it seems i am leaking testing information to CNN by adjusting my hyper-parameters based on my test results. Is my procedure faulty or is it right?

Kindly give your inputs.

Thanks for your time and help

--Venkat

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Bernhard Suhm 2018-4-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/394553-cnn-training-cross-validating-and-testing-procedure-procedure-and-issues#answer_316625

Yes, leaving-on-out is typically used to approximate performance on an unseen test set, but if you use it to optimize hyperparameters, your performance isn't really a test performance anymore. To rectify, you need to perform this tuning procedure using LOO (or some other method) on a held-out cross validation dataset.