How does crossval (for k-fold CV) work in MATLAB after training a classifier?

15 次查看（过去 30 天）

Sanjay Yadav 2016-3-7

7
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/271843-how-does-crossval-for-k-fold-cv-work-in-matlab-after-training-a-classifier

评论： seung ho yeom 2019-2-1

To my knowledge, k-fold CV is a technique for model selection where the data is first divided into k-folds where the data in each fold is stratified. Now, consider the following code:

trainedClassifier = fitcnb(X, Y);
partitionedModel = crossval(trainedClassifier, 'KFold', 10);
accuracy = 1 - kfoldLoss(partitionedModel, 'LossFun', 'ClassifError');

The above code first trains the data in matrix X as per the class labels in vector Y. The trainedClassifier is then used in the function crossval(). My doubt is very simple. Does this line of code

partitionedModel = crossval(trainedClassifier, 'KFold', 10);

divide the matrix X into ten folds and then trains on 9 folds, testes on the remaining fold and this is repeated 10 times with each fold as test matrix or does it simply use the trainedClassifier that was trained in the previous line on the whole matrix X and then testes on each fold as I can only see that the fitcnb has been used only once. Does the function crossval() works upon it internally? If it doesn't, then the training is being done on the whole data instead of on the 9 folds in each iteration as is defined by cross-validation.

Fellow members of the community, I will be highly obliged if this doubt of mine can be cleared. Thanking you in anticipation.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Raghav G Raghav G 2018-10-22

I too have the same question. Did anyone find the answer?

Fulin Wei 2018-12-28

I have the same question. Do you have any answer now?

请先登录，再进行评论。

请先登录，再回答此问题。

回答（3 个）

Don Mathis 2018-11-30

3
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/271843-how-does-crossval-for-k-fold-cv-work-in-matlab-after-training-a-classifier#answer_349860

The answer is that it divides the dataset into 10 folds and trains the model 10 times on 9 folds each time, using the remaining fold as the test set. The only information taken from 'trainedClassifier' are the hyperparameter values, which are used in each of the 10 trainings. 'fitcnb' is not called 10 times, 'ClassificationNaiveBayes.fit' is.

11 个评论
显示 9更早的评论隐藏 9更早的评论

Don Mathis 2019-1-15

在 MATLAB Online 中打开

No no, hyperparameters are not obtained from the data at all. Hyperparameters are the settings that you pass to fitcnb, such as 'DistributionNames' or 'Kernel'.

I will try to explain what is happening in the originally posted code:

trainedClassifier = fitcnb(X, Y);

This line fits a single Naive Bayes model to the full dataset {X,Y} using default hyperparameters (settings), such as 'normal' distributions. The result is a ClassificationNaiveBayes object.

partitionedModel = crossval(trainedClassifier, 'KFold', 10);

This line splits the data {X,Y} into 10 folds. It then trains 10 models, each trained on 9 folds. For each of these models, it uses the settings from 'trainedClassifier', namely 'normal' distributions. The result is a ClassificationPartitionedModel, which contains the 10 models that were just trained.

accuracy = 1 - kfoldLoss(partitionedModel, 'LossFun', 'ClassifError');

This line runs each of the 10 models inside 'partitionedModel' on its own held-out test set, and computes the Loss for each model on its test set. These 10 individual loss values are added up to obtain the full 'kfold' loss. This loss is the classification error rate over the full dataset, because the 10 test folds make up the full dataset. This is the "out of sample" error rate on the full dataset. 'accuracy' is then 1 minus the error rate.

Finally, you would then use 'trainedClassifier' to make predictions on new data. 'accuracy' is now an estimate of the out-of-sample accuracy of 'trainedClassifier'. At this point, 'partitionedModel' can be discarded. Its only purpose was to provide an estimate of the out-of-sample accuracy of trainedClassifier. In fact, you cannot use 'partitionedModel' for prediction. It has no 'predict' method.