Why is the accuracy reported in the Classification Learner app much higher than the accuracy of the exported model on the held out validation set?

3 次查看(过去 30 天)
Problem description
I used the default 5-fold cross-validation (CV) scheme in the Classification Learner app and trained all the available models. The best model (quadratic SVM) has 74.2% accuracy. I used
export model => generate code
and then ran the generated code, again examining the 5-fold CV accuracy. Surprisingly, the validation accuracy of this generated model was 66.8%, which is much lower than that reported by the app.
This has happened repeatedly - everytime I follow this scheme (with different datasets and models) I observe reduced accuracy for the exported model.
Potential explanations
I understand that not resetting the random number generation seed may lead to some variability between runs, but this effect seems too large and consistent to be explained by chance. I also understand that choosing the empirically-best model on the validation set, without having an additional test set, may lead to overestimation of its performance. If the exported model has a different CV partition, that could explain some of the reduction in accuracy, but I wonder if there might be some other explanation that I am missing.

回答(1 个)

Jalaj Gambhir
Jalaj Gambhir 2020-3-6
Hi,
You can have a look at the answer here. Hope it helps!
  1 个评论
asaf benjamin
asaf benjamin 2020-3-10
Hi,
Thanks for the response, but that answer deals with a very different issue, where one mistakingly compares cross-validation accuracy to accuracy on the training data. Under no circumstances would I try to test my model's performance on the data on which it trained. Rather, I am comparing
  1. The 5-fold cross-validation accuracy reported by the app, to
  2. The 5-fold cross-validation accuracy reported by the code generated by the app.
These should be equal (or at least very close), as I understand it, but they are in fact very different...

请先登录,再进行评论。

类别

Help CenterFile Exchange 中查找有关 Classification Learner App 的更多信息

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by