Why is the accuracy reported in the Classification Learner app much higher than the accuracy of the exported model on the held out validation set?

3 次查看（过去 30 天）
asaf benjamin
asaf benjamin 2020-3-3
编辑： asaf benjamin 2020-3-10
Problem description
I used the default 5-fold cross-validation (CV) scheme in the Classification Learner app and trained all the available models. The best model (quadratic SVM) has 74.2% accuracy. I used
export model => generate code
and then ran the generated code, again examining the 5-fold CV accuracy. Surprisingly, the validation accuracy of this generated model was 66.8%, which is much lower than that reported by the app.
This has happened repeatedly - everytime I follow this scheme (with different datasets and models) I observe reduced accuracy for the exported model.
Potential explanations
I understand that not resetting the random number generation seed may lead to some variability between runs, but this effect seems too large and consistent to be explained by chance. I also understand that choosing the empirically-best model on the validation set, without having an additional test set, may lead to overestimation of its performance. If the exported model has a different CV partition, that could explain some of the reduction in accuracy, but I wonder if there might be some other explanation that I am missing.

请先登录，再回答此问题。

回答（1 个）

Jalaj Gambhir
Jalaj Gambhir 2020-3-6
Hi,
You can have a look at the answer here. Hope it helps!
  1 个评论
asaf benjamin
asaf benjamin 2020-3-10
Hi,
Thanks for the response, but that answer deals with a very different issue, where one mistakingly compares cross-validation accuracy to accuracy on the training data. Under no circumstances would I try to test my model's performance on the data on which it trained. Rather, I am comparing
  1. The 5-fold cross-validation accuracy reported by the app, to
  2. The 5-fold cross-validation accuracy reported by the code generated by the app.
These should be equal (or at least very close), as I understand it, but they are in fact very different...

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Classification Learner App 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by