Why is the accuracy reported in the Classification Learner app much higher than the accuracy of the exported model on the held out validation set?

Question

asaf benjamin 2020-3-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/508732-why-is-the-accuracy-reported-in-the-classification-learner-app-much-higher-than-the-accuracy-of-the

编辑： asaf benjamin 2020-3-10

Problem description

I used the default 5-fold cross-validation (CV) scheme in the Classification Learner app and trained all the available models. The best model (quadratic SVM) has 74.2% accuracy. I used

export model => generate code

and then ran the generated code, again examining the 5-fold CV accuracy. Surprisingly, the validation accuracy of this generated model was 66.8%, which is much lower than that reported by the app.

This has happened repeatedly - everytime I follow this scheme (with different datasets and models) I observe reduced accuracy for the exported model.

Potential explanations

I understand that not resetting the random number generation seed may lead to some variability between runs, but this effect seems too large and consistent to be explained by chance. I also understand that choosing the empirically-best model on the validation set, without having an additional test set, may lead to overestimation of its performance. If the exported model has a different CV partition, that could explain some of the reduction in accuracy, but I wonder if there might be some other explanation that I am missing.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jalaj Gambhir 2020-3-6

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/508732-why-is-the-accuracy-reported-in-the-classification-learner-app-much-higher-than-the-accuracy-of-the#answer_418816

Hi,

You can have a look at the answer here. Hope it helps!

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

asaf benjamin 2020-3-10

Hi,

Thanks for the response, but that answer deals with a very different issue, where one mistakingly compares cross-validation accuracy to accuracy on the training data. Under no circumstances would I try to test my model's performance on the data on which it trained. Rather, I am comparing