Doubts about cross-validation

Question

Elena Casiraghi 2019-8-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/476564-doubts-about-cross-validation

评论： Elena Casiraghi 2019-8-22

I'm trying to test performance of a KNN and a svm (cecoc) classifier for a multiclass problem.

For the knn model I do it by building mdlknn, which is a cross validated model (leave one out) and then using kfoldLoss function

 mdlknn = fitcknn(X,labels, 'NumNeighbors', ...
                   k, 'Distance',@distKNN, 'Leaveout','on',...
                    'HyperparameterOptimizationOptions','UseParallel');
 perf  = 1-kfoldLoss(mdlknn);

Is this correct? My doubts regards the cross-bvalidation:

Using option ('Leaveout','on') in fitcknn is the same as not using it and then calling crossval on the trained model?

 mdlknnNoCrossVal = fitcknn(X,labels, 'NumNeighbors', ...
                   k, 'Distance',@distKNN, 'HyperparameterOptimizationOptions','UseParallel');
 mdlknn = crossval(mdlknnNoCrossVal, 'leaveout','on')
 perf  = 1-kfoldLoss(mdlknn);

Next, since in knn classifier I'm using my distance, I would like to do the same with the cecoc. How can I do that?

At this stage, the code I use for building the cecoc doesn't use it. how can I set it?

  t = templateSVM('Standardize',true,'KernelFunction','rbf');
  mdlsvm = fitcecoc(X,labels, 'Leaveout','on',...
                    'HyperparameterOptimizationOptions','UseParallel',...
                    'coding', 'ternarycomplete','Learners',t);
  perf = 1-kfoldLoss(mdlsvm);
           

And again: Using option ('Leaveout','on') in fitcecoc is the same as not using it and then calling crossval on the trained model, right?

THANKS A LOT

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Jyothis Gireesh 2019-8-22

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/476564-doubts-about-cross-validation#answer_388614

I am assuming that you want to have some clarification regarding the cross-validation syntax in KNN and SVM

KNN Classifier

mdlknn = fitcknn(X,labels, 'NumNeighbors', ... 
                   k, 'Distance',@distKNN, 'Leaveout','on',... 
                    'HyperparameterOptimizationOptions','UseParallel'); 
 
mdlknnNoCrossVal = fitcknn(X,labels, 'NumNeighbors', ... 
                   k, 'Distance',@distKNN, 'HyperparameterOptimizationOptions','UseParallel'); 
mdlknn = crossval(mdlknnNoCrossVal, 'leaveout','on') 

These two syntaxes may be equivalent for the creation of a cross-validated KNN model. “mdlknn" is a "ClassificationPartitionModel" classifier. "mdlknnNoCrossVal” creates a "ClassificationKNN" classifier which is cross validated using the “crossval” function.

ECOC classifier

In the case of ECOC classifier you may use the “templateKNN” as the learner. It provides the feature to customize the distance metric using the ‘Distance’ argument. The distance metric can be passed as a function handle like that of the KNN classifier syntax.

And the two syntaxes are valid for the process of cross validation in the case of ECOC also.