SVM: How is the classification error with leave-one-out cross validation calculated?

3 次查看（过去 30 天）
David Schubert
David Schubert 2015-7-20
评论： Ilya 2015-7-21
I am trying to understand what matlab's leave-one-out cross validation of an SVM is doing by comparing it to a leave-one-out cross validation written myself. Unfortunately, I do not get the same results.
First I create some random data
rng(0);
X = rand(10,20);
Y = [ones(5,1); zeros(5,1)];
n_samples = size(X,1);
Then I calculate the classification error with leave-one-out cross validation
CVSVMModel = fitcsvm(X, Y,...
'KernelFunction','rbf',...
'BoxConstraint',1,...
'LeaveOut', 'on',...
'CacheSize', 'maximal');
error1 = kfoldLoss(CVSVMModel, 'lossfun', 'classiferror')
Now I try to do the same by hand. For each iteration, one sample is taken out of the training set and predicted afterwards.
error2 = 0;
for fold = 1:n_samples
idx = [1:(fold-1), (fold+1):n_samples];
SVMModel = fitcsvm(X(idx,:), Y(idx),...
'KernelFunction','rbf',...
'BoxConstraint',1,...
'CacheSize', 'maximal');
label = predict(SVMModel, X(fold,:));
error2 = error2 + (label~=Y(fold));
end
error2 = error2/n_samples
However, I do not get the same results:
error1 =
0.7000
error2 =
1
Can anyone tell me why?
What also worries me: Why does the second method perfectly misclassify every point (error2=1)? This can't be a coincidence.

请先登录，再回答此问题。

采纳的回答

Ilya
Ilya 2015-7-20
fitcsvm passes class prior probabilities found from the entire data into each fold. Look at CVSVMModel.Trained{1}.Prior, CVSVMModel.Trained{2}.Prior etc - every time you should see [0.5 0.5]. When you cross-validate yourself, the priors are derived for each fold independently, and in each case you should have 5/9 for one class and 4/9 for the other. This explains the difference.
As to why the 2nd method errors 100% of the time, see my answer here. The short answer is - because the left-out label is always opposite to the majority class in the training set.
  3 个评论
显示 1更早的评论
David Schubert
David Schubert 2015-7-21
编辑：David Schubert 2015-7-21
Sorry, one more question: How do the prior probabilities affect the prediction? Here I just find that class(z)=sign(<w,z>+b), but I don't see how this takes the prior probabilities into account.
Ilya
Ilya 2015-7-21
Instead of using one box constraint for all observations, fitcsvm sets individual box constraints to C*N*wn, where C is what you pass, N is the total number of observations, and wn is the observation weight proportional to the class probability. In your case, each fold has 5 observations of one class weighted at 0.1 and 4 observations of the other class weighted at 0.125. Observations of the minority class can have larger alpha coefficients. This is why the model sometimes predicts into the minority class. Whereas in your manual cross-validation, all observations have equal weights and the model always predicts into the majority class.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Classification Trees 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by