Is there a way to create a custom cvpartition?

Question

Christian 2021-6-14

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/855560-is-there-a-way-to-create-a-custom-cvpartition

回答： Giovanni 2024-1-9

Hi,

I'd like you use crossvalidation in a model with a custom set of training/test sample vectors.

To use some of the built-in functionalities of Matlab I'd like to pass the input via the CVPartition Name/Value input parameter. Is there a way to define a custom set of train/test indices using cvpartition?

Thanks,

Christian

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Shraddha Jain 2021-6-25

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/855560-is-there-a-way-to-create-a-custom-cvpartition#answer_733120

Hi Christian,

As of now, the functionality to use custom indices in the cvpartition object is not available in the Statistics and Machine Learning toolbox. It might be considered in a future release.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Giovanni Attolico 2021-10-13

It seems to me that the indices are stored inside the cvpartition variable but they are read-only using the functions "training" and "test". While waiting for the "future release", it is not possible to define two functions "settraining" and "settest" that write indices in the variable? That would be really useful in many situation to allow the use of the resulting variable in all the situations where the current random partition can be used ...

Thanks

Christian 2021-10-13

I had the same thoughts on this. However, I stopped trying to alter the code in cvpartition after a few attempts and ended up in writing a complete custom approach for crossvalidation procedures. In addition to custom indices you're also more flexible with the used metrics and and and...

请先登录，再进行评论。

Answer 2

Giovanni 2024-1-9

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/855560-is-there-a-way-to-create-a-custom-cvpartition#answer_1386061

Custom partition has been recently introduced in MATLAB r2023B, but I suspect it does not work as expected since cross validation results from 2 neural netwroks differ even if trained and validated with the same folds (cv partition using the custom partition options with indices generated by crossvalind) and the networks are trained with the same parameters.

clear variables
close all
clc
tab = readtable("three_selected_feature.xlsx"); %features dataset
featuresTab = tab(:,2:end-1);
feature = table2array(tab(:,2:end-1));
y = table2array(tab(:,end));
for i = 1:6 %repeat for different training test splitting.
cTest = cvpartition(y,'HoldOut',0.2,'Stratify',true);
train = training(cTest);
y_train = y(train);
testing = test(cTest);
cvIndices = crossvalind('Kfold',size(y_train,1),5);
cv = cvpartition("CustomPartition",cvIndices);
% cv = cvpartition(y(train),"Resubstitution");
features_scaled = normalize(feature,'zscore');
modelNET1 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
modelNET2 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
%models trained with the same parameters and features and crossvalidated
%using custom partitions. cv is based on cvIndices generated by the
%crossvalind function. For the same "i" (for loop) results of the two
%models should be the same, but they differ.
% cvmodelLDA1 = crossval(modelLDA1,'CVPartition',cv);
lossNET1(i) = kfoldLoss(modelNET1);
accNET1(i) = 1 - lossNET1(i);
% cvmodelLDA2= crossval(modelLDA2,'CVPartition',cv);
lossNET2(i) = kfoldLoss(modelNET2);
accNET2(i) = 1 - lossNET2(i);
end