[Update: There is a known bug in kfoldLoss. See answer and workaround from Mathworks technical support in the answers section]
I couldn't work out why I was getting terrible results (as if completely random) with fitcdiscr() and I've found out that it is because I wasn't specifying the ClassNames argument in alphabetical order. Comparing MATLAB 2024a to 2022, this is new behaviour and presumably a bug. One of the reasons to specify ClassNames can be to change the order for the results summary, etc.
Example code that gives terrible accuracy:
Mdl = fitcdiscr(meas, species, "ClassNames", flip(unique(species)), "KFold", 10);
validationAccuracy = 1 - kfoldLoss(Mdl)
validationAccuracy = 0.3200
By simply removing the function flip(), the above code gives the expected accuracy of 0.98, otherwise it gives 0.32 (which is basically random for three classes). A side-effect of unique() is that it sorts the data into alphanumerical order, which isn't actually required for fisheriris because the observations happen to be in alphabetical order.
Here is some code that will give terrible accuracies if the order is randomised and the "stable" parameter is used to keep the random order:
r = randperm(length(meas));
Mdl = fitcdiscr(meas(r,:), species(r), "ClassNames", unique(species(r), "stable"), "KFold", 10);
validationAccuracy = 1 - kfoldLoss(Mdl)
validationAccuracy = 0.3267
If run a few times, I get results like:
validationAccuracy =
0.3267
validationAccuracy =
0.0067
validationAccuracy =
0.0067
validationAccuracy =
0.3267
validationAccuracy =
0.9800
Update: I have now been able to test the code on another computer that still has MATLAB 2022 installed and it gives the correct accuracy with the above code, so this appears to be a bug in the latest version of MATLAB! I have reported it to Mathworks.
