selectModels

Choose a subset of trained ECOC models composed of linear binary learners with various regularization strengths.

Load the NLP data set.

load nlpdata

X is a sparse matrix of predictor data, and Y is a categorical vector of class labels.

Create a set of 11 logarithmically spaced regularization strengths from $$10^{-8}$$ through $$10^{-1}$$.

Lambda = logspace(-8,-1,11);

Create a linear classification model template that specifies optimizing the objective function using SpaRSA. Use lasso penalties with the strengths specified in Lambda.

t = templateLinear('Solver','sparsa','Regularization','lasso',...
    'Lambda',Lambda);

Hold out 30% of the data for testing. Identify the test-sample indices.

rng(1); % For reproducibility
cvp = cvpartition(Y,'Holdout',0.30);
idxTest = test(cvp);

Train an ECOC model composed of linear classification models. For quicker execution time, orient the predictor data so that individual observations correspond to columns.

X = X';
PMdl = fitcecoc(X,Y,'Learners',t,'ObservationsIn','columns','CVPartition',cvp);
Mdl = PMdl.Trained{1};
numel(Mdl.BinaryLearners{1}.Lambda)

ans = 
11

Mdl is a CompactClassificationECOC model object. Because Lambda is an 11-dimensional vector of regularization strengths, you can think of Mdl as eleven trained models, each corresponding to a regularization strength.

Estimate the test-sample misclassification rates for each regularized model.

ce = loss(Mdl,X(:,idxTest),Y(idxTest),'ObservationsIn','columns');

Plot the misclassification rates with respect to regularization strength on the log scale.

figure
plot(log10(Lambda),log10(ce),'-o')
ylabel('log_{10} misclassification rates')
xlabel('log_{10} Lambda')
[~,minCEIdx] = min(ce);
minLambda = Lambda(minCEIdx);
hold on
plot(log10(minLambda),log10(ce(minCEIdx)),'ro');
hold off

Several values of Lambda yield similarly small classification error values. Consider choosing greater values of Lambda (that still yield good classification rates) because they lead to predictor variable sparsity.

Select the four models with regularization strengths that occur around the point at which the classification error starts increasing.

idx = 7:10;
MdlFinal = selectModels(Mdl,idx)

MdlFinal = 
  CompactClassificationECOC
      ResponseName: 'Y'
        ClassNames: [comm    dsp    ecoder    fixedpoint    hdlcoder    phased    physmod    simulink    stats    supportpkg    symbolic    vision    xpc]
    ScoreTransform: 'none'
    BinaryLearners: {78x1 cell}
      CodingMatrix: [13x78 double]

LambdaFinal = MdlFinal.BinaryLearners{1}.Lambda

LambdaFinal = 1×4

    0.0002    0.0008    0.0040    0.0200

MdlFinal is a CompactClassificationECOC model object. You can think of it as four models trained using the four regularization strengths in LambdaFinal.