selectModels

To determine a good lasso-penalty strength for a linear classification model that uses a logistic regression learner, compare test-sample classification error rates.

Load the NLP data set. Preprocess the data as in Specify Custom Classification Loss.

load nlpdata
Ystats = Y == "stats";
X = X'; 

rng(10) % For reproducibility
Partition = cvpartition(Ystats,Holdout=0.30);
testIdx = test(Partition);
XTest = X(:,testIdx);
YTest = Ystats(testIdx);

Create a set of 11 logarithmically-spaced regularization strengths from $$10^{-6}$$ through $$10^{-0.5}$$.

Lambda = logspace(-6,-0.5,11);

Train binary, linear classification models that use each of the regularization strengths. Optimize the objective function using SpaRSA. Lower the tolerance on the gradient of the objective function to 1e-8.

CVMdl = fitclinear(X,Ystats,ObservationsIn="columns", ...
    CVPartition=Partition,Learner="logistic",Solver="sparsa", ...
    Regularization="lasso",Lambda=Lambda,GradientTolerance=1e-8)

CVMdl = 
  ClassificationPartitionedLinear
    CrossValidatedModel: 'Linear'
           ResponseName: 'Y'
        NumObservations: 31572
                  KFold: 1
              Partition: [1×1 cvpartition]
             ClassNames: [0 1]
         ScoreTransform: 'none'


  Properties, Methods

Extract the trained linear classification model.

Mdl = CVMdl.Trained{1}

Mdl = 
  ClassificationLinear
      ResponseName: 'Y'
        ClassNames: [0 1]
    ScoreTransform: 'logit'
              Beta: [34023×11 double]
              Bias: [-12.1623 -12.1623 -12.1623 -12.1623 -12.1623 -6.2503 -5.0651 -4.2165 -3.3990 -3.2452 -2.9783]
            Lambda: [1.0000e-06 3.5481e-06 1.2589e-05 4.4668e-05 1.5849e-04 5.6234e-04 0.0020 0.0071 0.0251 0.0891 0.3162]
           Learner: 'logistic'


  Properties, Methods

Mdl is a ClassificationLinear model object. Because Lambda is a sequence of regularization strengths, you can think of Mdl as 11 models, one for each regularization strength in Lambda.

Estimate the test-sample classification error.

ce = loss(Mdl,X(:,testIdx),Ystats(testIdx),ObservationsIn="columns");

Because there are 11 regularization strengths, ce is a 1-by-11 vector of classification error rates.

Higher values of Lambda lead to predictor variable sparsity, which is a good quality of a classifier. For each regularization strength, train a linear classification model using the entire data set and the same options as when you cross-validated the models. Determine the number of nonzero coefficients per model.

Mdl = fitclinear(X,Ystats,ObservationsIn="columns", ...
    Learner="logistic",Solver="sparsa",Regularization="lasso", ...
    Lambda=Lambda,GradientTolerance=1e-8);
numNZCoeff = sum(Mdl.Beta~=0);

In the same figure, plot the test-sample error rates and frequency of nonzero coefficients for each regularization strength. Plot all variables on the log scale.

figure
yyaxis left
plot(log10(Lambda),log10(ce),"-o")
ylabel("log_{10} classification error")

yyaxis right
plot(log10(Lambda),log10(numNZCoeff + 1),"-o")
ylabel("log_{10} nonzero-coefficient frequency")

xlabel("log_{10} Lambda")
title("Test-Sample Statistics")

Choose the index of the regularization strength that balances predictor variable sparsity and low classification error. In this case, a value between $$10^{-4}$$ to $$10^{-1}$$ should suffice.

idxFinal = 7;

Select the model from Mdl with the chosen regularization strength.

MdlFinal = selectModels(Mdl,idxFinal);

MdlFinal is a ClassificationLinear model containing one regularization strength. To estimate labels for new observations, pass MdlFinal and the new data to predict.