verifyTreeRobustness

R2026b

Verify robustness of Statistics and Machine Learning Toolbox tree classifiers

Since R2026b

collapse all in page

Syntax

results = verifyTreeRobustness(Mdl,TblLower,TblUpper,responseVarName)

results = verifyTreeRobustness(Mdl,TblLower,TblUpper,labels)

results = verifyTreeRobustness(Mdl,XLower,XUpper,labels)

Description

Add-On Required: This feature requires the AI Verification Library for Deep Learning Toolbox add-on.

results = verifyTreeRobustness(Mdl,TblLower,TblUpper,responseVarName) returns verification results across the regions defined by the lower bounds in the TblLower table and the upper bounds in the TblUpper table.

For each region, the function verifies whether the expected class label in the responseVarName variable matches the predicted class label returned by the classification tree Mdl. That is, the function verifies whether Mdl is robust with respect to class label i when the input is between TblLower(i,:) and TblUpper(i,:). For more information, see Tree Robustness.

The verifyTreeRobustness function requires Statistics and Machine Learning Toolbox™.

example

results = verifyTreeRobustness(Mdl,TblLower,TblUpper,labels) returns verification results using the class labels in labels.

results = verifyTreeRobustness(Mdl,XLower,XUpper,labels) returns verification results using the numeric lower and upper bounds XLower and XUpper, respectively.

example

Examples

collapse all

Verify Tree Robustness

This example uses:

Open Live Script

Verify the robustness of a classification tree.

Load the fisheriris data set. The meas matrix contains iris measurements, and the species variable contains the species type for each iris.

load fisheriris

Partition the iris data into training and test sets. Use approximately 75% of the observations for training a classification tree model, and reserve the remaining observations for testing. Use stratified partitioning so that approximately the same proportions of iris species exist in both the training and test sets.

rng(0,"twister")
partition = cvpartition(species,Holdout=0.25);

trainingX = meas(training(partition),:);
trainingY = species(training(partition));

testX = meas(test(partition),:);
testY = species(test(partition));

Train a classification tree using trainingX as the predictor data and trainingY as the response variable (class labels). Predict the species for the irises with predictor data testX.

Mdl = fitctree(trainingX,trainingY);
predictedY = predict(Mdl,testX);

Perturb the test set observations. For each predictor, compute 1% of the interquartile range. For each observation, create a lower bound by subtracting the perturbation value, and create an upper bound by adding the perturbation value.

perturbation = 0.01*iqr(testX)

perturbation = 1×4

    0.0110    0.0063    0.0340    0.0153

XLower = testX - perturbation;
XUpper = testX + perturbation;

Verify the stability of Mdl for each test set observation. That is, for each observation, check whether the classification tree predicts the same label for all observations with predictor values in the region between the lower and upper bounds. Summarize the results.

results = verifyTreeRobustness(Mdl,XLower,XUpper,predictedY)

results = 37×1 categorical
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
     verified 
      ⋮

summary(results)

results: 37×1 categorical

     verified         37 
     violated          0 
     unproven          0 
     <undefined>       0

For each test set observation, Mdl is stable in the region between the lower and upper bounds.

Note that stability does not imply correctness. For example, find the test set observations where the model predicts a class label other than the true class.

misclassifiedIdx = find(~strcmp(testY,predictedY))

misclassifiedIdx = 
21

trueLabel = testY(misclassifiedIdx)

trueLabel = 1×1 cell array
    {'versicolor'}

predictedLabel = predictedY(misclassifiedIdx)

predictedLabel = 1×1 cell array
    {'virginica'}

Mdl incorrectly predicts the 21st iris in the test set as a virginica iris when its true species is versicolor. The robustness verification results imply that the model consistently classifies observations with predictor values in the region between XLower(21,:) and XUpper(21,:) as virginica.

For all other test set observations, Mdl is robust to perturbations in the specified regions because the model is both stable and correct.

Verify Tree Robustness for Increasing Perturbations

This example uses:

Open Live Script

Increase the perturbation region on which you can verify the robustness of a classification tree.

Load the carbig data set, which contains car measurements. First, convert Origin to a categorical variable with two categories, USA and NotUSA. Then, create a table from a subset of the variables. Include the categorical Origin variable.

load carbig

Origin = categorical(cellstr(Origin));
Origin = mergecats(Origin,["France","Japan","Germany", ...
    "Sweden","Italy","England"],"NotUSA");

cars = table(Acceleration,Displacement,Horsepower, ...
    Origin,MPG,Weight,Cylinders);

Train a classification tree using the data in cars. Specify Cylinders as the response variable.

Mdl = fitctree(cars,"Cylinders")

Mdl = 
  ClassificationTree
           PredictorNames: {'Acceleration'  'Displacement'  'Horsepower'  'Origin'  'MPG'  'Weight'}
             ResponseName: 'Cylinders'
    CategoricalPredictors: 4
               ClassNames: [3 4 5 6 8]
           ScoreTransform: 'none'
          NumObservations: 406


  Properties, Methods

Select an observation from the data. Check the class label that Mdl predicts for the observation.

idx = 5;
observation = cars(idx,:)

observation = 1×7 table
    Acceleration    Displacement    Horsepower    Origin    MPG    Weight    Cylinders
    ____________    ____________    __________    ______    ___    ______    _________

        10.5            302            140         USA      17      3449         8

predictedLabel = predict(Mdl,observation)

predictedLabel = 
8

Perturb the observation by increasing amounts. Note that you can perturb only numeric predictor values; categorical values must be the same for the lower and upper bounds.

To perturb the numeric predictor values, use the custom perturbPredictors function, which accepts numeric predictor data (numericPredictors) and a percentage (percent). For each predictor, the function computes the specified percentage of the interquartile range and returns the value (perturbation).

function perturbation = perturb(numericPredictors,percent)
    perturbation = percent*iqr(numericPredictors);
end

Increase the perturbation by increasing the percentage value from 5% to 25% in increments of 5%. Create a lower bound for the observation by subtracting the perturbation value, and create an upper bound by adding the perturbation value. Combine all the lower bounds in TblLower, and combine all the upper bounds in TblUpper.

numericPredictorNames = Mdl.PredictorNames;
numericPredictorNames(Mdl.CategoricalPredictors) = [];

percentRange = 0.05:0.05:0.25;
TblLower = repmat(observation,numel(percentRange),1);
TblUpper = TblLower;

for k = 1:numel(percentRange)
    percentk = percentRange(k);
    perturbk = @(numericPredictors)perturb(numericPredictors,percentk);
    perturbationk = varfun(perturbk,cars, ...
        InputVariables=numericPredictorNames);
    perturbationk.Properties.VariableNames = numericPredictorNames;

    TblLower(k,numericPredictorNames) = ...
        TblLower(k,numericPredictorNames) - perturbationk;

    TblUpper(k,numericPredictorNames) = ...
        TblUpper(k,numericPredictorNames) + perturbationk;
end

Verify the model robustness on the increasing perturbation regions.

results = verifyTreeRobustness(Mdl,TblLower,TblUpper,"Cylinders")

results = 5×1 categorical
     verified 
     verified 
     verified 
     verified 
     violated

In the first four perturbation regions, Mdl predicts the same class label (8) for all observations with values between the lower and upper bounds. In the last perturbation region, Mdl predicts a different class label (3, 4, 5, or 6) for at least one combination of predictor values between the lower bound TblLower(5,:) and the upper bound TblUpper(5,:).

Input Arguments

collapse all

`Mdl` — Trained tree classifier
`ClassificationTree` model object | `CompactClassificationTree` model object

Trained tree classifier, specified as a ClassificationTree (Statistics and Machine Learning Toolbox) or CompactClassificationTree (Statistics and Machine Learning Toolbox) model object.

You must specify a trained tree classifier that uses the default score transform. That is, Mdl.ScoreTransform must be "none" or "identity".

`TblLower` — Lower bounds
table

Lower bounds on the predictor data, specified as a table. The lower and upper bounds, TblLower and TblUpper, must have the same size and format. The function computes the results across the regions defined by the lower and upper bounds.

If variable k in TblLower is categorical, then TblLower(:,k) must match TblUpper(:,k). That is, for each region, the categorical predictor values for the lower bound and the categorical predictor values for the upper bound must be the same.

Data Types: table

`TblUpper` — Upper bounds
table

Upper bounds on the predictor data, specified as a table. The lower and upper bounds, TblLower and TblUpper, must have the same size and format. The function computes the results across the regions defined by the lower and upper bounds.

If variable k in TblUpper is categorical, then TblUpper(:,k) must match TblLower(:,k). That is, for each region, the categorical predictor values for the lower bound and the categorical predictor values for the upper bound must be the same.

Data Types: table

`responseVarName` — Response variable name
character vector | string scalar

Response variable name, specified as a character vector or string scalar. responseVarName must be the name of a variable in both TblLower and TblUpper. Each label i in the responseVarName variable is the expected class label for all observations in the region with lower bound TblLower(i,:) and upper bound TblUpper(i,:). For each region, the function verifies that the predicted class label returned by Mdl matches the label in the responseVarName variable.

Data Types: char | string

`labels` — Class labels
numeric vector | categorical vector | logical vector | character array | string array | cell array of character vectors

Class labels, specified as a numeric, categorical, or logical vector; a character or string array; or a cell array of character vectors. Each label i in labels is the expected class label for all observations in the region with lower bound i and upper bound i (for example, XLower(i,:) and XUpper(i,:), respectively). For each region, the function verifies that the predicted class label returned by Mdl matches the label in labels.

`XLower` — Numeric lower bounds
numeric matrix

Numeric lower bounds on the predictor data, specified as a numeric matrix. The lower and upper bounds, XLower and XUpper, must have the same size and format. The function computes the results across the regions defined by the lower and upper bounds.

Data Types: single | double

`XUpper` — Numeric upper bounds
numeric matrix

Numeric upper bounds on the predictor data, specified as a numeric matrix. The lower and upper bounds, XLower and XUpper, must have the same size and format. The function computes the results across the regions defined by the lower and upper bounds.

Data Types: single | double

Output Arguments

collapse all

`results` — Verification results
categorical array

Verification results, returned as a categorical array. For each set of lower and upper bounds, the function returns one of these values:

"verified" — The model is robust to perturbations in the region between the specified bounds for the specified label.
"violated" — The model is not robust to perturbations in the region between the specified bounds for the specified label.

Algorithms

collapse all

Tree Robustness

A tree classifier is stable at an observation if the predicted label remains the same when the observation is slightly perturbed (that is, the observation predictor values are slightly altered). If the predicted label is the same as the true class label, the classifier is also robust. Given the lower and upper bounds of the perturbation region, the verifyTreeRobustness function checks that all observations with predictor values in the region have the specified predicted class label.

The function leverages the structure of binary decision trees to determine the robustness of Mdl. Binary decision trees partition input data regions into hyperrectangles with different predictive outcomes. Observations belonging to the same hyperrectangle have the same predicted label. To verify robustness, verifyTreeRobustness uses an abstract interpretation algorithm that considers model predictions for the hyperrectangles instead of individual observations. For each set of lower and upper bounds, the function subdivides the specified region based on the hyperrectangles resulting from the tree splits. If all subsections of the specified region have the same classification as the predicted class label, verifyTreeRobustness returns the result "verified". If at least one subsection of the specified region has a different classification, the function returns the result "violated".

References

[1] Ranzato, Francesco, and Marco Zanella. “Abstract Interpretation of Decision Tree Ensemble Classifiers.” Proceedings of the AAAI Conference on Artificial Intelligence 34, no. 04 (2020): 5478–86. https://doi.org/10.1609/aaai.v34i04.5998.

Version History

Introduced in R2026b

verifyTreeRobustness

Syntax

Description

Examples

Verify Tree Robustness

Verify Tree Robustness for Increasing Perturbations

Input Arguments

Mdl — Trained tree classifier ClassificationTree model object | CompactClassificationTree model object

TblLower — Lower bounds table

TblUpper — Upper bounds table

responseVarName — Response variable name character vector | string scalar

labels — Class labels numeric vector | categorical vector | logical vector | character array | string array | cell array of character vectors

XLower — Numeric lower bounds numeric matrix

XUpper — Numeric upper bounds numeric matrix

Output Arguments

results — Verification results categorical array

Algorithms

Tree Robustness

References

Version History

See Also

`Mdl` — Trained tree classifier
`ClassificationTree` model object | `CompactClassificationTree` model object

`TblLower` — Lower bounds
table

`TblUpper` — Upper bounds
table

`responseVarName` — Response variable name
character vector | string scalar

`labels` — Class labels
numeric vector | categorical vector | logical vector | character array | string array | cell array of character vectors

`XLower` — Numeric lower bounds
numeric matrix

`XUpper` — Numeric upper bounds
numeric matrix

`results` — Verification results
categorical array