Classification Ensembles

Boosting, random forest, bagging, random subspace, and ECOC ensembles for multiclass learning

A classification ensemble is a predictive model composed of a weighted combination of multiple classification models. In general, combining multiple classification models increases predictive performance.

To explore classification ensembles interactively, use the Classification Learner app. For greater flexibility, use fitcensemble in the command-line interface to boost or bag classification trees, or to grow a random forest. For details on all supported ensembles, see Ensemble Algorithms. To reduce a multiclass problem into an ensemble of binary classification problems, train an error-correcting output codes (ECOC) model. For details, see fitcecoc.

To boost regression trees using LSBoost, or to grow a random forest of regression trees, see Regression Ensembles.

Apps

Classification Learner

Train models to classify data using supervised machine learning

Blocks

ClassificationEnsemble Predict	Classify observations using ensemble of decision trees (Since R2021a)
ClassificationECOC Predict	Classify observations using error-correcting output codes (ECOC) classification model (Since R2023a)
IncrementalClassificationECOC Predict	Classify observations using incremental ECOC classification model (Since R2024a)

Functions

Create Learner Template

`templateDiscriminant`	Discriminant analysis classifier template
`templateECOC`	Error-correcting output codes learner template
`templateEnsemble`	Ensemble learning template
`templateKNN`	k-nearest neighbor classifier template
`templateLinear`	Linear learner template
`templateNaiveBayes`	Naive Bayes classifier template
`templateSVM`	Support vector machine template
`templateTree`	Create decision tree template

Classification Ensemble

Create Classification Ensemble

`fitcensemble`	Fit ensemble of learners for classification
`compact`	Reduce size of classification ensemble model

Modify Classification Ensemble

`resume`	Resume training of classification ensemble model
`removeLearners`	Remove members of compact classification ensemble

Interpret Classification Ensemble

`lime`	Local interpretable model-agnostic explanations (LIME) (Since R2020b)
`partialDependence`	Compute partial dependence (Since R2020b)
`permutationImportance`	Predictor importance by permutation (Since R2024a)
`plotPartialDependence`	Create partial dependence plot (PDP) and individual conditional expectation (ICE) plots
`predictorImportance`	Estimates of predictor importance for classification ensemble of decision trees
`shapley`	Shapley values (Since R2021a)

Cross-Validate Classification Ensemble

`crossval`	Cross-validate machine learning model
`kfoldEdge`	Classification edge for cross-validated classification model
`kfoldLoss`	Classification loss for cross-validated classification model
`kfoldMargin`	Classification margins for cross-validated classification model
`kfoldPredict`	Classify observations in cross-validated classification model
`kfoldfun`	Cross-validate function for classification

Measure Performance

`loss`	Classification loss for classification ensemble model
`resubLoss`	Resubstitution classification loss for classification ensemble model
`compareHoldout`	Compare accuracies of two classification models using new data
`edge`	Classification edge for classification ensemble model
`margin`	Classification margins for classification ensemble model
`resubEdge`	Resubstitution classification edge for classification ensemble model
`resubMargin`	Resubstitution classification margins for classification ensemble model
`testckfold`	Compare accuracies of two classification models by repeated cross-validation

Classify Observations

`predict`	Predict labels using classification ensemble model
`resubPredict`	Classify observations in classification ensemble by resubstitution
`oobPredict`	Predict out-of-bag labels and scores of bagged classification ensemble

Gather Properties of Classification Ensemble

gather Gather properties of Statistics and Machine Learning Toolbox object from GPU (Since R2020b)

Bagged Classification Trees

`fitcensemble`	Fit ensemble of learners for classification
`TreeBagger`	Ensemble of bagged decision trees
`predict`	Predict responses using ensemble of bagged decision trees
`oobPredict`	Ensemble predictions for out-of-bag observations

Multiclass ECOC

Create ECOC

`fitcecoc`	Fit multiclass models for support vector machines or other classifiers
`compact`	Reduce size of multiclass error-correcting output codes (ECOC) model

Modify ECOC

discardSupportVectors Discard support vectors of linear SVM binary learners in ECOC model

Interpret ECOC

`lime`	Local interpretable model-agnostic explanations (LIME) (Since R2020b)
`partialDependence`	Compute partial dependence (Since R2020b)
`plotPartialDependence`	Create partial dependence plot (PDP) and individual conditional expectation (ICE) plots
`shapley`	Shapley values (Since R2021a)

Cross-Validate ECOC

`crossval`	Cross-validate machine learning model
`kfoldEdge`	Classification edge for cross-validated ECOC model
`kfoldLoss`	Classification loss for cross-validated ECOC model
`kfoldMargin`	Classification margins for cross-validated ECOC model
`kfoldPredict`	Classify observations in cross-validated ECOC model
`kfoldfun`	Cross-validate function using cross-validated ECOC model

Measure Performance

`loss`	Classification loss for multiclass error-correcting output codes (ECOC) model
`resubLoss`	Resubstitution classification loss for multiclass error-correcting output codes (ECOC) model
`compareHoldout`	Compare accuracies of two classification models using new data
`edge`	Classification edge for multiclass error-correcting output codes (ECOC) model
`margin`	Classification margins for multiclass error-correcting output codes (ECOC) model
`resubEdge`	Resubstitution classification edge for multiclass error-correcting output codes (ECOC) model
`resubMargin`	Resubstitution classification margins for multiclass error-correcting output codes (ECOC) model
`testckfold`	Compare accuracies of two classification models by repeated cross-validation

Classify Observations

`predict`	Classify observations using multiclass error-correcting output codes (ECOC) model
`resubPredict`	Classify observations in multiclass error-correcting output codes (ECOC) model

Gather Properties of ECOC

gather Gather properties of Statistics and Machine Learning Toolbox object from GPU (Since R2020b)

Objects

Classification Ensembles

`ClassificationEnsemble`	Ensemble classifier
`CompactClassificationEnsemble`	Compact classification ensemble
`ClassificationPartitionedEnsemble`	Cross-validated classification ensemble

Bagged Classification Trees

`TreeBagger`	Ensemble of bagged decision trees
`CompactTreeBagger`	Compact ensemble of bagged decision trees
`ClassificationBaggedEnsemble`	Classification ensemble grown by resampling

Classes

Multiclass ECOC

`ClassificationECOC`	Multiclass model for support vector machines (SVMs) and other classifiers
`CompactClassificationECOC`	Compact multiclass model for support vector machines (SVMs) and other classifiers
`ClassificationPartitionedECOC`	Cross-validated multiclass ECOC model for support vector machines (SVMs) and other classifiers

Topics

Train Ensemble Classifiers Using Classification Learner App
Create and compare ensemble classifiers, and export trained models to make predictions for new data.
Framework for Ensemble Learning
Obtain highly accurate predictions by using many weak learners.
Ensemble Algorithms
Learn about different algorithms for ensemble learning.
Train Classification Ensemble
Train a simple classification ensemble.
Test Ensemble Quality
Learn methods to evaluate the predictive quality of an ensemble.
Handle Imbalanced Data or Unequal Misclassification Costs in Classification Ensembles
Learn how to set prior class probabilities and misclassification costs.
Classification with Imbalanced Data
Use the RUSBoost algorithm for classification when one or more classes are over-represented in your data.
LPBoost and TotalBoost for Small Ensembles
Create small ensembles by using the LPBoost and TotalBoost algorithms.
Tune RobustBoost
Tune RobustBoost parameters for better predictive accuracy.
Surrogate Splits
Gain better predictions when you have missing data by using surrogate splits.
Train Classification Ensemble in Parallel
Train a bagged ensemble in parallel reproducibly.
Bootstrap Aggregation (Bagging) of Classification Trees Using TreeBagger
Create a TreeBagger ensemble for classification.
Credit Rating by Bagging Decision Trees
This example shows how to build an automated credit rating tool.
Random Subspace Classification
Increase the accuracy of classification by using a random subspace ensemble.
Predict Class Labels Using ClassificationEnsemble Predict Block
Train a classification ensemble model with optimal hyperparameters, and then use the ClassificationEnsemble Predict block for label prediction.
Predict Class Labels Using ClassificationECOC Predict Block
Train an ECOC classification model, and then use the ClassificationECOC Predict block for label prediction. (Since R2023a)