updateMetricsAndFit

R2026b

Update performance metrics in neural network incremental learning model given new data and train model

Since R2026b

collapse all in page

Syntax

Mdl = updateMetricsAndFit(Mdl,X,Y)
Mdl = updateMetricsAndFit(Mdl,X,Y,Name=Value)
[Mdl,ConvergenceInfo] = updateMetricsAndFit(___)

Description

Given streaming data, updateMetricsAndFit first evaluates the performance of a configured neural network incremental learning model for regression (incrementalRegressionNeuralNetwork model object) or classification (incrementalClassificationNeuralNetwork model object) by calling updateMetrics on incoming data. Then updateMetricsAndFit fits the model to that data by calling fit. In other words, updateMetricsAndFit performs prequential evaluation because it treats each incoming chunk of data as a test set, and tracks performance metrics measured cumulatively and over a specified window [1].

updateMetricsAndFit provides a simple way to update model performance metrics and train the model on each chunk of data. Alternatively, you can perform the operations separately by calling updateMetrics and then fit, which allows for more flexibility (for example, you can decide whether you need to train the model based on its performance on a chunk of data).

Mdl = updateMetricsAndFit(Mdl,X,Y) returns an incremental learning model Mdl, which is the input learning model Mdl with the following modifications:

  1. updateMetricsAndFit measures the model performance on the incoming predictor and response data, X and Y respectively. When the input model is warm (Mdl.IsWarm is true), updateMetricsAndFit overwrites previously computed metrics, stored in the Metrics property, with the new values. Otherwise, updateMetricsAndFit stores NaN values in Metrics instead.

  2. updateMetricsAndFit fits the modified model to the incoming data by updating the neural network weights and biases using the specified solver algorithm, and stores the new parameters in the output model Mdl.

example

Mdl = updateMetricsAndFit(Mdl,X,Y,Name=Value) uses additional options specified by one or more name-value arguments. For example, you can specify that the columns of the predictor data matrix correspond to observations, and set observation weights.

example

[Mdl,ConvergenceInfo] = updateMetricsAndFit(___) also returns solver convergence information in the structure ConvergenceInfo, using any of the input arguments from the previous syntaxes.

Examples

collapse all

Open Live Script

Load the human activity data set. Randomly shuffle the data.

load humanactivity
n = numel(actid);
rng(0,"twister"); % For reproducibility
idx = randsample(n,n);
X = feat(idx,:);
Y = actid(idx);

The class names map 1 through 5 to an activity—sitting, standing, walking, running, or dancing, respectively—based on biometric data measured on the subject. For details on the data set, enter Description at the command line.

Create an incremental neural network model for multiclass learning. Configure the model as follows:

  • Specify a metrics warm-up period of 5000 observations.

  • Specify a metrics window size of 500 observations.

  • Standardize the predictor data and specify an estimation period of 1000 observations.

  • Use the mini-batch LBFGS solver and a solver tuning period of 500 observations.

  • Double the penalty to the classifier when it mistakenly classifies class 2.

  • Track the classification error and minimal cost to measure the performance of the model. You do not have to specify mincost for Metrics because incrementalClassificationNeuralNetwork always tracks this metric.

C = ones(5) - eye(5);
C(2,[1 3 4 5]) = 2;
Mdl = incrementalClassificationNeuralNetwork(ClassNames=1:5, ...
    MetricsWarmupPeriod=5000,MetricsWindowSize=500, ...
    Standardize=true,EstimationPeriod=1000, ...
    TrainingOptions=incrementalTrainingOptions("minibatch-lbfgs", ...
    TuningPeriod=500),Cost=C,Metrics="classiferror")
Mdl = 
  incrementalClassificationNeuralNetwork

                   IsWarm: 0
                  Metrics: [2×2 table]
               ClassNames: [1 2 3 4 5]
           ScoreTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "softmax"
                   Solver: "minibatch-lbfgs"


  Properties, Methods

Mdl is an incrementalClassificationNeuralNetwork model object configured for incremental learning.

Fit the incremental model to the rest of the data by using the updateMetricsAndFit function. At each iteration:

  • Simulate a data stream by processing a chunk of 50 observations.

  • Overwrite the previous incremental model with a new one fitted to the incoming observations.

  • Store the standard deviation of the first predictor variable σ1, the cumulative metrics, and the window metrics to see how they evolve during incremental learning.

% Preallocation
numObsPerChunk = 50;
nchunk = floor(n/numObsPerChunk);
ce = array2table(zeros(nchunk,2),VariableNames=["Cumulative" "Window"]);
mc = array2table(zeros(nchunk,2),VariableNames=["Cumulative" "Window"]);
sigma1 = zeros(nchunk+1,1);    

% Incremental fitting
for j = 1:nchunk
    ibegin = min(n,numObsPerChunk*(j-1) + 1);
    iend   = min(n,numObsPerChunk*j);
    idx = ibegin:iend;    
    Mdl = updateMetricsAndFit(Mdl,X(idx,:),Y(idx));
    ce{j,:} = Mdl.Metrics{"ClassificationError",:};
    mc{j,:} = Mdl.Metrics{"MinimalCost",:};
    sigma1(j) = Mdl.Sigma(1);
end

Mdl is an incrementalClassificationNeuralNetwork model object trained on all the data in the stream. During incremental learning and after the model is warmed up, updateMetricsAndFit checks the performance of the model on the incoming observations, and then fits the model to those observations.

To see how the performance metrics and σ1 evolve during training, plot them on separate tiles.

tiledlayout(2,2)
nexttile
plot(sigma1)
ylabel("\sigma_{1}")
xlim([0 nchunk]);
xline(Mdl.EstimationPeriod/numObsPerChunk,"b--")
xlabel("Iteration")
nexttile
h = plot(ce.Variables);
xlim([0 nchunk])
ylabel("Classification Error")
xline((Mdl.EstimationPeriod + Mdl.TrainingOptions.TuningPeriod + ...
 Mdl.MetricsWarmupPeriod)/numObsPerChunk,"r-.")
legend(h,ce.Properties.VariableNames)
xlabel("Iteration")
nexttile
h = plot(mc.Variables);
xlim([0 nchunk]);
ylabel("Minimal Cost")
xline((Mdl.EstimationPeriod + Mdl.TrainingOptions.TuningPeriod + ...
    Mdl.MetricsWarmupPeriod)/numObsPerChunk,"r-.")
legend(h,mc.Properties.VariableNames)
xlabel("Iteration")

The plots indicate that updateMetricsAndFit performs the following actions:

  • Fit σ1 after the estimation period (blue vertical line).

  • Compute the performance metrics after the estimation period, tuning period, and metrics warm-up period (red vertical line) only.

  • Compute the cumulative metrics during each iteration.

  • Compute the window metrics after processing 500 observations (10 iterations).

Open Live Script

Train a neural network classification model by using fitcnet and convert it to an incremental learner by using incrementalLearner. Track the model performance on streaming data and fit the model to streaming data in one call by using updateMetricsAndFit. Specify the orientation of observations and the observation weights when you call updateMetricsAndFit.

Load and Preprocess Data

Load the human activity data set. Randomly shuffle the data.

load humanactivity
rng(0,"twister") % For reproducibility
n = numel(actid);
idx = randsample(n,n);
X = feat(idx,:);
Y = actid(idx);

For details on the data set, enter Description at the command line.

Suppose that the data from a stationary subject (Y <= 2) has double the quality of data from a moving subject. Create a weight variable that assigns a weight of 2 to observations from a stationary subject and 1 to a moving subject.

W = ones(n,1) + (Y <= 2);

Train Neural Network Classification Model

Fit a neural network classification model to a random sample of half the data.

idxtt = randsample([true false],n,true);
TTMdl = fitcnet(X(idxtt,:),Y(idxtt),Weights=W(idxtt))
TTMdl = 
  ClassificationNeuralNetwork
             ResponseName: 'Y'
    CategoricalPredictors: []
               ClassNames: [1 2 3 4 5]
           ScoreTransform: 'none'
          NumObservations: 12039
               LayerSizes: 10
              Activations: 'relu'
    OutputLayerActivation: 'softmax'
                   Solver: 'LBFGS'
          ConvergenceInfo: [1×1 struct]
          TrainingHistory: [1000×7 table]


  Properties, Methods

TTMdl is a ClassificationNeuralNetwork model object representing a traditionally trained neural network classification model.

Convert Trained Model

Convert the traditionally trained model to a model for incremental learning and specify to track the classification error metric.

IncrementalMdl = incrementalLearner(TTMdl,Metrics="classiferror")
IncrementalMdl = 
  incrementalClassificationNeuralNetwork

                   IsWarm: 0
                  Metrics: [2×2 table]
               ClassNames: [1 2 3 4 5]
           ScoreTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "softmax"
                   Solver: "minibatch-lbfgs"


  Properties, Methods

IncrementalMdl is an incrementalClassificationNeuralNetwork model object. Because class names are specified in IncrementalMdl.ClassNames, labels encountered during incremental learning must be in IncrementalMdl.ClassNames.

Track Performance Metrics and Fit Model

Perform incremental learning on the rest of the data by using the updateMetricsAndFit function. Transpose the predictor matrix, and specify the data orientation when you call updateMetricsAndFit. At each iteration:

  1. Simulate a data stream by processing 50 observations at a time.

  2. Call updateMetricsAndFit to update the cumulative and window performance metrics of the model given the incoming chunk of observations, and then fit the model to the data. Overwrite the previous incremental model with a new one. Specify that the observations are oriented in columns, and specify the observation weights.

  3. Store the misclassification error rate.

% Preallocation
idxil = ~idxtt;
nil = sum(idxil);
numObsPerChunk = 50;
nchunk = floor(nil/numObsPerChunk);
mc = array2table(zeros(nchunk,2),VariableNames=["Cumulative","Window"]);
Xil = X(idxil,:)';
Yil = Y(idxil);
Wil = W(idxil);

% Incremental fitting
for j = 1:nchunk
    ibegin = min(nil,numObsPerChunk*(j-1) + 1);
    iend   = min(nil,numObsPerChunk*j);
    idx = ibegin:iend;
    IncrementalMdl = updateMetricsAndFit(IncrementalMdl,Xil(:,idx),Yil(idx), ...
        Weights=Wil(idx),ObservationsIn="columns");
    mc{j,:} = IncrementalMdl.Metrics{"ClassificationError",:};
end

IncrementalMdl is an incrementalClassificationECOC model object trained on all the data in the stream.

Create a trace plot of the misclassification error rate.

plot(mc.Variables)
xlim([0 nchunk])
ylabel("Classification Error")
legend(mc.Properties.VariableNames)
xlabel("Iteration")

Figure contains an axes object. The axes object with xlabel Iteration, ylabel Classification Error contains 2 objects of type line. These objects represent Cumulative, Window.

The cumulative loss initially has a high value, but stabilizes around 0.05, whereas the window loss jumps throughout the training.

Open Live Script

Prepare an incremental regression learner by specifying a metrics warm-up period and a metrics window size. Train the model by using SGD, and adjust the SGD batch size, learning rate, and regularization parameter.

Load the robot arm data set. 

load robotarm

For details on the data set, enter Description at the command line.

Create an incremental neural network model for regression. Configure the model as follows:

  • Specify a metrics warm-up period of 1000 observations.

  • Specify a metrics window size of 500 observations.

  • Specify the FreeRex solver and apply parameter updates based on the L2-norm of the gradient.

  • Track the mean squared error (MSE) and mean absolute error (MAE) to measure the performance of the model. Create an anonymous function that measures the absolute error of each new observation. Create a structure array containing the name MeanAbsoluteError and its corresponding function.

maefcn = @(z,zfit)abs(z - zfit);
maemetric = struct("MeanAbsoluteError",maefcn);

Mdl = incrementalRegressionNeuralNetwork(MetricsWarmupPeriod=1000,MetricsWindowSize=500, ...
    TrainingOptions=incrementalTrainingOptions("freerex",UpdateMethod="l2-norm"), ...
    Metrics={"mse",maemetric})
Mdl = 
  incrementalRegressionNeuralNetwork

                   IsWarm: 0
                  Metrics: [2×2 table]
        ResponseTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "none"
                   Solver: "freerex"


  Properties, Methods

Mdl is an incrementalRegressionNeuralNetwork model object configured for incremental learning without an estimation period or solver tuning period.

Fit the incremental model to the data by using the updateMetricsAndFit function. At each iteration:

  • Simulate a data stream by processing a chunk of 50 observations.

  • Overwrite the previous incremental model with a new one fitted to the incoming observations.

  • Store the cumulative metrics, window metrics, and number of training observations to see how they evolve during incremental learning.

% Preallocation
n = numel(ytrain);
numObsPerChunk = 50;
nchunk = floor(n/numObsPerChunk);
mse = array2table(zeros(nchunk,2),VariableNames=["Cumulative","Window"]);
mae = array2table(zeros(nchunk,2),VariableNames=["Cumulative","Window"]);  
numtrainobs = zeros(nchunk,1);

% Incremental fitting
rng(0,"twister") % For reproducibility
for j = 1:nchunk
    ibegin = min(n,numObsPerChunk*(j-1) + 1);
    iend   = min(n,numObsPerChunk*j);
    idx = ibegin:iend;    
    Mdl = updateMetricsAndFit(Mdl,Xtrain(idx,:),ytrain(idx));
    mse{j,:} = Mdl.Metrics{"MeanSquaredError",:};
    mae{j,:} = Mdl.Metrics{"MeanAbsoluteError",:};
    numtrainobs(j) = Mdl.NumTrainingObservations;
end

Mdl is an incrementalRegressionNeuralNetwork model object trained on all the data in the stream. During incremental learning and after the model is warmed up, updateMetricsAndFit checks the performance of the model on the incoming observations, and then fits the model to those observations.

Plot a trace plot of the number of training observations and the performance metrics on separate tiles.

t = tiledlayout(3,1);
nexttile
plot(numtrainobs)
xlim([0 nchunk])
ylabel(["Number of","Training Observations"])
xline(Mdl.MetricsWarmupPeriod/numObsPerChunk,"--")
nexttile
plot(mse.Variables)
xlim([0 nchunk])
ylabel("MSE")
xline(Mdl.MetricsWarmupPeriod/numObsPerChunk,"--")
legend(mse.Properties.VariableNames)
nexttile
plot(mae.Variables)
xlim([0 nchunk])
ylabel("MAE")
xline(Mdl.MetricsWarmupPeriod/numObsPerChunk,"--")
legend(mae.Properties.VariableNames)
xlabel(t,"Iteration")

Figure contains 3 axes objects. Axes object 1 with ylabel Number of Training Observations contains 2 objects of type line, constantline. Axes object 2 with ylabel MSE contains 3 objects of type line, constantline. These objects represent Cumulative, Window. Axes object 3 with ylabel MAE contains 3 objects of type line, constantline. These objects represent Cumulative, Window.

The plot suggests that updateMetricsAndFit does the following:

  • Fit the model during all incremental learning iterations.

  • Compute the performance metrics after the metrics warm-up period only (dashed vertical line).

  • Compute the cumulative metrics during each iteration.

  • Compute the window metrics after processing 500 observations (10 iterations).

Input Arguments

collapse all

Incremental learning model whose performance is measured and then the model is fit to data, specified as an incrementalClassificationNeuralNetwork or incrementalRegressionNeuralNetwork model object. You can create Mdl directly or by converting a supported, traditionally trained machine learning model using the incrementalLearner function. For more details, see the corresponding reference page.

If Mdl.IsWarm is false, updateMetricsAndFit does not track the performance of the model. For more details, see Performance Metrics.

Chunk of predictor data, specified as a floating-point matrix of n observations and Mdl.NumPredictors predictor variables. The value of the ObservationsIn name-value argument determines the orientation of the variables and observations. The default ObservationsIn value is "rows", which indicates that observations in the predictor data are oriented along the rows of X.

The length of the observation responses (labels) Y and the number of observations in X must be equal; Y(j) is the response (label) of observation j (row or column) in X.

Note

  • updateMetricsAndFit supports only floating-point input predictor data. If your input data includes categorical data, you must prepare an encoded version of the categorical data. Use dummyvar to convert each categorical variable to a numeric matrix of dummy variables. Then, concatenate all dummy variable matrices and any other numeric predictors. For more details, see Dummy Variables.

Data Types: single | double

Chunk of responses (labels), specified as a categorical, character, or string array, a logical or floating-point vector, or a cell array of character vectors for classification problems; or a floating-point vector for regression problems.

The length of the observation responses Y and the number of observations in X must be equal; Y(j) is the response of observation j (row or column) in X.

For classification problems, updateMetricsAndFit issues an error when one or both of these conditions are met:

  • Y contains a new label and the maximum number of classes has already been reached (see the ClassNames and MaxNumClasses arguments of incrementalClassificationNeuralNetwork).

  • The ClassNames property of the input model Mdl is nonempty, and the data types of Y and Mdl.ClassNames are different.

Data Types: char | string | cell | categorical | logical | single | double

Note

If an observation (predictor or label) or weight contains at least one missing (NaN) value, updateMetricsAndFit ignores the observation. Consequently, updateMetricsAndFit uses fewer than n observations to compute the model performance and create an updated model, where n is the number of observations in X.

Name-Value Arguments

collapse all

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: updateMetricsAndFit(Mdl,X,Y,ObservationsIn="columns",Weights=W) specifies that the columns of the predictor matrix correspond to observations, and the vector W contains observation weights to apply during incremental learning.

Predictor data observation dimension, specified as "rows" or "columns".

Example: ObservationsIn="columns"

Data Types: char | string

Chunk of observation weights, specified as a floating-point vector of positive values. updateMetricsAndFit weighs the observations in X with the corresponding values in Weights. The size of Weights must equal n, which is the number of observations in X.

By default, Weights is ones(n,1).

For more details, including normalization schemes, see Observation Weights.

Example: Weights=W specifies the observation weights as the vector W.

Data Types: double | single

Output Arguments

collapse all

Updated incremental learning model, returned as an incremental learning model object of the same data type as the input model Mdl, either incrementalClassificationNeuralNetwork or incrementalRegressionNeuralNetwork.

If the model is not warm, updateMetricsAndFit does not compute performance metrics. As a result, the Metrics property of Mdl remains completely composed of NaN values. If the model is warm, updateMetricsAndFit computes the cumulative and window performance metrics on the new data X and Y, and overwrites the corresponding elements of Mdl.Metrics. For more details, see Performance Metrics.

After updating metrics, updateMetricsAndFit trains the model on the incoming data. Specifically, it updates the LayerWeights, LayerBiases, and NumTrainingObservations properties.

Solver convergence information, returned as a structure containing the following fields:

  • GradientsNorm — A real nonnegative scalar specifying the L-infinity norm of the gradient at the last iteration.

  • StepNorm — A real nonnegative scalar specifying the L2 norm of the step taken at the last iteration.

  • Gradients — A cell array of numeric matrices specifying the gradients of the loss with respect to the parameters at the last iteration. Gradients has 2*K cells, where K is the number of cells in the LayerWeights property of Mdl. The first K cells correspond to the gradients with respect to Mdl.LayerWeights, and the remaining cells correspond to the gradients with respect to Mdl.LayerBiases.

During the estimation and solver tuning periods, the function returns NaN values for all fields.

Algorithms

collapse all

References

[1] Bifet, Albert, Ricard Gavaldá, Geoffrey Holmes, and Bernhard Pfahringer. Machine Learning for Data Streams with Practical Example in MOA. Cambridge, MA: The MIT Press, 2007.

Version History

Introduced in R2026b

See Also

Objects

Functions

Topics