TrainingOptionsMiniBatchLBFGS

R2026b

Training options for mini-batch limited-memory BFGS (LBFGS) solver

Since R2026b

expand all in page

Description

A TrainingOptionsMiniBathLBFGS model object contains options for the mini-batch LBFGS solver. Use the model object with the TrainingOptions name-value argument of incrementalClassificationNeuralNetwork or incrementalRegressionNeuralNetwork to specify solver options for incremental learning.

Creation

Create an TrainingOptionsMiniBatchLBFGS object by using the incrementalTrainingOptions function and specifying "minibatch-lbfgs" as the first input argument.

Properties

expand all

Mini-batch size, specified as a positive integer. At each iteration, the software estimates the subgradient using BatchSize observations from the training data.

Relative convergence tolerance on the L-infinity norm of the gradient, specified as one of these values:

  • Positive scalar — Stop training when the relative gradient is less than or equal to the specified value.

  • 0 — Do not stop training based on the relative gradient.

Let ℒt be the loss function at training iteration t, ∇ℒt be the gradient of the loss function with respect to the weights and biases at iteration t, and ∇ℒ0 be the gradient of the loss function at an initial point. If max|∇ℒt|≤a⋅GradientTolerance, where a=max(1,min|ℒt|,max|∇ℒ0|), then the training process terminates.

Number of state updates to store, specified as a positive integer. Values between 3 and 20 suit most tasks. Larger values use more memory but can improve convergence. The LBFGS algorithm uses a history of gradient calculations to approximate the Hessian matrix recursively.

Initial value that characterizes the approximate inverse Hessian matrix, specified as a positive scalar.

To save memory, the LBFGS algorithm does not store and invert the dense Hessian matrix B. Instead, the algorithm uses the approximation Bk−m−1≈λkI, where m is the history size, the inverse Hessian factor λk is a scalar, and I is the identity matrix. The algorithm then stores the scalar inverse Hessian factor only. The algorithm updates the inverse Hessian factor at each step.

The initial inverse hessian factor is the value of λ0.

Initial learning rate, specified as a positive scalar or "auto". When set to "auto", the solver determines the initial learning rate during a tuning period controlled by TuningPeriod and TuningSubsetSize. When set to a numeric value, the tuning period is disabled (TuningPeriod and TuningSubsetSize are set to 0).

If the learning rate is too low, then training can take many iterations to converge. If the learning rate is too high, then training might converge to a suboptimal result or diverge.

Initial step size, specified as one of these values:

  • [] — Do not use an initial step size to determine the initial Hessian approximation.

  • "auto" — Determine the initial step size automatically. The software uses an initial step size of ‖s0‖∞=12‖W0‖∞+0.1, where W0 are the initial learnable parameters of the network.

  • Positive real scalar — Use the specified value as the initial step size ‖s0‖∞.

L2 regularization term strength, specified as a nonnegative scalar. Larger values apply stronger regularization to the model coefficients.

Learning rate schedule, specified as "decaying" or "constant". A decaying schedule reduces the learning rate over time, while a constant schedule keeps it fixed at the initial value.

Method to find suitable learning rate, specified as one of these values:

  • "weak-wolfe" — Search for a learning rate that satisfies the weak Wolfe conditions. This method maintains a positive definite approximation of the inverse Hessian matrix.

  • "strong-wolfe" — Search for a learning rate that satisfies the strong Wolfe conditions. This method maintains a positive definite approximation of the inverse Hessian matrix.

  • "backtracking" — Search for a learning rate that satisfies sufficient decrease conditions. This method does not maintain a positive definite approximation of the inverse Hessian matrix.

Maximum number of iterations per mini-batch LBFGS step, specified as a positive integer

Maximum number of line search iterations to determine the learning rate, specified as a positive integer.

L2 norm step size tolerance, specified as a nonnegative scalar.

If the step size at some iteration is smaller than StepTolerance, then the training process terminates.

Number of observations for learning rate tuning, specified as a nonnegative integer scalar. The solver uses the first TuningSubsetSize observations for testing, and the next TuningPeriod-TuningSubsetSize observations to determine a good initial learning rate when InitialLearnRate is "auto". If you specify TuningPeriod without specifying TuningSubsetSize, then TuningSubsetSize is set to ceil(0.1*TuningPeriod).

Number of observations in each tuning subset, specified as a nonnegative integer scalar. The value must be less than or equal to TuningPeriod. If you specify TuningSubsetSize without specifying TuningPeriod, then TuningPeriod is set to 10*TuningSubsetSize.

Object Functions

incrementalClassificationNeuralNetworkNeural network classification model for incremental learning
incrementalRegressionNeuralNetworkNeural network Regression model for incremental learning

Examples

collapse all

Open Live Script

Create a training options object that contains the default mini-batch LBFGS solver options for an incremental neural network learning object.

LBFGSOptions=incrementalTrainingOptions("minibatch-lbfgs")
LBFGSOptions = 
  TrainingOptionsMiniBatchLBFGS

               InitialLearnRate: "auto"
              LearnRateSchedule: "decaying"
                   TuningPeriod: 1000
               TuningSubsetSize: 100
                      BatchSize: 10
               L2Regularization: 1.0000e-05

   LBFGS Options
                    HistorySize: 10
    InitialInverseHessianFactor: 1
                InitialStepSize: []
               LineSearchMethod: "weak-wolfe"
     MaxNumLineSearchIterations: 20
               MaxNumIterations: 10
              GradientTolerance: 1.0000e-06
                  StepTolerance: 1.0000e-06


  Properties, Methods

LBFGSOptions is a TrainingOptionsMiniBatchLBFGS object. Create an incremental neural network classification model with a maximum of five expected classes using the default mini-batch LBFGS solver options.

IncrementalMdl = incrementalClassificationNeuralNetwork(MaxNumClasses=5,TrainingOptions=LBFGSOptions)
IncrementalMdl = 
  incrementalClassificationNeuralNetwork

                   IsWarm: 0
                  Metrics: [1×2 table]
               ClassNames: [1×0 double]
           ScoreTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "softmax"
                   Solver: "minibatch-lbfgs"


  Properties, Methods

IncrementalMdl is an incrementalClassificationNeuralNetwork model object. The solver options are stored in the object's TrainingOptions property.

Version History

Introduced in R2026b

See Also

| | |