Custom Training Loops

Customize deep learning training loops and loss functions for sequence and tabular data

If the trainingOptions function does not provide the training options that you need for your task, or you have a loss function that the trainnet function does not support, then you can define a custom training loop. For models that cannot be specified as networks of layers, you can define the model as a function. To learn more, see Define Custom Training Loops, Loss Functions, and Networks.

Functions

expand all

Custom Training Loops

`dlnetwork`	Deep learning neural network
`trainingProgressMonitor`	Monitor and plot training progress for deep learning custom training loops (Since R2022b)
`minibatchqueue`	Create mini-batches for deep learning (Since R2020b)
`padsequences`	Pad or truncate sequence data to same length (Since R2021a)
`dlarray`	Deep learning array for customization
`dlgradient`	Compute gradients for custom training loops using automatic differentiation
`dljacobian`	Jacobian matrix deep learning operation (Since R2024b)
`dldivergence`	Divergence of deep learning data (Since R2024b)
`dllaplacian`	Laplacian of deep learning data (Since R2024b)
`dlfeval`	Evaluate deep learning model for custom training loops

Loss Operations

`crossentropy`	Cross-entropy loss for classification tasks
`indexcrossentropy`	Index cross-entropy loss for classification tasks (Since R2024b)
`l1loss`	L₁ loss for regression tasks (Since R2021b)
`l2loss`	L₂ loss for regression tasks (Since R2021b)
`huber`	Huber loss for regression tasks (Since R2021a)
`ctc`	Connectionist temporal classification (CTC) loss for unaligned sequence classification (Since R2021a)
`mse`	Half mean squared error

Deep Learning Operations

`dlconv`	Deep learning convolution
`dltranspconv`	Deep learning transposed convolution
`lstm`	Long short-term memory
`gru`	Gated recurrent unit (Since R2020a)
`attention`	Dot-product attention (Since R2022b)
`embed`	Embed discrete data (Since R2020b)
`fullyconnect`	Sum all weighted input data and apply a bias
`dlode45`	Deep learning solution of nonstiff ordinary differential equation (ODE) (Since R2021b)
`batchnorm`	Normalize data across all observations for each channel independently
`crosschannelnorm`	Cross channel square-normalize using local responses (Since R2020a)
`groupnorm`	Normalize data across grouped subsets of channels for each observation independently (Since R2020b)
`instancenorm`	Normalize across each channel for each observation independently (Since R2021a)
`layernorm`	Normalize data across all channels for each observation independently (Since R2021a)
`avgpool`	Pool data to average values over spatial dimensions
`maxpool`	Pool data to maximum value
`maxunpool`	Unpool the output of a maximum pooling operation

Activations

`relu`	Apply rectified linear unit activation
`leakyrelu`	Apply leaky rectified linear unit activation
`gelu`	Apply Gaussian error linear unit (GELU) activation (Since R2022b)
`softmax`	Apply softmax activation to channel dimension
`sigmoid`	Apply sigmoid activation

Topics

Custom Training Loops

Train Deep Learning Model in MATLAB
Learn how to training deep learning models in MATLAB^®.
Define Custom Training Loops, Loss Functions, and Networks
Learn how to define and customize deep learning training loops, loss functions, and models.
Train Sequence Classification Network Using Custom Training Loop
This example shows how to train a network that classifies sequences with a custom learning rate schedule.
Monitor Custom Training Loop Progress
Track and plot custom training loop progress.
Train Network with Multiple Outputs
This example shows how to train a deep learning network with multiple outputs that predict both labels and angles of rotations of handwritten digits.
Classify Videos Using Deep Learning with Custom Training Loop
This example shows how to create a network for video classification by combining a pretrained image classification model and a sequence classification network.
Train Neural ODE Network
This example shows how to train an augmented neural ordinary differential equation (ODE) network.
Solve Ordinary Differential Equation Using Neural Network
This example shows how to solve an ordinary differential equation (ODE) using a neural network.
Speed Up Deep Neural Network Training
Learn how to accelerate deep neural network training.

Automatic Differentiation

Create Bidirectional LSTM (BiLSTM) Function
This example shows how to create a bidirectional long-short term memory (BiLSTM) function for custom deep learning functions. (Since R2023b)
List of Functions with dlarray Support
View the list of functions that support dlarray objects.
Automatic Differentiation Background
Learn how automatic differentiation works.
Use Automatic Differentiation In Deep Learning Toolbox
How to use automatic differentiation in deep learning.

Featured Examples

Solve PDE Using Physics-Informed Neural Network

Train a physics-informed neural network (PINN) to predict the solutions of the Burger's equation.

Open Live Script

Train Latent ODE Network with Irregularly Sampled Time-Series Data

Train a latent ordinary differential equation (ODE) autoencoder with time-series data that is sampled at irregular time intervals.

Open Live Script

Multivariate Time Series Anomaly Detection Using Graph Neural Network

Detect anomalies in multivariate time series data using a graph neural network (GNN).

Open Live Script

Dynamical System Modeling Using Neural ODE

Train a neural network with neural ordinary differential equations (ODEs) to learn the dynamics of a physical system.

Open Live Script

Sequence-to-Sequence Translation Using Attention

Convert decimal strings to Roman numerals using a recurrent sequence-to-sequence encoder-decoder model with attention.

Open Live Script

Image Captioning Using Attention

Train a deep learning model for image captioning using attention.

Open Live Script

Multilabel Text Classification Using Deep Learning

Classify text data that has multiple independent labels.

Open Live Script