batchNormalizationLayer

Batch normalization layer

Description

A batch normalization layer normalizes a mini-batch of data across all observations for each channel independently. To speed up training of the convolutional neural network and reduce the sensitivity to network initialization, use batch normalization layers between convolutional layers and nonlinearities, such as ReLU layers.

After normalization, the layer scales the input with a learnable scale factor γ and shifts it by a learnable offset β.

Creation

Syntax

layer = batchNormalizationLayer

layer = batchNormalizationLayer(Name,Value)

Description

layer = batchNormalizationLayer creates a batch normalization layer.

layer = batchNormalizationLayer(Name,Value) creates a batch normalization layer and sets the optional TrainedMean, TrainedVariance, Epsilon, Parameters and Initialization, Learning Rate and Regularization, and Name properties using one or more name-value pairs. For example, batchNormalizationLayer('Name','batchnorm') creates a batch normalization layer with the name 'batchnorm'.

example

Properties

expand all

Batch Normalization

`TrainedMean` — Mean statistic used for prediction
numeric vector

Mean statistic used for prediction, specified as a numeric vector of per-channel mean values.

Depending on the type of layer input, the trainnet and dlnetwork functions automatically reshape this property to have of the following sizes:

Layer Input	Property Size
feature input	`NumChannels`-by-1
vector sequence input	`NumChannels`-by-1
1-D image input	1-by-`NumChannels`
1-D image sequence input	1-by-`NumChannels`
2-D image input	1-by-1-by-`NumChannels`
2-D image sequence input	1-by-1-by-`NumChannels`
3-D image input	1-by-1-by-1-by-`NumChannels`
3-D image sequence input	1-by-1-by-1-by-`NumChannels`

If the BatchNormalizationStatistics training option is 'moving', then the software approximates the batch normalization statistics during training using a running estimate and, after training, sets the TrainedMean and TrainedVariance properties to the latest values of the moving estimates of the mean and variance, respectively.

If the BatchNormalizationStatistics training option is 'population', then after network training finishes, the software passes through the data once more and sets the TrainedMean and TrainedVariance properties to the mean and variance computed from the entire training data set, respectively.

The layer uses TrainedMean and TrainedVariance to normalize the input during prediction.

Data Types: single | double

`TrainedVariance` — Variance statistic used for prediction
numeric vector

Variance statistic used for prediction, specified as a numeric vector of per-channel variance values.

Depending on the type of layer input, the trainnet and dlnetwork functions automatically reshape this property to have of the following sizes:

Layer Input	Property Size
feature input	`NumChannels`-by-1
vector sequence input	`NumChannels`-by-1
1-D image input	1-by-`NumChannels`
1-D image sequence input	1-by-`NumChannels`
2-D image input	1-by-1-by-`NumChannels`
2-D image sequence input	1-by-1-by-`NumChannels`
3-D image input	1-by-1-by-1-by-`NumChannels`
3-D image sequence input	1-by-1-by-1-by-`NumChannels`

The layer uses TrainedMean and TrainedVariance to normalize the input during prediction.

Data Types: single | double

`Epsilon` — Constant to add to mini-batch variances
`1e-5` (default) | positive scalar

Constant to add to the mini-batch variances, specified as a positive scalar.

The software adds this constant to the mini-batch variances before normalization to ensure numerical stability and avoid division by zero.

Before R2023a: Epsilon must be greater than or equal to 1e-5.

`NumChannels` — Number of input channels
`"auto"` (default) | positive integer

This property is read-only.

Number of input channels, specified as one of the following:

"auto" — Automatically determine the number of input channels at training time.
Positive integer — Configure the layer for the specified number of input channels. NumChannels and the number of channels in the layer input data must match. For example, if the input is an RGB image, then NumChannels must be 3. If the input is the output of a convolutional layer with 16 filters, then NumChannels must be 16.

Parameters and Initialization

`ScaleInitializer` — Function to initialize channel scale factors
`'ones'` (default) | `'narrow-normal'` | function handle

Function to initialize the channel scale factors, specified as one of the following:

'ones' – Initialize the channel scale factors with ones.
'zeros' – Initialize the channel scale factors with zeros.
'narrow-normal' – Initialize the channel scale factors by independently sampling from a normal distribution with a mean of zero and standard deviation of 0.01.
Function handle – Initialize the channel scale factors with a custom function. If you specify a function handle, then the function must be of the form scale = func(sz), where sz is the size of the scale. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the channel scale factors when the Scale property is empty.

Data Types: char | string | function_handle

`OffsetInitializer` — Function to initialize channel offsets
`'zeros'` (default) | `'ones'` | `'narrow-normal'` | function handle

Function to initialize the channel offsets, specified as one of the following:

'zeros' – Initialize the channel offsets with zeros.
'ones' – Initialize the channel offsets with ones.
'narrow-normal' – Initialize the channel offsets by independently sampling from a normal distribution with a mean of zero and standard deviation of 0.01.
Function handle – Initialize the channel offsets with a custom function. If you specify a function handle, then the function must be of the form offset = func(sz), where sz is the size of the scale. For an example, see Specify Custom Weight Initialization Function.

The layer only initializes the channel offsets when the Offset property is empty.

Data Types: char | string | function_handle

`Scale` — Channel scale factors
`[]` (default) | numeric array

Channel scale factors γ, specified as a numeric array.

The channel scale factors are learnable parameters. When you train a network using the trainnet function or initialize a dlnetwork object, if Scale is nonempty, then the software uses the Scale property as the initial value. If Scale is empty, then the software uses the initializer specified by ScaleInitializer.

Depending on the type of layer input, the trainnet and dlnetwork functions automatically reshape this property to have of the following sizes:

Layer Input	Property Size
feature input	`NumChannels`-by-1
vector sequence input	`NumChannels`-by-1
1-D image input	1-by-`NumChannels`
1-D image sequence input	1-by-`NumChannels`
2-D image input	1-by-1-by-`NumChannels`
2-D image sequence input	1-by-1-by-`NumChannels`
3-D image input	1-by-1-by-1-by-`NumChannels`
3-D image sequence input	1-by-1-by-1-by-`NumChannels`

Data Types: single | double

`Offset` — Channel offsets
`[]` (default) | numeric array

Channel offsets β, specified as a numeric vector.

The channel offsets are learnable parameters. When you train a network using the trainnet function or initialize a dlnetwork object, if Offset is nonempty, then the software uses the Offset property as the initial value. If Offset is empty, then the software uses the initializer specified by OffsetInitializer.

Depending on the type of layer input, the trainnet and dlnetwork functions automatically reshape this property to have of the following sizes:

Layer Input	Property Size
feature input	`NumChannels`-by-1
vector sequence input	`NumChannels`-by-1
1-D image input	1-by-`NumChannels`
1-D image sequence input	1-by-`NumChannels`
2-D image input	1-by-1-by-`NumChannels`
2-D image sequence input	1-by-1-by-`NumChannels`
3-D image input	1-by-1-by-1-by-`NumChannels`
3-D image sequence input	1-by-1-by-1-by-`NumChannels`

Data Types: single | double

`MeanDecay` — Decay value for moving mean computation
0.1 (default) | numeric scalar between `0` and `1`

Decay value for the moving mean computation, specified as a numeric scalar between 0 and 1.

When you use the trainNetwork or trainnet function and the BatchNormalizationStatistics training option is 'moving', at each iteration, the layer updates the moving mean value using

$μ^{*} = λ_{μ} \hat{μ} + (1 - λ_{μ}) μ,$

where $μ^{*}$ denotes the updated mean, $λ_{μ}$ denotes the mean decay value, $\hat{μ}$ denotes the mean of the layer input, and $μ$ denotes the latest value of the moving mean value.

When you use the trainNetwork or trainnet function and the BatchNormalizationStatistics training option is 'population', this option has no effect.

`VarianceDecay` — Decay value for moving variance computation
0.1 (default) | numeric scalar between `0` and `1`

Decay value for the moving variance computation, specified as a numeric scalar between 0 and 1.

When you use the trainNetwork or trainnet function and the BatchNormalizationStatistics training option is 'moving', at each iteration, the layer updates the moving variance value using

$σ^{2}^{*} = λ_{σ^{2}} \hat{σ^{2}} + (1 - λ_{σ^{2}}) σ^{2},$

where $σ^{2}^{*}$ denotes the updated variance, $λ_{σ^{2}}$ denotes the variance decay value, $\hat{σ^{2}}$ denotes the variance of the layer input, and $σ^{2}$ denotes the latest value of the moving variance value.

When you use the trainNetwork or trainnet function and the BatchNormalizationStatistics training option is 'population', this option has no effect.

Learning Rate and Regularization

`ScaleLearnRateFactor` — Learning rate factor for scale factors
`1` (default) | nonnegative scalar

Learning rate factor for the scale factors, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the scale factors in a layer. For example, if ScaleLearnRateFactor is 2, then the learning rate for the scale factors in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions function.

`OffsetLearnRateFactor` — Learning rate factor for offsets
`1` (default) | nonnegative scalar

Learning rate factor for the offsets, specified as a nonnegative scalar.

The software multiplies this factor by the global learning rate to determine the learning rate for the offsets in a layer. For example, if OffsetLearnRateFactor is 2, then the learning rate for the offsets in the layer is twice the current global learning rate. The software determines the global learning rate based on the settings specified with the trainingOptions function.

`ScaleL2Factor` — L₂ regularization factor for scale factors
`1` (default) | nonnegative scalar

L₂ regularization factor for the scale factors, specified as a nonnegative scalar.

The software multiplies this factor by the global L₂ regularization factor to determine the learning rate for the scale factors in a layer. For example, if ScaleL2Factor is 2, then the L₂ regularization for the offsets in the layer is twice the global L₂ regularization factor. You can specify the global L₂ regularization factor using the trainingOptions function.

`OffsetL2Factor` — L₂ regularization factor for offsets
`1` (default) | nonnegative scalar

L₂ regularization factor for the offsets, specified as a nonnegative scalar.

The software multiplies this factor by the global L₂ regularization factor to determine the learning rate for the offsets in a layer. For example, if OffsetL2Factor is 2, then the L₂ regularization for the offsets in the layer is twice the global L₂ regularization factor. You can specify the global L₂ regularization factor using the trainingOptions function.

Layer

`Name` — Layer name
`""` (default) | character vector | string scalar

Layer name, specified as a character vector or string scalar. For Layer array input, the trainnet and dlnetwork functions automatically assign names to layers with the name "".

The BatchNormalizationLayer object stores this property as a character vector.

Data Types: char | string

`NumInputs` — Number of inputs
`1` (default)

This property is read-only.

Number of inputs to the layer, returned as 1. This layer accepts a single input only.

Data Types: double

`InputNames` — Input names
`{'in'}` (default)

This property is read-only.

Input names, returned as {'in'}. This layer accepts a single input only.

Data Types: cell

`NumOutputs` — Number of outputs
`1` (default)

This property is read-only.

Number of outputs from the layer, returned as 1. This layer has a single output only.

Data Types: double

`OutputNames` — Output names
`{'out'}` (default)

This property is read-only.

Output names, returned as {'out'}. This layer has a single output only.

Data Types: cell

Examples

collapse all

Create Batch Normalization Layer

Open Live Script

Create a batch normalization layer with the name BN1.

layer = batchNormalizationLayer(Name="BN1")

layer = 
  BatchNormalizationLayer with properties:

               Name: 'BN1'
        NumChannels: 'auto'

   Hyperparameters
          MeanDecay: 0.1000
      VarianceDecay: 0.1000
            Epsilon: 1.0000e-05

   Learnable Parameters
             Offset: []
              Scale: []

   State Parameters
        TrainedMean: []
    TrainedVariance: []

Use properties method to see a list of all properties.

Include batch normalization layers in a Layer array.

layers = [
    imageInputLayer([32 32 3]) 
  
    convolution2dLayer(3,16,Padding=1)
    batchNormalizationLayer
    reluLayer   
    
    maxPooling2dLayer(2,Stride=2)
    
    convolution2dLayer(3,32,Padding=1)
    batchNormalizationLayer
    reluLayer
          
    fullyConnectedLayer(10)
    softmaxLayer
    ]

layers = 
  10x1 Layer array with layers:

     1   ''   Image Input           32x32x3 images with 'zerocenter' normalization
     2   ''   2-D Convolution       16 3x3 convolutions with stride [1  1] and padding [1  1  1  1]
     3   ''   Batch Normalization   Batch normalization
     4   ''   ReLU                  ReLU
     5   ''   2-D Max Pooling       2x2 max pooling with stride [2  2] and padding [0  0  0  0]
     6   ''   2-D Convolution       32 3x3 convolutions with stride [1  1] and padding [1  1  1  1]
     7   ''   Batch Normalization   Batch normalization
     8   ''   ReLU                  ReLU
     9   ''   Fully Connected       10 fully connected layer
    10   ''   Softmax               softmax

Algorithms

expand all

Batch Normalization Layer

The layer first normalizes the activations of each channel by subtracting the mini-batch mean and dividing by the mini-batch standard deviation. Then, the layer shifts the input by a learnable offset β and scales it by a learnable scale factor γ. β and γ are themselves learnable parameters that are updated during network training.

Batch normalization layers normalize the activations and gradients propagating through a neural network, making network training an easier optimization problem. To take full advantage of this fact, you can try increasing the learning rate. Since the optimization problem is easier, the parameter updates can be larger and the network can learn faster. You can also try reducing the L₂ and dropout regularization. With batch normalization layers, the activations of a specific image during training depend on which images happen to appear in the same mini-batch. To take full advantage of this regularizing effect, try shuffling the training data before every training epoch. To specify how often to shuffle the data during training, use the 'Shuffle' name-value pair argument of trainingOptions.

The batch normalization operation normalizes the elements x_i of the input by first calculating the mean μ_B and variance σ_B² over the spatial, time, and observation dimensions for each channel independently. Then, it calculates the normalized activations as

$\hat{x_{i}} = \frac{x_{i} - μ_{B}}{\sqrt{σ_{B}^{2} + ϵ}},$

where ϵ is a constant that improves numerical stability when the variance is very small.

To allow for the possibility that inputs with zero mean and unit variance are not optimal for the operations that follow batch normalization, the batch normalization operation further shifts and scales the activations using the transformation

$y_{i} = γ {\hat{x}}_{i} + β,$

where the offset β and scale factor γ are learnable parameters that are updated during network training.

To make predictions with the network after training, batch normalization requires a fixed mean and variance to normalize the data. This fixed mean and variance can be calculated from the training data after training, or approximated during training using running statistic computations.

The layer uses TrainedMean and TrainedVariance to normalize the input during prediction.

Layer Input and Output Formats

Layers in a layer array or layer graph pass data to subsequent layers as formatted dlarray objects. The format of a dlarray object is a string of characters in which each character describes the corresponding dimension of the data. The formats consist of one or more of these characters:

"S" — Spatial
"C" — Channel
"B" — Batch
"T" — Time
"U" — Unspecified

For example, you can describe 2-D image data that is represented as a 4-D array, where the first two dimensions correspond to the spatial dimensions of the images, the third dimension corresponds to the channels of the images, and the fourth dimension corresponds to the batch dimension, as having the format "SSCB" (spatial, spatial, channel, batch).

You can interact with these dlarray objects in automatic differentiation workflows, such as those for developing a custom layer, using a functionLayer object, or using the forward and predict functions with dlnetwork objects.

This table shows the supported input formats of BatchNormalizationLayer objects and the corresponding output format. If the software passes the output of the layer to a custom layer that does not inherit from the nnet.layer.Formattable class, or a FunctionLayer object with the Formattable property set to 0 (false), then the layer receives an unformatted dlarray object with dimensions ordered according to the formats in this table. The formats listed here are only a subset. The layer may support additional formats such as formats with additional "S" (spatial) or "U" (unspecified) dimensions.

Input Format	Output Format
`"CB"` (channel, batch)	`"CB"` (channel, batch)
`"SCB"` (spatial, channel, batch)	`"SCB"` (spatial, channel, batch)
`"SSCB"` (spatial, spatial, channel, batch)	`"SSCB"` (spatial, spatial, channel, batch)
`"SSSCB"` (spatial, spatial, spatial, channel, batch)	`"SSSCB"` (spatial, spatial, spatial, channel, batch)
`"CBT"` (channel, batch, time)	`"CBT"` (channel, batch, time)
`"SCBT"` (spatial, channel, batch, time)	`"SCBT"` (spatial, channel, batch, time)
`"SSCBT"` (spatial, spatial, channel, batch, time)	`"SSCBT"` (spatial, spatial, channel, batch, time)
`"SSSCBT"` (spatial, spatial, spatial, channel, batch, time)	`"SSSCBT"` (spatial, spatial, spatial, channel, batch, time)
`"CU"` (channel, unspecified)	`"CU"` (channel, unspecified)
`"SC"` (spatial, channel)	`"SC"` (spatial, channel)
`"SSC"` (spatial, spatial, channel)	`"SSC"` (spatial, spatial, channel)
`"SSSC"` (spatial, spatial, spatial, channel)	`"SSSC"` (spatial, spatial, spatial, channel)

In dlnetwork objects, BatchNormalizationLayer objects also support these input and output format combinations.

Input Format	Output Format
`"CT"` (channel, time)	`"CT"` (channel, time)
`"SCT"` (spatial, channel, time)	`"SCT"` (spatial, channel, time)
`"SSCT"` (spatial, spatial, channel, time)	`"SSCT"` (spatial, spatial, channel, time)
`"SSSCT"` (spatial, spatial, spatial, channel, time)	`"SSSCT"` (spatial, spatial, spatial, channel, time)

References

[1] Ioffe, Sergey, and Christian Szegedy. “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift.” Preprint, submitted March 2, 2015. https://arxiv.org/abs/1502.03167.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.

GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Version History

Introduced in R2017b

expand all

R2023a: `Epsilon` supports values less than `1e-5`

The Epsilon option also supports positive values less than 1e-5.

batchNormalizationLayer

Description

Creation

Syntax

Description

Properties

Batch Normalization

TrainedMean — Mean statistic used for prediction numeric vector

TrainedVariance — Variance statistic used for prediction numeric vector

Epsilon — Constant to add to mini-batch variances 1e-5 (default) | positive scalar

NumChannels — Number of input channels "auto" (default) | positive integer

Parameters and Initialization

ScaleInitializer — Function to initialize channel scale factors 'ones' (default) | 'narrow-normal' | function handle

OffsetInitializer — Function to initialize channel offsets 'zeros' (default) | 'ones' | 'narrow-normal' | function handle

Scale — Channel scale factors [] (default) | numeric array

Offset — Channel offsets [] (default) | numeric array

MeanDecay — Decay value for moving mean computation 0.1 (default) | numeric scalar between 0 and 1

VarianceDecay — Decay value for moving variance computation 0.1 (default) | numeric scalar between 0 and 1

Learning Rate and Regularization

ScaleLearnRateFactor — Learning rate factor for scale factors 1 (default) | nonnegative scalar

OffsetLearnRateFactor — Learning rate factor for offsets 1 (default) | nonnegative scalar

ScaleL2Factor — L2 regularization factor for scale factors 1 (default) | nonnegative scalar

OffsetL2Factor — L2 regularization factor for offsets 1 (default) | nonnegative scalar

Layer

Name — Layer name "" (default) | character vector | string scalar

NumInputs — Number of inputs 1 (default)

InputNames — Input names {'in'} (default)

NumOutputs — Number of outputs 1 (default)

OutputNames — Output names {'out'} (default)

Examples

Create Batch Normalization Layer

Algorithms

Batch Normalization Layer

Layer Input and Output Formats

References

Extended Capabilities

C/C++ Code Generation Generate C and C++ code using MATLAB® Coder™.

GPU Code Generation Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

Version History

R2023a: Epsilon supports values less than 1e-5

See Also

Topics

`TrainedMean` — Mean statistic used for prediction
numeric vector

`TrainedVariance` — Variance statistic used for prediction
numeric vector

`Epsilon` — Constant to add to mini-batch variances
`1e-5` (default) | positive scalar

`NumChannels` — Number of input channels
`"auto"` (default) | positive integer

`ScaleInitializer` — Function to initialize channel scale factors
`'ones'` (default) | `'narrow-normal'` | function handle

`OffsetInitializer` — Function to initialize channel offsets
`'zeros'` (default) | `'ones'` | `'narrow-normal'` | function handle

`Scale` — Channel scale factors
`[]` (default) | numeric array

`Offset` — Channel offsets
`[]` (default) | numeric array

`MeanDecay` — Decay value for moving mean computation
0.1 (default) | numeric scalar between `0` and `1`

`VarianceDecay` — Decay value for moving variance computation
0.1 (default) | numeric scalar between `0` and `1`

`ScaleLearnRateFactor` — Learning rate factor for scale factors
`1` (default) | nonnegative scalar

`OffsetLearnRateFactor` — Learning rate factor for offsets
`1` (default) | nonnegative scalar

`ScaleL2Factor` — L₂ regularization factor for scale factors
`1` (default) | nonnegative scalar

`OffsetL2Factor` — L₂ regularization factor for offsets
`1` (default) | nonnegative scalar

`Name` — Layer name
`""` (default) | character vector | string scalar

`NumInputs` — Number of inputs
`1` (default)

`InputNames` — Input names
`{'in'}` (default)

`NumOutputs` — Number of outputs
`1` (default)

`OutputNames` — Output names
`{'out'}` (default)

C/C++ Code Generation
Generate C and C++ code using MATLAB® Coder™.

GPU Code Generation
Generate CUDA® code for NVIDIA® GPUs using GPU Coder™.

R2023a: `Epsilon` supports values less than `1e-5`