Why does layerNormalizationLayer in Deep Learning Toolbox include T dimension into the batch?

1 次查看（过去 30 天）
John Smith
John Smith 2023-3-13
回答： John Smith 2023-3-24
Hello,
While implementing a ViT transformer in Matlab, I found at that the layerNormalizationLayer does include the T dimension in the statistics calculated for each sample in the batch. This is problematics when implementing a transformer, since tokens correspond to the T dimension and reference implementations calculate the statistics separately for each token.
Thx

请先登录，再回答此问题。

采纳的回答

John Smith
John Smith 2023-3-24
It seems Mathworks have listened and changed the behavior of layerNormalizationLayer in R2023a.:
https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.layernormalizationlayer.html
Starting in R2023a, by default, the layer normalizes sequence data over the channel and spatial dimensions. In previous versions, the software normalizes over all dimensions except for the batch dimension (the spatial, time, and channel dimensions). Normalization over the channel and spatial dimensions is usually better suited for this type of data. To reproduce the previous behavior, set OperationDimension to "batch-excluded".

更多回答（1 个）

Matt J
Matt J 2023-3-13
Perhaps you can fold your T dimension into the C dimension and use a groupNormalizationLayer instead, with the groups defined so that different T belong to different groups.
  7 个评论
显示 5更早的评论
John Smith
John Smith 2023-3-15
Perhaps lamenting would cause someone from Mathworks to take notice and add the capability to the code base. Sigh ...

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by