Why doesn't concatLayer in Deep Learning Toolbox concatenate the 'T' dimension?

3 次查看（过去 30 天）
John Smith
John Smith 2023-3-13
评论： Artem Lensky 2023-8-19
Hello,
While implementing a ViT transformer in Matlab, I found at that the concatLayer does not concatenate over the T dimension. This is needed to concatenate the class token with patch tokens, since the natural representation is CBT with C corresponding to features, B to batch and T to token within a batch (this is also the canonical representation in the attention function).
It's possible to work around this by hacking to e.g. SCB, but then other problems pop up which also need to be hacked around.
Thx

请先登录，再回答此问题。

采纳的回答

Ben
Ben 2023-3-14
You can create a layer that concatenates on the T dimension with functionLayer
sequenceCatLayer = functionLayer(@(x,y) cat(3,x,y));
This will work in dlnetwork to concatenate two CBT dlarray-s.
Since you're concatenating the class token, it might also be worth considering creating a custom layer that has the class token embedding as a Learnable property, and performs the concatenation in the predict method.
  3 个评论
显示 1更早的评论
Catalytic
Catalytic 2023-3-23
编辑：Catalytic 2023-3-23
@John Smith - Since Ben's answer yielded a solution for you, you should hit the Accept this Answer button, and likewise with other answers you might not have accepted.
Artem Lensky
Artem Lensky 2023-8-19
Are there any plans to make concatenationLayer support concatetnation along the T dimension?

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

产品


版本

R2022b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by