Does the selfattentionLayer also perform softmax and scaling?

Question

Chih 2023-4-3

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1940374-does-the-selfattentionlayer-also-perform-softmax-and-scaling

编辑： cui,xingxing 2024-4-27

采纳的回答： Rohit

In https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.selfattentionlayer.html, it states that:

A self-attention layer computes single-head or multihead self-attention of its input.

The layer:

Computes the queries, keys, and values from the input
Computes the scaled dot-product attention across heads using the queries, keys, and values
Merges the results from the heads
Performs a linear transformation on the merged result

I wonder if the layer also apply softmax to the scaling (i.e. divide (Q*K) by sqrt(dim))? My understanding is that, within step 2, this softmax and scaling should happen.

Please clarify that for me or more general users.

Thanks.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Rohit 2023-4-20

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1940374-does-the-selfattentionlayer-also-perform-softmax-and-scaling#answer_1219478

I understand that you want to know whether ‘selfAttentionLayer’ performs softmax and scaling operations which are involved to compute attention score.

Yes, we perform both operations to compute scaled attention score and then apply softmax as required in attention mechanism.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Chih 2023-4-20

Thank you very much, Rohit.

请先登录，再进行评论。

Answer 2

cui,xingxing 2024-1-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1940374-does-the-selfattentionlayer-also-perform-softmax-and-scaling#answer_1387576

编辑：cui,xingxing 2024-4-27

Hi,@Chih

Please check out the details of the code I wrote here link.

-------------------------Off-topic interlude, 2024-------------------------------

I am currently looking for a job in the field of CV algorithm development, based in Shenzhen, Guangdong, China,or a remote support position. I would be very grateful if anyone is willing to offer me a job or make a recommendation. My preliminary resume can be found at: https://cuixing158.github.io/about/ . Thank you!

Email: cuixingxing150@gmail.com