selfAttentionLayer can't process sequence-to-label problem?

2024 1 5
1 个回答
回答已采纳
更新时间：2024 4 27
10 次查看（30 天）
Ran in:

1 个投票

Ran in:
selfAttentionLayer why can't handle the following simple sequence classification problem, already through the flattenLayer into one-dimensional data, on the contrary, lstm specify "outputMode" as "last" will pass.
% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
flattenLayer;
selfAttentionLayer(6,48);
% lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
layerNormalizationLayer;
fullyConnectedLayer(numClasses);
softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
MaxEpochs=1, ...
InitialLearnRate=0.01,...
Shuffle="every-epoch", ...
GradientThreshold=1, ...
Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);
Error using trainnet
Number of observations in predictors (1000) and targets (1) must match. Check that the data and network are consistent.

请先登录，再回答此问题。

 采纳的回答

xingxingcui
xingxingcui 2024-1-7
编辑：xingxingcui 2024-4-27
Ran in:

0 个投票

Ran in:
In terms of the output feature map dimensions, there is a time "T" dimension that has to be eliminated in order to match the output dimensions, which can usually be done by indexing1dLayer. So the layers array is added before the fullyConnectedLayer.
% Here use simple data, for demonstration purposes only
XTrain = rand(3,200,1000); % dims "CTB"
TTrain = categorical(randi(4,1000,1));
% define my layers
numClasses = numel(categories(TTrain));
layers = [inputLayer(size(XTrain),"CTB");
flattenLayer;
selfAttentionLayer(6,48);
% lstmLayer(20,OutputMode="last"); % use lstmLayer is ok!
layerNormalizationLayer;
indexing1dLayer; % Add this!!!
fullyConnectedLayer(numClasses);
softmaxLayer];
net = dlnetwork(layers);
% train network
lossFcn = "crossentropy";
options = trainingOptions("adam", ...
MaxEpochs=1, ...
InitialLearnRate=0.01,...
Shuffle="every-epoch", ...
GradientThreshold=1, ...
Verbose=true);
netTrained = trainnet(XTrain,TTrain,net,lossFcn,options);
Iteration Epoch TimeElapsed LearnRate TrainingLoss _________ _____ ___________ _________ ____________ 1 1 00:00:02 0.01 1.5374 7 1 00:00:06 0.01 1.5272 Training stopped: Max epochs completed
-------------------------Off-topic interlude-------------------------------
I am currently looking for a job in the field of CV algorithm development, based in Shenzhen, Guangdong, China. I would be very grateful if anyone is willing to offer me a job or make a recommendation. My preliminary resume can be found at: https://cuixing158.github.io/about/ . Thank you!
Email: cuixingxing150@gmail.com

5 个评论
显示 3更早的评论 隐藏 3更早的评论

Imrana
Imrana 2024-1-22
Hello Cui,
Can you ellobrate it little more, I am stucked in same kind of a problem but is time series forecasting. I have to use lstm to irregular time series data. How to use tspan and put mask on them in self attention layer?
Best Regards,
xingxingcui
xingxingcui 2024-1-22
Hi,@Imrana
you can use analyzeNetwork function to visual network.
Look at the part I've circled in red. The T-dimension is eliminated before you can pick up the next layers.
DGM
DGM 2024-3-5
Posted as a comment-as-flag by chang gao:
Useful answer.

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

产品

版本

R2023b

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by