Is it possible to implement a LSTM layer after a CNN layer?

Question

Sofía 2018-4-26

2
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/397582-is-it-possible-to-implement-a-lstm-layer-after-a-cnn-layer

评论： krishna Chauhan 2020-6-26

I'm trying to implement a CNN layer + a LSTM layer, but I have an error: "Network: Incompatible layer types". Is it not possible to implement this combination in MATLAB or am I just writing it not properly?

My code:

layers = [ ...
    sequenceInputLayer(inputSize)    
      convolution2dLayer(3,8,'Padding','same')
      batchNormalizationLayer
      reluLayer 
      maxPooling2dLayer(2,'Stride',2)
      lstmLayer(numHiddenUnits,'OutputMode','last')
      fullyConnectedLayer(numClasses)
      softmaxLayer
      classificationLayer
      ];

Error:

Error using trainNetwork (line 154)
Invalid network.
Caused by:
    Network: Incompatible layer types. The network contains layer types not supported with recurrent layers.
    Detected recurrent layers:
        layer 6 (LSTM)
    Detected incompatible layers:
        layer 2 (Convolution)
        layer 3 (Batch Normalization)
        layer 5 (Max Pooling)
    Layer 2: Input size mismatch. Size of input to this layer is different from the expected input size.
    Inputs to this layer:
        from layer 1 (output size 500)

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Jake 2018-10-9

I would like to do this as well.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Mona 2018-9-19

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/397582-is-it-possible-to-implement-a-lstm-layer-after-a-cnn-layer#answer_337397

As far as I know, no, you can't combine the two. You can train a CNN independently on your training data, then use the learned features as an input to your LSTM. However, learning and updating CNN weights while training an LSTM is unfortunately not possible.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

krishna Chauhan 2020-6-26

Maam can i store the weights after say a number of epochs of CNN and then use those weights as input to LSTM?

请先登录，再进行评论。

Answer 2

charu 2018-7-9

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/397582-is-it-possible-to-implement-a-lstm-layer-after-a-cnn-layer#answer_328028

在 MATLAB Online 中打开

use bilstmLayer layer instead of lstm layer as in example

 inputSize = 12;
numHiddenUnits = 100;
numClasses = 9;
layers = [ ...
      sequenceInputLayer(inputSize)
      bilstmLayer(numHiddenUnits,'OutputMode','last')
      fullyConnectedLayer(numClasses)
      softmaxLayer
      classificationLayer]

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Guillaume JUBIEN 2018-9-3

在 MATLAB Online 中打开

I have the same problem by using a bilstm Layer. The error message is :

if true
Error using trainNetwork (line 154)
Invalid network.
Error in test_spa_REG (line 168)
net = trainNetwork(XTR,TTR,Layers,options);
Caused by:
  Network: Incompatible layer types. The network contains layer types not supported with recurrent layers.
  Detected recurrent layers:
      layer 9 (BiLSTM)
  Detected incompatible layers:
      layer 1 (Image Input)
      layer 2 (Transposed Convolution)
      layer 'temp1' (Convolution)
      layer 5 (Average Pooling)
      and 1 other layers.
  Layer 10: Input size mismatch. Size of input to this layer is different from the expected input size.
  Inputs to this layer:
      from layer 9 (output size 20)

Is it possible to combine CNN with LSTM layer ?

请先登录，再进行评论。

Answer 3

Shounak Mitra 2019-7-11

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/397582-is-it-possible-to-implement-a-lstm-layer-after-a-cnn-layer#answer_382864

Hello Everyone,

As of 19a, MATLAB supports workflows containing both CNN and LSTM layers.

Please check the link that contains an example showing the CNN+LSTM workflow --> https://www.mathworks.com/help/deeplearning/examples/classify-videos-using-deep-learning.html

2 个评论
显示无隐藏无

Bhavna Rajasekaran 2019-11-8

编辑：Bhavna Rajasekaran 2019-11-8

Is it possible to implement LSTM regression on an image (N-by-M array) such that the output is also a 2-dimesional array? Which means that the Predictors are an N-by-M array of sequences?

suraj sahoo 2019-11-11