How to use the LSTM network to forecast the image？

58 次查看（过去 30 天）
Ning Liu
Ning Liu 2021-5-22
评论： Dieter Mayer 2022-10-21
Hi, I have image time series datasets and each image size is 785*785*3, the time series length is 400. Now I want to establish a LSTM network to fit , is the image at time t and is the image at time t-1. I use the previrous 350 images to prepare the train data and the last 50 images to test the forecast results. I construct a simple LSTM network as follows,
layers = [ ...
sequenceInputLayer([785,785,3],'Name','input')
flattenLayer('Name','flatten')
lstmLayer(200,'OutputMode','sequence','Name','lstm1')
lstmLayer(200,'OutputMode','sequence','Name','lstm2')
fullyConnectedLayer(785*785*3,'Name','fc')
regressionLayer('routput')];
options = trainingOptions('adam', ...
'MaxEpochs',300, ...
'GradientThreshold',1, ...
'InitialLearnRate',1e-4, ...
'LearnRateSchedule','piecewise', ...
'LearnRateDropPeriod',125, ...
'LearnRateDropFactor',0.2, ...
'Verbose',0, ...
'Plots','training-progress');
% X_train is a cell with length of 350, and each cell element is a [785,785,3] double matrix
% y_train is a cell with length of 350, and each cell element is a [785*785*3,1] double matrix
net = trainNetwork(X_train,y_train,layers,options);
for i=1:length(X_train)
[net,YPred] = predictAndUpdateState(net,X_train{i},'ExecutionEnvironment','cpu');
end
% forecast
Ypred = zeros(785,785,3,50);
[net,YPred] = predictAndUpdateState(net,YPred,'ExecutionEnvironment','cpu');
Ypred(:,:,:,1) = reshape(YPred,785,785,3);
for i=2:length(X_test)
[net,YPred] = predictAndUpdateState(net,Ypred(:,:,:,i-1),'ExecutionEnvironment','cpu');
Ypred(:,:,:,i) = reshape(YPred,785,785,3);
end
I compare the forecast result Ypred with my test data X_test, I got an undesirable compare result. I think the reason is I designed an inappropriate LSTM network. Any comments and suggestions are welcome! Thanks a lot!
  1 个评论
Dieter Mayer
Dieter Mayer 2022-10-21
Hi,
What you need is the use of a real convolutional LSTM (conv-LSTM). This is not a series of a convolutional layer, followed by an LSTM layer but a LSTM which internally uses convolutions instead of matrix-multiplications (see also Link)
A conv-LSTM recognizes and preserves spatial patterns, otherwise, each pixel is handled individually.
It seems that matlab does not offer conv-LSTMs yet, I really hope, that this type of layer is in development.
Another approach is to use the UNET-architecture to make image sequence to image sequence forecasts. Unets can be build up in matlab using convolutions (encode), transposed convolutions (decode) and concatenation layers (are connecting kernels between encoder and decoder blocks).

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Prateek Rai
Prateek Rai 2021-9-13
编辑：Prateek Rai 2021-9-13
To my understanding, you are trying to use the LSTM Network to forecast the image. A possible workaround could be :
(a) Use CNN Layers to extract features from image. The final result of Convolution must be a 1-D Vector.
(b) Use this 1-D vector to feed it to LSTM which will again output a new 1-D Vector.
(c) Output of LSTM can be feeded to a set of transposed convolutional layers to retrieve the output image.
You can refer to Convolutional Neural Network MathWorks documentation page to find more on CNN and transposedConv2dLayer MathWorks documentation page to find more on transposed Convolutional layers. You can also refer to Long Short-Term Memory Networks MathWorks documentation page to find more on LSTM.

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by