Creating and using Datastore for LSTM time sequence data

Question

Narayan 2024-2-11

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2080776-creating-and-using-datastore-for-lstm-time-sequence-data

评论： Narayan 2024-2-12

I have time sequence data files more than 10000 numbers stored individually at csv files. Each sequence data file consists of a sample of data from 6300 features taken at 5 time sequences. Each column is a measurement data from a feature. The labels are stored in separate file sequencially.

-0.7 -1.7 -5.09 -4.79 ....

-1.06 -1.59 -5.08 -4.76 .....

-1.42 -1.86 -5.61 -4.86 ....

-1.34 -2.01 -5.1 -4.62 .....

numFeatures= 6300;
numHiddenUnits = 100;
numClasses = 3;
layers = [ ...
    sequenceInputLayer(numFeatures)
    lstmLayer(numHiddenUnits,'OutputMode','last')
    fullyConnectedLayer(numClasses)
    softmaxLayer
    classificationLayer];
options = trainingOptions('adam', ...
    'MiniBatchSize',20,...
    'MaxEpochs',10, ...
    'Shuffle','once',...
    'GradientThreshold',0.001, ...
    'Verbose',1, ...
    'Plots','training-progress');

I want to use the data for LSTM classification. I could not load all the data for training purpose.

Matlab asks for cell data for each time sequence sample data for training.

So, How can I load the files and train the network using the datastore for such large data?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Angelo Yeo 2024-2-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2080776-creating-and-using-datastore-for-lstm-time-sequence-data#answer_1406511

tabularTextDatastore supports to manage a large set of "csv" files. To quote from the doc:

Use a TabularTextDatastore object to manage large collections of text files containing column-oriented or tabular data where the collection does not necessarily fit in memory.