Dealing with Large training datasets saved in a number of .mat files

Question

Ruohao Zhang 2020-8-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/577669-dealing-with-large-training-datasets-saved-in-a-number-of-mat-files

评论： Ruohao Zhang 2020-8-21

Hello all,

I have run into a problem where I need to train a LSTM signal classifier with huge amount of data.

Each 1D signal is around 100k samples, every 48 signals are saved in a .mat file. The total number of .mat files is around 2000.

The labels are similarly saved in corresponding .mat files in a different folder.

I would like to know if there's a way to train the network without the necessity of loading the whole thing into memory. (with 64GB ram I can only load ~1300files at once)

Your help will be very much appreciated.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Divya Gaddipati 2020-8-13

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/577669-dealing-with-large-training-datasets-saved-in-a-number-of-mat-files#answer_479088

You can use fileDatastore for this purpose.

trainData = fileDatastore('/path/to/data', 'ReadFcn', @load, 'FileExtensions', '.mat');

You can either use "load" or your own custom function defining how to load the data.

You can also refer to this link for more information on training LSTM while loading data using fileDatastore.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Ruohao Zhang 2020-8-18

Thank you for this answer, what I ended up doing is to decompose all the files that I currently have into smaller files each containing only one matrix and create a datastore from there. The problem I encountered is the loading function is acting weird when you call it in filedatastore. when I load the .mat in the command it returns as a matrix where when it is loaded in by calling filedatastore and combine, it loaded as a 1x1 cell. So I have to write my own load func to make sure the correct format

请先登录，再进行评论。

Answer 2

Frantz Bouchereau 2020-8-20

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/577669-dealing-with-large-training-datasets-saved-in-a-number-of-mat-files#answer_482768

编辑：Frantz Bouchereau 2020-8-20

Ruohao,

You can use two signalDatastores - one to read your signal files and another one to read your labels. You can then combine them using combine(), split the combinedDatastore into training and test sets using subset() and then feeding the combined datastores into the training function of the LSTM network.

With signslDatastore you do not need to write a load function. You specify the variable names you want read from the mat file and those are returned at every read.

HTH

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Ruohao Zhang 2020-8-21

Thank you for your reply. Indeed, signaldatastores can be really helpful, but sadly this function is only introduced in 2020a where I am still using 2019b. It can be a good motivation to update my software now:)

请先登录，再进行评论。