Load data one file at time for CNN training

Question

Sunny 2019-1-8

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/438824-load-data-one-file-at-time-for-cnn-training

回答： Vishal Bhutani 2019-1-11

I have dataset with 4700 images with dimensions 2000*102. All these images are converted to seperate CSV files (pixel values). I think I can't load all these csv files at a time as this will crash my RAM (32GB) when the network starts to train. Is it possible in Matlab to access each image(.csv) file during training and testing from storage location rather than placing it on RAM. This is to train and test CNN in deep learning extension.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Vishal Bhutani 2019-1-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/438824-load-data-one-file-at-time-for-cnn-training#answer_355953

Try the solutions mentioned in the link attached below. It contains solution how to access data from CSV file directly when size of CSV file is large.

https://www.mathworks.com/matlabcentral/answers/231857-loading-large-csv-files

https://www.mathworks.com/matlabcentral/answers/71837-way-of-conserving-memory-when-extracting-data-from-csv

Hope it helps.