Importing large CSVs and consolidating in one matrix.

Question

Pranjal 2015-1-12

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/169642-importing-large-csvs-and-consolidating-in-one-matrix

评论： Titus Edelhofer 2015-1-12

Hello I need to consolidate around 800 CSVs(total size ~ 6GB) into 1 matrix. I need to process(mainly filtering) the data and plot the graphs. What should be the approach of the task so that the processing time in loading the CSVs and plotting takes minimum time.

TIA

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Titus Edelhofer 2015-1-12

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/169642-importing-large-csvs-and-consolidating-in-one-matrix#answer_164643

Hi Pranjal,

if you have R2014b available, then datastore + mapreduce would be the choice of tools.

Titus

2 个评论
显示无隐藏无

Pranjal 2015-1-12

Hi Titus,

Thanks for the answer,but I am working on R2011b. Can you suggest any approach using that version of matlab.

Pranjal

Titus Edelhofer 2015-1-12

In this case use textscan for reading the data. Each file is probably not too large (assuming the 800 csv are somewhat similar in size), i.e., about 7-10 MByte. Therefore reading each file in one sweep should be no problem.

You might use the Parallel Computing Toolbox (parfor) to read the files in parallel. Either process the data right away or save to .mat so that you can read them later faster in a loop.

Titus

请先登录，再进行评论。