Reading huge number of files quickly (I/O)

Question

Vishnu Swaroop 2012-6-4

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/40189-reading-huge-number-of-files-quickly-i-o

For analysis of EOD data for stocks, I need to read 3650 .csv files, but using a for loop takes around 30 seconds for 15 files, which would mean around 2 hours just to read data. Is there any way I can do it faster?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2012-6-4

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/40189-reading-huge-number-of-files-quickly-i-o#answer_49836

Are you currently reading using csvread(), or import(), or load() ?

If each file has exactly the same structure (same number of fields, some number of headers, same fields are character vs numeric) then you might be able to do better by using textscan()

If you are currently using xlsread() on MS Windows then you can almost certainly do better than you are doing now (but textscan() might be better still). xlsread() opens and closes Excel each time; Image Analyst has posted code several times showing how to open Excel just once and do several operations with it.