Why does Matlab need so much memory to read in a .csv file?

Question

Randy 2013-9-21

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/87860-why-does-matlab-need-so-much-memory-to-read-in-a-csv-file

I have a 370mb .csv file that I want to read in to Matlab. I cannot do it, because the program sequesters all of my available memory (approximately 5gb) and is still not done with the import. This is as large file, but it is not that big. Is this normal?

2 个评论
显示无隐藏无

Matt J 2013-9-22

It doesn't sound normal. Your code, if you show it, could tell us why.

Randy 2013-9-22

My "code" consists of hitting "import data", browsing to the file, selecting the vectors I want from the preview, and hitting "import selection". I do not know what code this runs (it does not show up in the command history).

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2013-9-22

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/87860-why-does-matlab-need-so-much-memory-to-read-in-a-csv-file#answer_97471

Just a guess, but if your file is full of numeric data, each data value can become about 8 times as large when converted from text to doubles. Seeing your actual code would provide more clues, though, as I commented above.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Matt J 2013-9-23

编辑：Matt J 2013-9-23

If your file is 370MB and contains 20 million elements, it means that each element consumes about 18 bytes on average. That sounds a bit weird to me. Are these 18 digit numbers?

Anyway, you could try xlsread, dlmread, or textscan to see if you can read 1 column at a time and see if it makes a difference.

Image Analyst 2013-9-23

18 bytes per number is fine. Each number could be 18 characters if it had, say for example, 3 numbers, a decimal place, 12 digits of mantissa, a comma, and a trailing space, like this "123.123456789012, " That's 18 ASCII characters and if there were about 20 million ASCII sequences like that in the CSV text file, you'd get 370 MB.

请先登录，再进行评论。