Matlab uses much more memory reading a file

2 次查看（过去 30 天）
Changyue Song
Changyue Song 2017-4-24
评论： Walter Roberson 2017-4-26
Hi, I have a csv file with 22 columns and 871,000 rows. Columns are separated by comma. Each value is a text or number embraced by quote. The size of the file is about 150 MB. But after I read the file into MATLAB using textscan, the variable which stores the data takes up about 2GB of memory! For another csv file with 47 columns and 7,000,000 rows which have similar structure and is about 2GB in hard disk, MATLAB takes forever to read it using textscan. However, R is able to read these two files and the memory used is approximately the same as the space needed in hard disk. Are there any explanations for this? Thanks.
PS: Thank you for your help guys. Yes I can process the files line by line and discard the columns that I do not need. I am just curious why Matlab uses so much more memory than R or than the original file in the hard disk.
  8 个评论
显示 6更早的评论
per isakson
per isakson 2017-4-26
编辑：per isakson 2017-4-26
You write "Each value is a text or number embraced by quote." Given the quotes, there is no alternative to your format specifier - afaik.
Recently Matlab has become better to handle large files and strings. Which release do you use?
Are the sizes of the different strings approximately equal?
Walter Roberson
Walter Roberson 2017-4-26
If there are fields known to be numeric but enclose by quotes anyhow, then you can code a literal " character in the textscan format.

请先登录，再进行评论。

请先登录，再回答此问题。

回答（1 个）

AstroGuy1984
AstroGuy1984 2017-4-25
Do you need to read the whole matrix at once or are you just doing a routine line by line? If you are just running something on each line, I'd suggest NOT reading the whole file at once and doing something like:
fid = fopen('file.dat');
tline = fgetl(fid);
while ischar(tline)
% STUFF YOU WANT TO DO
tline = fgetl(fid);
end
fclose(fid); %DONT FOGET THIS!
Unfortunately, handling huge data files you eventually have to get creative in how you handle things. I would also suggest looking into using properties such as fseek and frewind if you're needing to pull out specific entries from a huge database. It will save you a ton of time and effort if you're just referencing the textfile for something.

类别

在 Help Center 和 File Exchange 中查找有关 Data Import and Export 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by