importing large files with text/numbers

2 次查看（过去 30 天）

John 2014-7-25

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/143348-importing-large-files-with-text-numbers

评论： dpb 2014-7-25

Hi,

I am trying to import a text (not binary) data file into Matlab. It has fields of text and numbers arranged in lines. Each line may have a different numbers of fields, but 23 is about the maximum. The fields are comma separated and each line ends with CR/NL.

xlsread() works, if the file is small, say 100 lines. Then I get a cell array with 100x23 entries.

This is what Matlab uses: [~, ~, untitled] = xlsread('JB100.csv','JB100'); untitled(cellfun(@(x) ~isempty(x) && isnumeric(x) && isnan(x),untitled)) = {''};

However, my files can be over 1 million lines long and I seem to run out of memory at around 100,000 lines. [Also if memory were not an issue I think there would be a limit of 1048576 lines as in Excel.]

How do I get around this issue?

Thanks, John

2 个评论
显示无隐藏无

per isakson 2014-7-25

Attach a sample file, which includes all types of lines.

How do you want the result?

dpb 2014-7-25

在 MATLAB Online 中打开

   [~, ~, untitled] = xlsread('JB100.csv','JB100'); 
   untitled(cellfun(@(x) ~isempty(x) && isnumeric(x) && isnan(x),untitled)) = {''};

This way is reading the full file as text and then trying to do something with it. I'd suggest using the first two return variables instead to convert the numeric data to internal representation during the read process--this is bound to reduce memory requirements, particularly since to get the numeric values from the input text you're going to require both being in memory simultaneously.

It would be even better if you could dispense with the text data altogether.

请先登录，再进行评论。

请先登录，再回答此问题。