importing large 4GB+ tab-delimited ascii file makes MATLAB crash

Question

Sean Phillips 2019-5-17

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/462737-importing-large-4gb-tab-delimited-ascii-file-makes-matlab-crash

回答： Sean Phillips 2019-5-18

采纳的回答： Sean Phillips

在 MATLAB Online 中打开

i didnt have a problem until now the data files have been scaled up in complexity

i usually use

IMPORT = importdata('components.dat')

in this case components.dat is 4 GB+

it is a table of numerical values consisting of 12 columns and millions of rows

for smaller models this has worked but now MATLAB crashes when i try to import at this size.

I have 96GB of RAM and dual Xeons.

Do i need to change setting in MATLAB or is there better ways to import this data

this is an exmaple of the file

       0.01       -50.00         0.00     -3950.00 -2.0000E+04    0.000       -4000.        -1.409E-15  -2.138E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9900E+04    0.000       -4000.        -1.410E-15  -2.193E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9800E+04    0.000       -4000.        -1.411E-15  -2.249E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9700E+04    0.000       -4000.        -1.412E-15  -2.306E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9600E+04    0.000       -4000.        -1.411E-15  -2.364E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9500E+04    0.000       -4000.        -1.411E-15  -2.424E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9400E+04    0.000       -4000.        -1.409E-15  -2.485E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9300E+04    0.000       -4000.        -1.407E-15  -2.547E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9200E+04    0.000       -4000.        -1.405E-15  -2.611E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9100E+04    0.000       -4000.        -1.401E-15  -2.676E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.9000E+04    0.000       -4000.        -1.397E-15  -2.742E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.8900E+04    0.000       -4000.        -1.392E-15  -2.810E-15    0.00E+00   0.00E+00
       0.01       -50.00         0.00     -3950.00 -1.8800E+04    0.000       -4000.        -1.386E-15  -2.879E-15    0.00E+00   0.00E+00

it is tab delimited

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Sean Phillips 2019-5-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/462737-importing-large-4gb-tab-delimited-ascii-file-makes-matlab-crash#answer_375669

在 MATLAB Online 中打开

thank you for your answers. I found a very simple solution

using

M = dlmread('filename.dat')

as my data is asscii-delimited file of numerical data

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Steven Lord 2019-5-17

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/462737-importing-large-4gb-tab-delimited-ascii-file-makes-matlab-crash#answer_375553

Consider making a tall array or perhaps using memmapfile. These tools are linked from this documentation page.

Please also send the crash log file to Technical Support so we can investigate why MATLAB crashed.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Sean Phillips 2019-5-18

在 MATLAB Online 中打开

Hi Steven Thank you for your answer,

I have tried this so far using memmapfile

 datamem = memmapfile('components.dat', 'Format', 'double', 'Writable', true);
 D = datamem.data;

how then do i use mapped data for regualr MATLAB operations?

请先登录，再进行评论。

Answer 3

per isakson 2019-5-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/462737-importing-large-4gb-tab-delimited-ascii-file-makes-matlab-crash#answer_375570

编辑：per isakson 2019-5-18

在 MATLAB Online 中打开

That's strange. I use Win10 and R2018b (64bit). btw: What do you mean by "[...] up in complexity" ?

Anyhow try

>> num = cssm( 'components.dat' ); 
>> whos num
  Name       Size            Bytes  Class     Attributes
  num       13x12             1248  double  

where

function    num = cssm( ffs )
    fid = fopen( ffs );
    cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f'         ...
                ,   'Delimiter','\t', 'CollectOutput',true  ); 
    fclose( fid );
    num = cac{1};
end

If that doesn't work, I guess there is some strange stuff in the middle of your text file.

I added

, 'ReturnOnError',false

to the textscan call and the string, "an error", to the text file an run

>> tic, num = cssm( 'cssm.txt' );toc
Error using textscan
Mismatch between file and format character vector.
Trouble reading 'Numeric' field from file (row number 1005147, field number 1) ==> an error\n
Error in cssm (line 4)
    cac = textscan( fid, '%f%f%f%f%f%f%f%f%f%f%f%f'         ... 
>> 