How can I speed up importing large .d files?

Question

Felix Lauwaert 2015-8-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/232013-how-can-i-speed-up-importing-large-d-files

编辑： Cedric 2015-8-1

Hi,

soon I'll have large files, about 50000x6 (more than one of them) and I'll have to work on them several times to plot different stuff. I've realised that "importdata" command is very slow and I've been recommended to use GNUPlot, but I would prefer to stick to MATLAB if possible. So, is there any command to load data to workspace in a faster way? Maybe using .xls instead of .d?

Thanks.

2 个评论
显示无隐藏无

Cedric 2015-8-1

Could you provide a sample file?

Felix Lauwaert 2015-8-1

test.txt

I had to upload it as .txt because of website requirements. I tic-tocked the importdata and it was arround 12s, but I'll have to deal with way larger files soon.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Cedric 2015-8-1

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/232013-how-can-i-speed-up-importing-large-d-files#answer_187936

编辑：Cedric 2015-8-1

在 MATLAB Online 中打开

I would do something along the following line:

 buffer = fileread( 'test.txt' ) ; 
 data   = sscanf( buffer(76:end), '%f' ) ;
 data   = reshape( data, 6, [] )' ;

I built a file with more than 50k rows to test, and it takes half the time of IMPORTDATA. We can easily improve it so it doesn't rely on a fixed header size. You could also do something like:

 fName  = 'test.txt' ;
 fId    = fopen( fName, 'r' ) ;
 header = strsplit( strtrim( fgetl( fId ) ), ' ' ) ;
 data   = fscanf( fId, '%f' ) ;
 data   = reshape( data, 6, [] )' ;
 fclose( fId ) ;

but this is slower.

PS: I don't understand why your IMPORTDATA is that slow. On my test file with >50k lines, here is the timing:

 IMPORTDATA: Elapsed time is 0.312664 seconds.
 FSCANF    : Elapsed time is 0.457961 seconds.
 FILEREAD  : Elapsed time is 0.171987 seconds.

5 个评论
显示 3更早的评论隐藏 3更早的评论

per isakson 2015-8-1

编辑：per isakson 2015-8-1

在 MATLAB Online 中打开

I created a test file with

cssm0(50000)

where

function    cssm0( N )
    h = sprintf( 'H%06d ', 1:N );
    d = sprintf( '%f '   , 1:N );
    fid = fopen('test_long_rows.txt','w');
    fprintf( fid, '%s\n', h,d,d,d,d,d,d );
    fclose(fid);
end

and used profile. Maybe, I misread the original question. Anyhow, whether the rows or the columns are long and short, respectively, makes a huge difference with importdata

Cedric 2015-8-1

编辑：Cedric 2015-8-1

It's interesting, 6x5e4 -> 2s and 5e4x6 -> 0.3s.

In any case, I've always been avoiding IMPORTDATA like plague (especially after looking at its source code), because its behavior is size-dependent and difficult to predict. This leads to situations like the one reported recently on the forum, where it works with a files that contains thousands of rows of data, but fails when there are only 30 lines (for the same data structure).

请先登录，再进行评论。

Answer 2

per isakson 2015-8-1

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/232013-how-can-i-speed-up-importing-large-d-files#answer_187934

编辑：per isakson 2015-8-1

Does the file consist of header rows followed by data rows, which contains only numerical data? (No string data such as date time data.)

Try txt2mat, by Andres "txt2mat basically is a wrapper for sscanf, it quickly converts ascii files containing m-by-n numeric data, allowing for header lines"