Fast Export Method

Question

Brian 2012-6-11

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/40893-fast-export-method

I have always used the export command to export a dataset to a comma delimited text file. When the files get a little larger (50-100MB) the export function seems to run very slow. Are there other functions that are much faster than the export dataset function?

My dataset is simple (just large). Col 1 is text 2:4 are numeric.

MyDS = dataset(MyData(:,1),MyData(:,2),MyData(:,3),MyData(:,4));
export(MyDS,'file','R:\Equity_Quant\BrianB\Factor Rotation\BulkData.txt','Delimiter',',');

Thanks much, Brian

3 个评论
显示 1更早的评论隐藏 1更早的评论

Walter Roberson 2012-6-12

export() is a method of the dataset class.

http://www.mathworks.com/help/toolbox/stats/dataset.export.html

per isakson 2012-6-12

Walter, thanks!

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

per isakson 2012-6-12

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/40893-fast-export-method#answer_50527

在 MATLAB Online 中打开

The functions save, load and whos is an alternative

    save( 'my_datasets.mat', 'MyDS' )
    save( 'my_datasets.mat', 'MyDS_2', '-append' )

or even faster

save( 'my_datasets.mat', 'MyDS', '-v6' )

without guarantee. (Dataset does not overload save as far as I can see.)

--- Faster method to export to text file ---

Some data. I use three columns to avoid word wrap.

    N       = 1e1;
    MyData  = randn( N, 3 );

The variant with dataset

    tic
    ds  = dataset( MyData(:,1), MyData(:,2), MyData(:,3) );
    export( ds, 'file','c:\temp\test_ds.txt', 'Delimiter', ',' )    
    toc

produces this output

    Var1,Var2,Var3
    0.490621219889722,-0.64500446971637,1.6174852368957
    0.0660146644635964,-0.408559384693175,0.445251540387096
    ...

A faster variant

    tic
    ms  = permute( MyData, [ 2, 1 ] );
    fid = fopen( 'c:\temp\test_fp.txt', 'w' );
    fprintf( fid, '%s,%s,%s\n', 'Var1', 'Var2', 'Var3' );
    fprintf( fid, '%f,%f,%f\n', ms );
    fclose( fid );
    toc

produces this output

    Var1,Var2,Var3
    0.490621,-0.645004,1.617485
    0.066015,-0.408559,0.445252
    ...

I've run these two variants with different values for N. The faster variant is at least an order of magnitude faster.

With N=5e6 on my three years old vanilla desktop I get "Elapsed time is 16.478986 seconds." with the faster variant. That is 8 MB/s - something.

How many decimals do you need in the text file?

.

--- fprintf is hard to exceed ---

I added this test

    tic
    dlmwrite( 'c:\temp\test_dlm.txt', MyData )
    toc

With N=5e6 I got the following elapsed times

fprintf (A faster variant): Elapsed time is 16.507707 seconds.
dlmwrite: Elapsed time is 202.905913 seconds. (without header)
dateset.export: Elapsed time is 819.649789 seconds.

With plain Matlab I don't think there is a faster alternative. Maybe it is possible to do something faster with a MEX-function.

3 个评论
显示 1更早的评论隐藏 1更早的评论

per isakson 2012-6-13

That's correct. I didn't realize the purpose. Maybe it is an option to write directly to the database.

Brian 2012-6-13

Since the database is on a server separate from my local machine, my job is running some calculations via matlab and exporting the large files to the database server so that I can do a bulk import via the SQL server. Hence the need for exporting the data. Would fprintf be my fastest option for writing to a text file?

Thanks a lot,

Brian

请先登录，再进行评论。