Fastest way to append file with matrix data

Question

Koen 2020-11-19

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/653363-fastest-way-to-append-file-with-matrix-data

回答： tetra 2023-3-18

I'm using Matlab to measure data in a rateControl timed loop at a certain frequency, ~ 20 Hz. In my app, I want to plot this data during the measurement, so it is important to do this efficiently (15 channels, 20 Hz, running for a day creates a 10^6 x 15 sized matrix). I'm trying to use animatedline such that I do not have to concatenate and plot an increasingly lengthy matrix. In my timed loop I would overwrite a 1 x 15 matrix containing measured data, use addpoints for plotting, and append the measured data to my file. Appending a file regularly is preferred over saving a large matrix at the end anyway, should the measurement crash for some reason.

My question is what the best/fastest way is of saving/appending a file. The support page for dlmwrite recommends writematrix. However, using the below example I find that dlmwrite is faster.

num_writes = 1000;              
time = zeros(1,num_writes);     % track in case appending long files takes longer
M = randn(1,15);                % random start data
freq = 20;                      % 'measurement' frequency
for mode = ["dlmwrite","writematrix"]
    if strcmp(mode,'dlmwrite')
        dlmwrite('test.csv',M); % in reality, write headers instead of random data at start
        for i = 1:num_writes    % 'measurement' loop
            tic
            N = randn(size(M));
            dlmwrite('test.csv',N,'-append');
            time(i) = toc;
        end
        
    elseif strcmp(mode,'writematrix')
        writematrix(M,'test.csv')
        for i = 1:num_writes
            tic
            N = randn(size(M));
            writematrix(N,'test.csv','WriteMode','append')
            time(i) = toc;
        end
    end
    
    tperoperation = sum(time)/num_writes;
    disp(mode)
    disp(['total elapsed time is ',num2str(sum(time)),' s'])
    disp(['or ',num2str(tperoperation),' s per operation'])
    disp(['this is ',num2str(freq*100*tperoperation),'% of loop time (',num2str(freq),' Hz)'])
end

Is it possible to improve on the write speed, e.g. using a different file format? I found mex_WriteMatrix on the file exchange, however this seems to improve speed only when appending a large amount of data, which is not the case for me.

If my method of a timed loop with animated lines is flawed in general, I'm also open to suggestions.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Koen 2020-11-19

Thank you, I will look into it.

Walter Roberson 2020-11-19

Yes, binary will always be faster.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

tetra 2023-3-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/653363-fastest-way-to-append-file-with-matrix-data#answer_1195800

在 MATLAB Online 中打开

I was forwarded to this thread and implemented user dpb's suggestion. fwrite is indeed much faster - roughly 100 times. For others looking for an implementation of that answer I have adapted the code provided by OP.

num_writes = 1000;              
time = zeros(1,num_writes);     % track in case appending long files takes longer
M = randn(1,15);                % random start data
freq = 20;                      % 'measurement' frequency
for mode = ["dlmwrite","writematrix","binary"]
    if strcmp(mode,'dlmwrite')
        dlmwrite('test.csv',M); % in reality, write headers instead of random data at start
        for i = 1:num_writes    % 'measurement' loop
            tic
            N = randn(size(M));
            dlmwrite('test.csv',N,'-append');
            time(i) = toc;
        end
        
    elseif strcmp(mode,'writematrix')
        writematrix(M,'test.csv')
        for i = 1:num_writes
            tic
            N = randn(size(M));
            writematrix(N,'test.csv','WriteMode','append')
            time(i) = toc;
        end
    elseif strcmp(mode,'binary')
        fidbin = fopen('test.bin','a');         % 'a' = append data
        for i = 1:num_writes
            tic
            N = randn(size(M));
            fwrite(fidbin,N,'double');
            time(i) = toc;
        end
        fclose(fidbin);
    end
    
    tperoperation = sum(time)/num_writes;
    disp(mode)
    disp(['total elapsed time is ',num2str(sum(time)),' s'])
    disp(['or ',num2str(tperoperation),' s per operation'])
    disp(['this is ',num2str(freq*100*tperoperation),'% of loop time (',num2str(freq),' Hz)'])
end
dlmwrite
total elapsed time is 0.27458 s
or 0.00027458 s per operation
this is 0.54916% of loop time (20 Hz)
writematrix
total elapsed time is 1.8607 s
or 0.0018607 s per operation
this is 3.7214% of loop time (20 Hz)
binary
total elapsed time is 0.007412 s
or 7.412e-06 s per operation
this is 0.014824% of loop time (20 Hz)

On Matlab desktop, dlmwrite is slower than when executed as live script: my results are 8.04, 10.55 and 0.02 seconds respectively. You can open the binary file using

bincheck=fopen('test.bin','r');                     % 'r' = read data
bindata=fread(bincheck,[lenght(M),inf],'double')';  % transposing will result in 'correct' size
fclose(bincheck);