Fastest way to append file with matrix data

49 次查看（过去 30 天）
Koen
Koen 2020-11-19
回答： tetra 2023-3-18
I'm using Matlab to measure data in a rateControl timed loop at a certain frequency, ~ 20 Hz. In my app, I want to plot this data during the measurement, so it is important to do this efficiently (15 channels, 20 Hz, running for a day creates a 10^6 x 15 sized matrix). I'm trying to use animatedline such that I do not have to concatenate and plot an increasingly lengthy matrix. In my timed loop I would overwrite a 1 x 15 matrix containing measured data, use addpoints for plotting, and append the measured data to my file. Appending a file regularly is preferred over saving a large matrix at the end anyway, should the measurement crash for some reason.
My question is what the best/fastest way is of saving/appending a file. The support page for dlmwrite recommends writematrix. However, using the below example I find that dlmwrite is faster.
num_writes = 1000;
time = zeros(1,num_writes); % track in case appending long files takes longer
M = randn(1,15); % random start data
freq = 20; % 'measurement' frequency
for mode = ["dlmwrite","writematrix"]
if strcmp(mode,'dlmwrite')
dlmwrite('test.csv',M); % in reality, write headers instead of random data at start
for i = 1:num_writes % 'measurement' loop
tic
N = randn(size(M));
dlmwrite('test.csv',N,'-append');
time(i) = toc;
end
elseif strcmp(mode,'writematrix')
writematrix(M,'test.csv')
for i = 1:num_writes
tic
N = randn(size(M));
writematrix(N,'test.csv','WriteMode','append')
time(i) = toc;
end
end
tperoperation = sum(time)/num_writes;
disp(mode)
disp(['total elapsed time is ',num2str(sum(time)),' s'])
disp(['or ',num2str(tperoperation),' s per operation'])
disp(['this is ',num2str(freq*100*tperoperation),'% of loop time (',num2str(freq),' Hz)'])
end
Is it possible to improve on the write speed, e.g. using a different file format? I found mex_WriteMatrix on the file exchange, however this seems to improve speed only when appending a large amount of data, which is not the case for me.
If my method of a timed loop with animated lines is flawed in general, I'm also open to suggestions.

请先登录，再回答此问题。

回答（1 个）

tetra
tetra 2023-3-18
Ran in:
I was forwarded to this thread and implemented user dpb's suggestion. fwrite is indeed much faster - roughly 100 times. For others looking for an implementation of that answer I have adapted the code provided by OP.
num_writes = 1000;
time = zeros(1,num_writes); % track in case appending long files takes longer
M = randn(1,15); % random start data
freq = 20; % 'measurement' frequency
for mode = ["dlmwrite","writematrix","binary"]
if strcmp(mode,'dlmwrite')
dlmwrite('test.csv',M); % in reality, write headers instead of random data at start
for i = 1:num_writes % 'measurement' loop
tic
N = randn(size(M));
dlmwrite('test.csv',N,'-append');
time(i) = toc;
end
elseif strcmp(mode,'writematrix')
writematrix(M,'test.csv')
for i = 1:num_writes
tic
N = randn(size(M));
writematrix(N,'test.csv','WriteMode','append')
time(i) = toc;
end
elseif strcmp(mode,'binary')
fidbin = fopen('test.bin','a'); % 'a' = append data
for i = 1:num_writes
tic
N = randn(size(M));
fwrite(fidbin,N,'double');
time(i) = toc;
end
fclose(fidbin);
end
tperoperation = sum(time)/num_writes;
disp(mode)
disp(['total elapsed time is ',num2str(sum(time)),' s'])
disp(['or ',num2str(tperoperation),' s per operation'])
disp(['this is ',num2str(freq*100*tperoperation),'% of loop time (',num2str(freq),' Hz)'])
end
dlmwrite
total elapsed time is 0.27458 s
or 0.00027458 s per operation
this is 0.54916% of loop time (20 Hz)
writematrix
total elapsed time is 1.8607 s
or 0.0018607 s per operation
this is 3.7214% of loop time (20 Hz)
binary
total elapsed time is 0.007412 s
or 7.412e-06 s per operation
this is 0.014824% of loop time (20 Hz)
On Matlab desktop, dlmwrite is slower than when executed as live script: my results are 8.04, 10.55 and 0.02 seconds respectively. You can open the binary file using
bincheck=fopen('test.bin','r'); % 'r' = read data
bindata=fread(bincheck,[lenght(M),inf],'double')'; % transposing will result in 'correct' size
fclose(bincheck);

类别

在 Help Center 和 File Exchange 中查找有关 Text Files 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by