Write and read numeric data line by line
显示 更早的评论
Here is a working example of what I have in mind:
N=10000; M=10000;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
u=dlmread('file',' ',[k-1 0 k-1 M-1]);
%calculations using vector u
end
toc
The writing part is fine (about 5 seconds on my PC), but the reading is
awfully slow - about 43 seconds. My guess is that it would help if reading
is resumed automatically after the last line read. The command "fgetl"
appears to do this, but is designed for character, rather than numeric,
variables.
Any other suggestions?
(The problem entails generating an array
of size 10^5 X 10^5, which is used by another program.)
采纳的回答
Michael Soskind
2020-5-5
Hi Chris,
These are quite a bit faster in my experience, and testing with smaller example arrays, give a much better result.
I compare the two in the following code:
%% Using fprintf and fscanf, which seem order of magnitude faster
N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
fid = fopen('file.txt', 'a');
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
fprintf(fid,repmat('%f ', 1,M), x);
end
fclose(fid);
toc
tic
fid = fopen('file.txt', 'r');
for k=1:N
u=fscanf(fid,repmat('%f ', 1,M),[1 M]);
%calculations using vector u
end
fclose(fid);
toc
%% Using dlmwrite and dlmread
N=100; M=100;
x=rand(1,M);
dlmwrite('file',x,'delimiter',' ')
tic
for k=1:N-1
x=rand(1,M);
%In the real application x is the result of some calculations
dlmwrite('file',x,'delimiter',' ','-append')
end
toc
tic
for k=1:N
u=dlmread('file',' ',[k-1 0 k-1 M-1]);
%calculations using vector u
end
toc
You can test this with any data size you like, I suppose my computer is not nearly as fast as yours at the file processing.
The append feature of fprintf is particularly useful.
Hopefully that helps,
Michael
2 个评论
Michael Soskind
2020-5-5
Sorry, a few notes
- I did not remove the dlmwrite at the beginning of the code there.
- Further, the speed up is quite good for a 1000 element case on my computer.
I have ~2.5 seconds for reading and writing with fprintf, and 5.6 and 60 seconds for writing and reading, respectively with dlmread, using a 1000 element array.
Chris Koen
2020-5-6
更多回答（0 个）
类别
在 帮助中心 和 File Exchange 中查找有关 Text Files 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
