fprintf to txt too slow, anyway to accelerate it?

Question

Yu Li 2017-3-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/328119-fprintf-to-txt-too-slow-anyway-to-accelerate-it

编辑： dpb 2017-3-5

采纳的回答： dpb

A.mat

在 MATLAB Online 中打开

hi:

I have data with different column, such as

a=ones(100,2), b=ones(50,2);

but I want to append them and export to a .txt file, so what I did is:

construct a new matrix that has the size (max row of a&b, sum of the column of a&b), here is with the size: (100,4).
for each row, find the data that equal to '1', and build a new row that set all the element '1' to ' ,', and save to .txt file using fprinf. however, I found that it is too slow.

below is my test code, and I also attatched the test file A.

load A
tic
fid = fopen('test.txt','w');    
for i=1:1:length(A(:,1))
    str={''};
    for j=1:1:length(A(1,:))
        if A(i,j)==1e5;
            str=strcat(str,{' ,'});
        else
            str=strcat(str,sprintf('%0.5e',A(i,j)),',');
        end
      end
      fprintf(fid,'%s\n',str{1});
  end
  fclose(fid);
toc
tic
dlmwrite('test_dlm.txt',A)
toc

here the 1e5 is the identifier that need to be set to ' ,'. the result shows that fprintf will cost about 129 seconds, while the dlmwrite cost only 12 seconds.

thanks!

Li

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

dpb 2017-3-5

在 MATLAB Online 中打开

If you'll profile the code, you'll find all the time is spent in the strcat operations, not in fprintf

Unclear to me what you're really wanting to do; if the point is to append the two datasets, why not just

a=ones(100,2), b=ones(50,2);
csvwrite('aplusb.csv',[a;b]

and be done with it instead of creating a file that's 100x4 elements instead of 150x2? If the point is to only put values in the accumlated file that are unique somehow to the two arrays, then do that processing first on the combined and then write.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

dpb 2017-3-5

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/328119-fprintf-to-txt-too-slow-anyway-to-accelerate-it#answer_257367

编辑：dpb 2017-3-5

在 MATLAB Online 中打开

A=[a;b];              % append to one array for convenience
A(A==badValue)=nan;   % replace the bum values with NaN
csvwrite('FixedUpCombined.csv',A)  % write to csv file with missing value indicator

Much simpler than creating the specific format to write an empty field; let NaN serve as the placeholder instead. This will also be unequivocal as to what is bum data; "csvread fills empty delimited fields with zero." from the documentation.

Note, however, if you're adamant about using empty delimited fields, the way to go about it is to find the locations in the row and use those locations to build the proper format string with the appropriate number of fields of given type, then write the record using that format string. repmat is exceedingly useful in these machinations since unfortunately the C-like formatting strings cannot accept a repeat count.