Is it possible to accelerate the speed of saving data into files with parallel way?

Question

wei zhang 2020-7-18

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/567093-is-it-possible-to-accelerate-the-speed-of-saving-data-into-files-with-parallel-way

编辑： Bruno Luong 2020-7-21

I am trying to save data to files, i.e. txt file. The speed is very low. This step takes almost 80% time over my whole script. My computer has many cores. Could I use them, i.e. parallel way, to help to accelerate the speed in Matlab? I don't know much about know this. May it is impossible. Please just tell me some idea or suggestion about it? Thank you very much.

(edited as Rik suggested)

My code uses fprintf function in a big loop. This is the most time consuming step.

i.e. I need to save a large matrix a, which is 4000000*4

p = fopen(filename,'w');
for i=1:length(a)
    fprintf(fp,'%d %d %d %d\n',a(i,1),a(i,2),a(i,3),a(i,4)); 
end

Is there any method to accelerate it?

4 个评论
显示 2更早的评论隐藏 2更早的评论

Rik 2020-7-19

Without knowing more about your code, this is only conjecture. You need to post exact code if you want specific advice. You can also run the profiler to find bottlenecks in your code.

wei zhang 2020-7-21

@Rik I had add a mini example of my fprintf codes. I haven't find any method to improve it. Maybe it is impossible.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Bruno Luong 2020-7-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/567093-is-it-possible-to-accelerate-the-speed-of-saving-data-into-files-with-parallel-way#answer_468618

编辑：Bruno Luong 2020-7-21

在 MATLAB Online 中打开

You might try to remove the for-loop

fp = fopen(filename,'w');
fprintf(fp,'%d %d %d %d\n', a(:,1:4).'); % remove the indexing with your array has 4 columns
fclose(fp);

A file is a sequential access device. There is no use of parallel CPU. When you write a file a hard disk head must go sequentially in tracks and put a little electrical signal there to store your data. More or less samething happens with all other HW, even SSD.

Another thing that takes time is convert the internal binary format to digital of your fprintf('%d').

If you can, just write file in binary format. That is the best method (fatest).

2 个评论
显示无隐藏无

wei zhang 2020-7-21

在 MATLAB Online 中打开

I have tested your codethat your way could accelerate the process remarkably. Really thankful. The inputs should be matrix, not number. Could you give another advice to the code below to remove the for loop? I have no ideas about it with only symbol.

for i = 1:n_face
    fprintf(fp,'0 0\n');
end

Bruno Luong 2020-7-21

在 MATLAB Online 中打开

Try this:

fp = fopen('test.txt','w')
fwrite(fp, repmat(sprintf('0 0\n'),1,n_face));
fclose(fp);

Or you could do with the same method as I have showed

fp = fopen('test.txt','w');
a = repmat([0 0], n_face, 1);
fprintf(fp,'%d %d\n', a); % no need for transpose since it's all 0s
fclose(fp);

请先登录，再进行评论。

Answer 2

Mohammad Sami 2020-7-21

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/567093-is-it-possible-to-accelerate-the-speed-of-saving-data-into-files-with-parallel-way#answer_468576

编辑：Mohammad Sami 2020-7-21

在 MATLAB Online 中打开

Is there a reason why you want to use a for loop to write this ?

You can write the entire matrix to file in one go using writematrix function.

a = randi([0 10],1000000,4);
tic;
writematrix(a,'abc.txt','FileType','text','Delimiter',' ');
toc;

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Rik 2020-7-21

I'm on mobile so I can't write and test the code, but you can also use fprintf. You just need to make sure the orientation of a is correct (it will read the input column by column when writing the lines in the text file).

请先登录，再进行评论。