Saving values in parfor loops

8 次查看（过去 30 天）
Ronaldo
Ronaldo 2013-4-17
I have a large database (1e8,4) and I want to use parfor command to do some calculations. I also want to use matlabpool command. My problem is in saving the output of the calculations. I used the following line to save the outputs:
parsave(sprintf('%d.mat', i), NumCr, NumNi);
where parsave is a function to save variables NumCr and NumNi.
The problem is that since this line of the code creates a huge number of *.mat files, MATLAB crashes. I also know that with matlabpool open command, global variables are not working. Is it possible to mention how I can save the values?
  2 个评论
Edric Ellis
Edric Ellis 2013-4-17
Why do you want to save data from the workers - is it not sufficient to let the PARFOR loop complete and save the data from the client?
Are you creating one file per 1e8 iterations of your PARFOR loop?
Ronaldo
Ronaldo 2013-4-17
I would appreciate it if you mention how I can modify the following code to save the values of NumCr and NumNi in each iteration?
parfor i=1:size(A,1)
O=A(i,1:4);
Om=ones(size(A,1),3);
Om1=O(1,1)*ones(size(A,1),1);
Om2=O(1,2)*ones(size(A,1),1);
Om3=O(1,3)*ones(size(A,1),1);
d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
distance=d>=R1m & d<=R2m;
NumCr=sum(distance.*Cr);
NumNi=sum(distance.*Ni);
parsave(sprintf('%d.mat', i), NumCr, NumNi);
end

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Thomas
Thomas 2013-4-17
I assume it doesn't really crash with an error but it takes close to forever to output that many files...(?)
I don't know about the memory limitations you have .. your database should be around 1.5GB ... but can't you save the results in an matrix or a cell array first? Like
results = zeros(length(database)) %need to define this outside parfor
parfor i=1:length(database)
result(i) = NumNi;
end;
save...
Maybe you could even do this inplace, so database(i,1) = NumNi to save memory. Alternatively, if you run into memory problems you could divide your loop into smaller chunks and process some 10000 or 100000 elements at a time in parallel, then save and then continue.
If you have a lot of memory you could also try to setup a ramdisk and output your mat files on this disk - which is much faster even compared to an ssd disk.
  3 个评论
显示 1更早的评论
Thomas
Thomas 2013-4-17
编辑：Thomas 2013-4-17
There is no reason why result(i)=NumNi should not work with parfor. I just checked it does. However, it is crucial to define the whole result matrix/vector BEFORE the loop as i did above. Otherwise it would grow dynamically, which is not possible with parallel constructs.
Also try if your loop works in general with a small example database.
Ronaldo
Ronaldo 2013-4-17
It works. Thanks a lot for your great help.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by