Saving values in parfor loops

2013 4 17
1 个回答
回答已采纳
8 次查看（30 天）

0 个投票

I have a large database (1e8,4) and I want to use parfor command to do some calculations. I also want to use matlabpool command. My problem is in saving the output of the calculations. I used the following line to save the outputs:
parsave(sprintf('%d.mat', i), NumCr, NumNi);
where parsave is a function to save variables NumCr and NumNi.
The problem is that since this line of the code creates a huge number of *.mat files, MATLAB crashes. I also know that with matlabpool open command, global variables are not working. Is it possible to mention how I can save the values?

2 个评论
显示 无 隐藏 无

Edric Ellis
Edric Ellis 2013-4-17
Why do you want to save data from the workers - is it not sufficient to let the PARFOR loop complete and save the data from the client?
Are you creating one file per 1e8 iterations of your PARFOR loop?
Ronaldo
Ronaldo 2013-4-17
I would appreciate it if you mention how I can modify the following code to save the values of NumCr and NumNi in each iteration?
parfor i=1:size(A,1)
O=A(i,1:4);
Om=ones(size(A,1),3);
Om1=O(1,1)*ones(size(A,1),1);
Om2=O(1,2)*ones(size(A,1),1);
Om3=O(1,3)*ones(size(A,1),1);
d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
distance=d>=R1m & d<=R2m;
NumCr=sum(distance.*Cr);
NumNi=sum(distance.*Ni);
parsave(sprintf('%d.mat', i), NumCr, NumNi);
end

请先登录，再进行评论。

请先登录，再回答此问题。

 采纳的回答

Thomas
Thomas 2013-4-17

0 个投票

I assume it doesn't really crash with an error but it takes close to forever to output that many files...(?)
I don't know about the memory limitations you have .. your database should be around 1.5GB ... but can't you save the results in an matrix or a cell array first? Like
results = zeros(length(database)) %need to define this outside parfor
parfor i=1:length(database)
result(i) = NumNi;
end;
save...
Maybe you could even do this inplace, so database(i,1) = NumNi to save memory. Alternatively, if you run into memory problems you could divide your loop into smaller chunks and process some 10000 or 100000 elements at a time in parallel, then save and then continue.
If you have a lot of memory you could also try to setup a ramdisk and output your mat files on this disk - which is much faster even compared to an ssd disk.

3 个评论
显示 1更早的评论 隐藏 1更早的评论

Ronaldo
Ronaldo 2013-4-17
result(i)=NumNi is not working in parfor loops. Below is the code that I wrote. I would appreciate it if you help me understand what my mistake is.
parfor i=1:size(A,1)
O=A(i,1:4);
Om=ones(size(A,1),3);
Om1=O(1,1)*ones(size(A,1),1);
Om2=O(1,2)*ones(size(A,1),1);
Om3=O(1,3)*ones(size(A,1),1);
d=sqrt(((Om1-A1).^2)+((Om2-A2).^2)+((Om3-A3).^2));
distance=d>=R1m & d<=R2m;
NumCr=sum(distance.*Cr);
NumNi=sum(distance.*Ni);
parsave(sprintf('%d.mat', i), NumCr, NumNi);
end
Thomas
Thomas 2013-4-17
编辑：Thomas 2013-4-17
There is no reason why result(i)=NumNi should not work with parfor. I just checked it does. However, it is crucial to define the whole result matrix/vector BEFORE the loop as i did above. Otherwise it would grow dynamically, which is not possible with parallel constructs.
Also try if your loop works in general with a small example database.
Ronaldo
Ronaldo 2013-4-17
It works. Thanks a lot for your great help.

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by