Out of memory in parfor loop

61 次查看（过去 30 天）
John R
John R 2011-11-7
I am getting an error of being out of memory when running the following code, but only when I use a parfor loop, it works fine with a for loop, albeit much much slower. I am using sparse matrices because of the large size size of the matrix I am calculating.
A=spalloc(m,n,maxElements); %m and n are large, <200,000
parfor index=1:n
value=zeros(m,1);
valIndex=findIndex %findIndex is just a custom function
val=findArea(n); %findArea is just a custom function
value(valIndex)=val;
A(:,index)=value;
end
I am not sure what the issue is with the memory.
I have also tried indexing the elements of the matrix by storing them in a row, column, and value variable, but then indexing within those variable gets complicated and ends up either not being compatible with the parfor loop or using up more memory.
Can anyone explain to me why I am running out of memory? Is it copying the matrix to each worker before writing to it?
I am trying to figure out a way to save each "value" iteration, and then build the matrix "A" outside the loop, but I haven't come up with a good idea of how to do this?
  2 个评论
Edric Ellis
Edric Ellis 2011-11-8
How does the out-of-memory error manifest itself?
John R
John R 2011-11-9
It simply says "Out of memory, run HELP MEMORY for more options" or something along those lines.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Yoav Livneh
Yoav Livneh 2011-11-8
You are running out of memory because parfor uses a lot more memory.
Let's say for example, that each iteration of the regular for loop reuires 300 MB of RAM. IF you have, for example, 8 workers running the parfor loop, each iteration will require at least 2400 MB of RAM, not including the overhead data used by the parfor loop itself.
Maybe if you decrease the number of parallel workers you could find the optimum point between memory usage and running time.
Another solution is to optimize the contents of your loops. For example you don't need to call zeros(m,1) every iteration, you can use:
value = zeros(m,n);
parfor index=1:n
valIndex=findIndex;
val=findArea(n);
value(valIndex,n) = val
A(valIndex,index) = val;
end
This will result in much less overhead memory transfers, which will reduce run time and memory usage.
The code above might not be legal in a parfor loop, depending on the custom funtions, but I hope you understand the principle.
  3 个评论
显示 1更早的评论
Yoav Livneh
Yoav Livneh 2011-11-9
Maybe if you could write what findIndex does and what are valIndex's dimensions, we can figure it out together.
John R
John R 2011-11-10
findIndex produces a vector of indices that correspond to the rows that values from "value" should go in the matrix "A".
I feel like there should be some easy fix for my out of memory problem, I just need to save the column of values for each loop, I don't need the entire matrix A available in memory on each loop, I just need it at the end of the script. Is there some way I can do this? Do I have to resort to using "save" and then reconstruct the matrix after all 200,000 rows are saved on my HD?

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by