0 个投票

Hi all,
It seems that my problem is an old one and many people have faced it, but I still have not fixed problem yet so hope to get help here.
My problem is that: I ran a program (complex, including many functions) on the lab cluster and used 25gb of memory. If I use "for" loop I did not see any problem, but when I used "parfor" I soon received a message says that the required memory is exceeded than 25gb I arranged.
I made a search on this problem and know some thing that it is likely due to data cloning problem but I did not see a good suggestion how to fix it.
Can anyone suggest me a good solution?
Thank you in advance.

请先登录，再回答此问题。

 采纳的回答

Walter Roberson
Walter Roberson 2016-4-30

0 个投票

parfor statically examines all of the variables used in the body of the parfor. Any of the defined variables that are not indexed by the loop variable will generally be completely copied into the process that does the work, even if only a portion of the variable is accessed. If the number of workers times the amount of data read in the loop is greater than available memory, you have an inherent problem. If not all of the data of a variable is needed, sometimes using distributed arrays or co-distributed or sliced helps; other times you should extract subsets of the data before the parfor.
Sometimes the question is one of efficiency rather than amount of storage. In that case see http://www.mathworks.com/matlabcentral/fileexchange/31972-worker-object-wrapper
http://www.mathworks.com/matlabcentral/answers/121831-how-to-make-data-persist-on-workers-between-calls-to-parfor#comment_202694
http://www.mathworks.com/help/distcomp/sliced-variables.html
I know that a recent MATLAB release introduced a replacement for Worker Object Wrapper, but unfortunately I do not remember the name of the feature at the moment.

3 个评论
显示 1更早的评论 隐藏 1更早的评论

Tony Tom
Tony Tom 2016-5-2
编辑：Tony Tom 2016-5-2
Hi Roberson,
Thank you for your suggestions. I am learning Worker Object Wrapper, However, I am aware that this function still makes a copy to all workers, thus a decent amount of ram is still required.
As you advised, we should index variables by the loop index, so only data needed for a worker is accessed. That means we slice variables.
But what about the case a heavy data need to be accessed in each loop: for example:
parfor i=1:100
g=a_function(H_D);
end
where H_D is a big array (my case,~9GB). Because the whole data of H_D is accessed in a loop, thus if each worker needs a copy of the whole data of H_D, the total amount of ram required is very big. Can we have some way that H_D stored in a worker and be used for all other workers?
Walter Roberson
Walter Roberson 2016-5-2
Have a look in the File Exchange for Shared Array.
Tony Tom
Tony Tom 2016-5-3
Thank you Roberson. I have checked, worker object wrapper does not work well on my case, I still face memory problem. I will try with shared array.

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Loops and Conditional Statements 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by