Writing error in parfor loop

3 次查看（过去 30 天）
J T
J T 2015-6-5
评论： Walter Roberson 2015-6-8
Hi everyone,
I have a script that randomly removes and repredicts links in networks 100 times. Of course the bigger the networks are the longer it takes so, I made the 100 iterations into a parfor loop. It works fine for smaller networks but with the big ones I get the error
A write error occurred while sending to worker xy.
I assumed its due to memory problems but I can check in the task manager that the error occurs before the memory of my pc is used up. Also it says writing error...So does anybody have an idea what this error message refers to? Im using windows 7 pro, 64bit and matlab R2014a and I have 196GB of ram(I know quite a lot (:).
Any help would be appreciated! Thanks a lot, Josephine
  3 个评论
显示 1更早的评论
J T
J T 2015-6-5
Dear Adam, thanks a lot for the fast answer! It depends on the run, sometimes the memory needed goes to 180gb before the crash mostly only to 140gb. The code is for yet unpublished material, so give me a minute I need to think about what I may write about it...
J T
J T 2015-6-5
Dear Adam, Im sorry Im not so sure what I can post or not, I will have to wait until my supervisor is back on monday. But maybe you can be so kind, to comment the following: The code runs without parfor(just so slow it will probably take years) and needs about 100gb. So if it is that for two workers then u need 200gb of course I get the error. But I tried running the parfor with only one worker, even that does not work. I tried printing out something in the parfor and it seems it doesnt even enter the first iteration. Is it possible, that the transferring alone needs so much memory?? Thanks! Josephine

请先登录，再进行评论。

请先登录，再回答此问题。

回答（2 个）

Thomas Koelen
Thomas Koelen 2015-6-5
Try writing a function that writes the data, then call this function in your parfor loop.
  1 个评论
J T
J T 2015-6-5
Dear Thomas, so simplified u mean I should do parfor 1:x; a=f(x); where f=x**2 or so instead of parfor 1:x; a=x**2; ? The function that is doing the major work Im already using like this, I put the rest also in a function now, but unfortunately I get the same error..

请先登录，再进行评论。

Walter Roberson
Walter Roberson 2015-6-5
The data needs to be transferred to even the one worker so Yes you can run out of memory even with one worker.
Have a look at Worker Object Wrapper as it might help in your situation.
  2 个评论
J T
J T 2015-6-8
Dear Walter thanks for your answer! Is there a bit more of a detailed description of that worker object wrapper? Im new to everything with parallelization and although it seems pretty simple to use I guess Im using it wrong because it opens several matlabpools where i didnt ask for them to be opened and that of course is not working.
Walter Roberson
Walter Roberson 2015-6-8
The documentation is the File Exchange Contribution page and the code itself which is there. It works using handle objects.
There is one part of it that invokes "spmd" as part of the initialization. That suggests that you should open your parfor pool first before using the worker object wrapper.

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by