Writing error in parfor loop

Question

J T 2015-6-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/222603-writing-error-in-parfor-loop

评论： Walter Roberson 2015-6-8

Hi everyone,

I have a script that randomly removes and repredicts links in networks 100 times. Of course the bigger the networks are the longer it takes so, I made the 100 iterations into a parfor loop. It works fine for smaller networks but with the big ones I get the error

A write error occurred while sending to worker xy.

I assumed its due to memory problems but I can check in the task manager that the error occurs before the memory of my pc is used up. Also it says writing error...So does anybody have an idea what this error message refers to? Im using windows 7 pro, 64bit and matlab R2014a and I have 196GB of ram(I know quite a lot (:).

Any help would be appreciated! Thanks a lot, Josephine

3 个评论
显示 1更早的评论隐藏 1更早的评论

J T 2015-6-5

Dear Adam, thanks a lot for the fast answer! It depends on the run, sometimes the memory needed goes to 180gb before the crash mostly only to 140gb. The code is for yet unpublished material, so give me a minute I need to think about what I may write about it...

J T 2015-6-5

Dear Adam, Im sorry Im not so sure what I can post or not, I will have to wait until my supervisor is back on monday. But maybe you can be so kind, to comment the following: The code runs without parfor(just so slow it will probably take years) and needs about 100gb. So if it is that for two workers then u need 200gb of course I get the error. But I tried running the parfor with only one worker, even that does not work. I tried printing out something in the parfor and it seems it doesnt even enter the first iteration. Is it possible, that the transferring alone needs so much memory?? Thanks! Josephine

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Thomas Koelen 2015-6-5

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/222603-writing-error-in-parfor-loop#answer_181674

Try writing a function that writes the data, then call this function in your parfor loop.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

J T 2015-6-5

Dear Thomas, so simplified u mean I should do parfor 1:x; a=f(x); where f=x**2 or so instead of parfor 1:x; a=x**2; ? The function that is doing the major work Im already using like this, I put the rest also in a function now, but unfortunately I get the same error..

请先登录，再进行评论。

Answer 2

Walter Roberson 2015-6-5

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/222603-writing-error-in-parfor-loop#answer_181742

The data needs to be transferred to even the one worker so Yes you can run out of memory even with one worker.

Have a look at Worker Object Wrapper as it might help in your situation.

2 个评论
显示无隐藏无

J T 2015-6-8

Dear Walter thanks for your answer! Is there a bit more of a detailed description of that worker object wrapper? Im new to everything with parallelization and although it seems pretty simple to use I guess Im using it wrong because it opens several matlabpools where i didnt ask for them to be opened and that of course is not working.

Walter Roberson 2015-6-8

The documentation is the File Exchange Contribution page and the code itself which is there. It works using handle objects.

There is one part of it that invokes "spmd" as part of the initialization. That suggests that you should open your parfor pool first before using the worker object wrapper.

请先登录，再进行评论。