Parfor loop just hangs, CPU usage goes to zero

Question

JohnDapper 2016-3-14

5
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero

回答：海粟吴 2024-5-30

Hi all. Here is a sample code of what I am attempting to run.

parfor i = 1:num
answer(:,i) = someFunction(someData(:,i));
end

Key information: "someFunction" is a C++ mex file. "someData" is a memmapfile (memmapfilename.data) because it is too large to be loaded onto each worker

Oddly, the parfor loop just hangs, the CPU usage goes to zero, and when I CTRL+C, here is what I get:

Operation terminated by user during distcomp.remoteparfor/getCompleteIntervals (line
127)
In parallel_function>distributed_execution (line 820)
        [tags, out] = P.getCompleteIntervals(chunkSize);
In parallel_function (line 587)
        R = distributed_execution(...

This isn't an issue if I replace the "parfor" with a simple "for" - everything works fine. What seems to happen is that some of the workers become unresponsive. After the above issue is encountered, even running a simple command such as

pctRunOnAll 1+1

will return "2" on only some, but not all, workers.

Any help would be great. A fresh re-installation did not help. Validation for "parpool" passed.

6 个评论
显示 4更早的评论隐藏 4更早的评论

David Saidman 2020-1-14

Did anybody get any success with this? I'm having exact same 2017b, definitely no keyboard statement.

If I wait a bit, it ends up running but on a single CPU (event tho I have 18 in my pool on a cpu with 20 physical and 40 logical cores, about 10gb spare memory in performance monitor).

海粟吴 2024-5-30

parallel.internal.parfor.ParforEngine/getCompleteIntervals

位置 parallel_function>distributed_execution (第 746 行)

[tags, out] = P.getCompleteIntervals(chunkSize);

位置 parallel_function (第 578 行)

R = distributed_execution(...

Same problem observed in 2024a, this problem remains for 7 years, and no solution came out yet.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Dave Behera 2016-3-24

2
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero#answer_214909

It seems that there is a deadlock when the workers are trying to the access the file using the same object (that you got from memmapfile). Due to that, the progress is getting stalled with zero CPU usage and no abort message.

Can you try creating a separate memmapfile object within each parfor iteration and passing it to the someFunction function? This may make the file access thread-safe.

Also, could you try the same workflow with spmd?

http://www.mathworks.com/help/distcomp/spmd.html

9 个评论
显示 7更早的评论隐藏 7更早的评论

Arabarra 2021-1-11

Same problem here. In my case it is not reproductible, sometimes it will work, sometimes not. No deadlocks or anything suspicious in the code.

海粟吴 2024-5-30

Agree on what had been discussed above, it happened on 2024a too. When the MATLAB could solve this problem. Soon or Nerver?

请先登录，再进行评论。

Answer 2

arvid Martens 2018-1-9

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero#answer_299313

I noticed that the problem started to occur after I updated the drivers of the GPUs that are being used during the calculations. Rolling back the drivers resolved the problem. However, new GPU hardware is on its way, as the current ones are pretty old. So I hope the problem is resolved by then.

Is there a way to throw an error when this stalling occurs? I could write an error handling to reduce the time lost by this stalling.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 3

Andrea Stevanato 2018-7-13

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero#answer_328721

I have the same error with matlab 2018a.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 4

Sanjay Manohar 2019-6-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero#answer_377630

编辑：Sanjay Manohar 2019-6-3

I was having the same parfor problem, until I noticed I had a "keyboard" instruction in my code.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 5

DeepSea 2021-8-15

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero#answer_767262

在 MATLAB Online 中打开

I've been stucked in this problem for couples of weeks, and fixed it by removing "continue" in an if-judgement and a for-loop.

for CondA
    ...
    if CondB 
        continue; % Avoid using "continue"
    end
    ...
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

海粟吴 2024-5-30

My code have the similar structure, it has while in forloop

请先登录，再进行评论。

Answer 6

Aditya Shukla 2021-10-23

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/273509-parfor-loop-just-hangs-cpu-usage-goes-to-zero#answer_814918

I suddenly got this problem since yesterday, before which all the code ran nicely. I really do not know why this happened. it is so annoying. Any one found a solution?