Specify the parallel pool job timeout

10 次查看(过去 30 天)
Hi,
I am running some tests using a remote cluster (no local cluster). I send my functions in batch mode. I know that the functions take a large execution time, around 2 / 4 hours. When I try to run, I get the message:
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I looked into the documentation how to change the default timeout time. The only way I could find is with the "wait" command, as
wait(job,"finished",18000);
However, I keep on getting the same error. How can I change the default parallel pool job timeout in the remote cluster?

回答(1 个)

Raymond Norris
Raymond Norris 2021-10-1
So you doing something like the following
cluster = parcluster;
job = cluster.batch(@mycode,...., 'Pool',size);
Then what you're suggesting is that let's say your code looked something like
function mycode
pause(10 * 60)
parfor idx = 1:N
...
end
On the cluster, the workers have a default timeout of 5 minutes and therefore errors out because you're running code (the pause) for 10 minutes before the workers are being used in the parfor.
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I tried quickly reproducing this with the local scheduler, but didn't (it shouldn't matter if I'm jusing local). How are you getting the error message? And which scheduler are you using MJS or generic (e.g. PBS)?
  9 个评论
Maria
Maria 2021-10-1
Aha, this is good to know!
But now I took away the distributed and I get
MATLAB worker exited with status 9 during task execution.
What does that mean?
Raymond Norris
Raymond Norris 2021-10-2
One of the workers crashed, possibly because of an out of memory issue. Email Technical Support (support@mathworks.com) and they can walk you through debug steps and getting the MJS log files to troubleshoot.

请先登录,再进行评论。

产品


版本

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by