Specify the parallel pool job timeout
12 次查看(过去 30 天)
显示 更早的评论
Hi,
I am running some tests using a remote cluster (no local cluster). I send my functions in batch mode. I know that the functions take a large execution time, around 2 / 4 hours. When I try to run, I get the message:
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I looked into the documentation how to change the default timeout time. The only way I could find is with the "wait" command, as
wait(job,"finished",18000);
However, I keep on getting the same error. How can I change the default parallel pool job timeout in the remote cluster?
0 个评论
回答(1 个)
Raymond Norris
2021-10-1
So you doing something like the following
cluster = parcluster;
job = cluster.batch(@mycode,...., 'Pool',size);
Then what you're suggesting is that let's say your code looked something like
function mycode
pause(10 * 60)
parfor idx = 1:N
...
end
On the cluster, the workers have a default timeout of 5 minutes and therefore errors out because you're running code (the pause) for 10 minutes before the workers are being used in the parfor.
'The parallel pool job was cancelled because it failed to finish within the specified parallel pool job timout of 300 seconds'
I tried quickly reproducing this with the local scheduler, but didn't (it shouldn't matter if I'm jusing local). How are you getting the error message? And which scheduler are you using MJS or generic (e.g. PBS)?
9 个评论
Raymond Norris
2021-10-2
One of the workers crashed, possibly because of an out of memory issue. Email Technical Support (support@mathworks.com) and they can walk you through debug steps and getting the MJS log files to troubleshoot.
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!