Why workers keep aborting during parallel computation on cluster?

43 次查看（过去 30 天）
Muh Alam
Muh Alam 2020-12-7
评论： Kojiro Saito 2021-2-8
I keep getting the warning
Warning: A worker aborted during execution of the parfor loop. The parfor loop will now run again on the remaining workers.
In distcomp/remoteparfor/handleIntervalErrorResult (line 245)
In distcomp/remoteparfor/getCompleteIntervals (line 392)
In parallel_function>distributed_execution (line 741)
In parallel_function (line 573)
In fuction_pa1 (line 100)]
when I run a simulation that has parfor loop on the cluster. I noticed that workers abort excution one after another and that seems to happen more when on a cluster compated to my PC.
I would like to know the reason of this issue, and is there a way to avoid it ?
Thanks.
  19 个评论
显示 17更早的评论
Kojiro Saito
Kojiro Saito 2021-2-7
Heterogenous would be a cause. This link is a system requirement of Parallel Server not Parallel Computing Toolbox, but it says an important point;
"Parallel processing constructs that work on the infrastructure enabled by parpool—parfor, parfeval spmd, distributed arrays, and message passing functions—cannot be used on a heterogeneous cluster configuration. The underlying MPI infrastructure requires that all cluster computers have matching word sizes and processor endianness."

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Kojiro Saito
Kojiro Saito 2021-2-8
Heterogenous environment would be a cause of this issue. This link is a system requirement of Parallel Server not Parallel Computing Toolbox, but it says an important point;
"Parallel processing constructs that work on the infrastructure enabled by parpool—parfor, parfeval spmd, distributed arrays, and message passing functions—cannot be used on a heterogeneous cluster configuration. The underlying MPI infrastructure requires that all cluster computers have matching word sizes and processor endianness."
  2 个评论
Muh Alam
Muh Alam 2021-2-8
I forgot to ask, are there ways to work around this issue? would chosing computing nodes on the cluster having same archeticture suffice ? if so how to do that using slurm ?
Kojiro Saito
Kojiro Saito 2021-2-8
If you know the nodes names which are homogeneous, you can specify the nodes with sbatch. For example, if node0 to node4 are the same OS, you can use nodelist option (or -w option).
sbatch --nodelist node[0-4] yourscript.sh

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Third-Party Cluster Configuration 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by