parfor loop errors on AMD cores limits

2021 1 19
1 个回答
回答已采纳
更新时间：2021 3 2
6 次查看（30 天）

0 个投票

Hello,
I am trying to run a simple parfor script on nodes on our cluster. The code works fine until I try to use > 46 CPUs (workers) at once, on one server. Some of our latest nodes have 128 AMD cores. I can run up to 56 cores on our Intel CPU servers (nodes) , but on any AMD I get errors (java runtime and others) when using >46 cores. It would be great to use all 128 cores on these new nodes for our MATLAB code. I have tried increasing memory and I still get these errors when using > 46 cores.
I will attach the MATLAB crash dump, code and sbatch files.
My sbatch file (I have tried many, many different parameters) -
#!/bin/bash
#SBATCH -J pfor_matlab
#SBATCH -o pfor".%j".out
#SBATCH -e pfor".%j".err
#SBATCH -t 45:00
#SBATCH -N 1
#SBATCH -p normal
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=48
module load matlab
hostname -s
env | egrep SLURM
matlab -nosplash -nodesktop -r "pfor"
The sbatch produces this output in the SLURM .err file-
Error using parpool (line 145)
Parallel pool failed to start with the following error. For more detailed
information, validate the profile 'local' in the Cluster Profile Manager.
Error in pfor (line 5)
parpool('local', str2num(getenv('SLURM_CPUS_PER_TASK')))
Caused by:
Error using parallel.internal.pool.InteractiveClient>iThrowWithCause (line
670)
Failed to initialize the interactive session.
Error using
parallel.internal.pool.InteractiveClient>iThrowIfBadParallelJobStatus
(line 781)
The interactive communicating job failed with no message
Thank you for any pointers!
Mark

2 个评论
显示 无 隐藏 无

Walter Roberson
Walter Roberson 2021-2-9
The volunteers are not likely to know the solution for this; you should open a support case.
Mark PIERCY
Mark PIERCY 2021-2-9
Thanks Walter, I just submitted one.
Best,
Mark

请先登录，再进行评论。

请先登录，再回答此问题。

 采纳的回答

Mark PIERCY
Mark PIERCY 2021-3-2
编辑：Mark PIERCY 2021-3-2

1 个投票

This happens on either AMD or Intel nodes in our cluster. On our 128-core nodes, parpool(128) fails with an nproc limit at 32,768.
Our systems architect figured this out. It turns out that Matlab was seg faulting on the default nproc limit (max number of user processes), which is set by default to 4,096.
The issue is with the /etc/security/limits.d/20-nproc.conf provided by the PAM RPM on CentOS 7, which limits every user to 4096 processes at once. But not for Matlab.
Details:
>> parpool('local', 46)
Starting parallel pool (parpool) using the 'local' profile ...
*** Error in `/share/software/user/restricted/matlab/R2020a/bin/glnxa64/MATLAB': double free or corruption (!prev): 0x00007f4e4027d090 ***
*** Error in `/share/software/user/restricted/matlab/R2020a/bin/glnxa64/MATLAB*** Error in `/share/software/user/restricted/matlab/R2020a/bin/glnxa64/MATLAB': free(): corrupted unsorted chunks: 0x00007f4e401e10d0 ***
A bad case of segmentation fault:
[7191121.585896] MATLAB[120580]: segfault at 118c0f20 ip 00007fa1cd076b8d sp 00007fa1973fbc60 error 4 in libc-2.17.so[7fa1cd03d000+1c2000]
[7191121.610055] traps: MATLAB[120453] general protection ip:7fcb5d0b0b8d sp:7fcb273fbc60 error:0 in libc-2.17.so[7fcb5d077000+1c2000]

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Third-Party Cluster Configuration 的更多信息

产品

版本

R2020a

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by