Parallel Computing on cluster
1 次查看(过去 30 天)
显示 更早的评论
I can use Matlab on Pronghorn(a cluster on university of nevada Reno). For my work I need to utilize the parallel computing option on this cluster.
This is my test code that starts parallel computing for matlab(I name this code, PCTest.m, for calling it on sbatch file).
clear all
tic
parfor i = 1:5
fprintf('Progress %d of %d...\n',i,5);
pause(5);
end
toc
The above code should take around 5 seconds to run, if parallel computing is working.
This is my job submission code for cluster(xyz.sh),
#!/bin/bash
#SBATCH --job-name disable_multithreading
#SBATCH --time=01:00:00
#SBATCH --nodes=1 --ntasks-per-node=6
#SBATCH --cpus-per-task=1
module load matlab
matlab -nosplash -nodesktop -r "PCTest; exit"
I type sbatch xyz.sh to run the matlab file. But this does not start workers on cluster, like it does on my desktop or laptop. Any help will be hightly appreciated. Thanks
0 个评论
回答(1 个)
Raymond Norris
2021-4-12
As it is written, the parallel pool will start when the parfor is called, which is then included in the timing. And since you're not explicitly starting it, it could be starting as many as 12 workers. Try the following:
clear all
% Time how long it takes to start the pool
tic
parpool(5);
toc
% Time how long it takes to run the parfor
tic
parfor i = 1:5
fprintf('Progress %d of %d...\n',i,5);
pause(5);
end
toc
To make this a bit more robust, query Slurm for the number of assigned tasks to start your pool (doing this off the top of my head, I think it's SLURM_NTASKS).
sz = getenv('SLURM_NTASKS');
if isempty(sz)
% For some reason, we're not running in a Slurm job
sz = maxNumCompThreads;
end
% Time how long it takes to start the pool
tic
parpool(sz-1);
toc
% Time how long it takes to run the parfor
tic
parfor i = 1:5
fprintf('Progress %d of %d...\n',i,5);
pause(5);
end
toc
Lastly, since R2019a, you can shorten
matlab -nosplash -nodesktop -r "PCTest; exit"
to
matlab -batch PCTest
4 个评论
Raymond Norris
2021-4-13
I've seen this before. Are you submitting several Slurm scripts at once? If so, contact Technical Support (support@mathworks.com) and they'll show you the best way to create a temporary job storage location (i.e. local_cluster_jobs) for each job.
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Third-Party Cluster Configuration 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!