I need help with parpool and optimizing recources for image reconstruction, running matlab with batch script

1 次查看（过去 30 天）
Pseudoscientist
Pseudoscientist 2020-8-4
回答： Raymond Norris 2020-8-5
Im running an image reconstruction (1798 head volumes) script on a computation cluster
These are my recources
Nodes: 1
Mem per core: 43 GB
Cores per node: 24
Mem tot: 1024 GB
The reconstruction script reconstructs each full head volume (total 1798 head volumes) in the for loop for k=1:length(tframes), tframes=1798:
Now, my idea is to use parpool to parallelize the for loop. When using parpool one needs to specify the number of workers. This number should match the number of CPUs requested.
IN the computation cluster I use recources are requested as follows:
#SBATCH -p hugemem % name of the partition
#SBATCH -t 072:00:00 &time
#SBATCH --nodes=1 %nodes
#SBATCH --mem-per-cpu= 43G
#SBATCH --ntasks= 24
#SBATCH --cpus-per-task= 1
My question, how should I use parpool to parallelize this task of reconstructing the 1798 head volumes? Or do you think in this case parfor would be better?

请先登录，再回答此问题。

回答（1 个）

Raymond Norris
Raymond Norris 2020-8-5
Hi,
I haven't taken the time to see if parfor will work in your case, but have some general comments/thoughts.
parfor is a MATLAB parallel construct for parallelizing a for loop. In order for a parfor to run correctly, it need a pool of workers (i.e. parpool). By default, if a parallel pool is not running, calling parfor will instantiate the pool with a default number of workers (12 in your case).
I would suggest the following. Change your Slurm jobscript as such
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24
You're only running 1 task (MATLAB), but you'd like to dedicate 24 cores to it(your pool). In your MATLAB code, add the following
% Query for available cores
sz = str2num(getenv('SLURM_CPUS_PER_TASK'));
if isempty(sz), sz = maxNumCompThreads; end
p = parpool('local',sz);
parfor k = 1:length(tframes)
...
end
You might decriment sz by 1 to account for the MATLAB client running.
Raymond

类别

在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by