I need help with parpool and optimizing recources for image reconstruction, running matlab with batch script
2 次查看(过去 30 天)
显示 更早的评论
Im running an image reconstruction (1798 head volumes) script on a computation cluster
These are my recources
Nodes: 1
Mem per core: 43 GB
Cores per node: 24
Mem tot: 1024 GB
The reconstruction script reconstructs each full head volume (total 1798 head volumes) in the for loop for k=1:length(tframes), tframes=1798
:
![](https://www.mathworks.com/matlabcentral/answers/uploaded_files/341170/image.png)
Now, my idea is to use parpool to parallelize the for loop. When using parpool one needs to specify the number of workers. This number should match the number of CPUs requested.
IN the computation cluster I use recources are requested as follows:
#SBATCH -p hugemem % name of the partition
#SBATCH -t 072:00:00 &time
#SBATCH --nodes=1 %nodes
#SBATCH --mem-per-cpu= 43G
#SBATCH --ntasks= 24
#SBATCH --cpus-per-task= 1
My question, how should I use parpool to parallelize this task of reconstructing the 1798 head volumes? Or do you think in this case parfor would be better?
0 个评论
回答(1 个)
Raymond Norris
2020-8-5
Hi,
I haven't taken the time to see if parfor will work in your case, but have some general comments/thoughts.
parfor is a MATLAB parallel construct for parallelizing a for loop. In order for a parfor to run correctly, it need a pool of workers (i.e. parpool). By default, if a parallel pool is not running, calling parfor will instantiate the pool with a default number of workers (12 in your case).
I would suggest the following. Change your Slurm jobscript as such
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=24
You're only running 1 task (MATLAB), but you'd like to dedicate 24 cores to it(your pool). In your MATLAB code, add the following
% Query for available cores
sz = str2num(getenv('SLURM_CPUS_PER_TASK'));
if isempty(sz), sz = maxNumCompThreads; end
p = parpool('local',sz);
parfor k = 1:length(tframes)
...
end
You might decriment sz by 1 to account for the MATLAB client running.
Raymond
0 个评论
另请参阅
类别
在 Help Center 和 File Exchange 中查找有关 Parallel Computing Fundamentals 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!