Parallel sever: Request GPU via SLURM

2022 12 15
1 个回答
回答已采纳
更新时间：2022 12 19
5 次查看（30 天）

0 个投票

Hello,
I have access to an HPC and have been running CPU based code on it for some time now with help from the mathworks team. Parrellel server dir() for files local to the server - MATLAB Answers - MATLAB Central (mathworks.com)
I now wish to use GPU acceleration. When I try to use a gpuArray I get an error that the GPU devices are not found. This is because they were not requested in my batch command to SLURM which makes sense.
Is there a method to inject the additional SLURM gpu request argument e.g., --gres=gpu:2 into the .batch request (I am thinking something like the additionalSubmitArgs below) or is there a normal workflow for doing so without changing the submitCommunicatingJob function? It seems like a normal thing to request however I cannot find a straightforward workflow for it.
function job = submit_brainwave_job()
cluster = parcluster();
idir = '/inputDir'
idir = '/outDir'
job = cluster.batch(@batch_build_0_0_1,1,{idir,odir}, ...
"AutoAddClientPath",false, "CaptureDiary",true, ...
"CurrentFolder",".", "Pool",3);
%additionalSubmitArgs = sprintf('--ntasks=%d --gres=gpu:2', props.NumberOfTasks);
end
Kind regards,
Christopher

请先登录，再回答此问题。

 采纳的回答

Raymond Norris
Raymond Norris 2022-12-15

1 个投票

Hi @Christopher McCausland, try the following
cluster = parcluster();
cluster.AdditionalProperties.AdditionalSubmitArgs = '--gres=gpu:2';
You might also need to specify a gpu partition, as such
cluster.AdditionalProperties.AdditionalSubmitArgs = '--gres=gpu:2 -p <name-of-gpu-partition>';
Remember to unset AdditionalSubmitArgs if you don't need a GPU for you next job.

3 个评论
显示 1更早的评论 隐藏 1更早的评论

Hi Raymond,
Thank you once again for the help! I will give this a go and let you know.
I take it any SLURM command request could be submitted via cluster.AdditionalProperties.AdditionalSubmitArgs = in the same way? For example;
cluster.AdditionalProperties.AdditionalSubmitArgs = '--gres=gpu:2','--cpus-per-gpu=2';
Matching the AdditionalSubmitArgs to the SLURM documentation on what can be requested?
Kind regards,
Christopher
Raymond Norris
Raymond Norris 2022-12-16
Yes, that's right. In fact, for individual users familiar with their cluster, this is a viable option. When our team works with Cluster Admins, to make a more robust solution, we expand out AdditionalProperties, as such
cluster = parcluster();
cluster.AdditionalProperties.CpusPerGpu = 2;
cluster.AdditionalProperties.GpusPerNode = 2;
cluster.AdditionalProperties.Partition = 'gpu';
then on the backend, we implement getCommonSubmitArgs to convert this to the full Slurm syntax. This way users don't have to know the Slurm syntax themselves.
Hi Raymond,
That is perfect, thank you for the help once again! This submission pipeline is brillent!
Kind regards,
Christopher

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Third-Party Cluster Configuration 的更多信息

产品

版本

R2021b

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by