Is there a utility like nvidia-smi within matlab to determine which gpus are in current use?

Question

RLeighton 2023-10-10

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2031614-is-there-a-utility-like-nvidia-smi-within-matlab-to-determine-which-gpus-are-in-current-use

回答： Joss Knight 2023-10-11

采纳的回答： Walter Roberson

10 GPUs, 4 users. Want to avoid usage conflict.

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Walter Roberson 2023-10-10

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2031614-is-there-a-utility-like-nvidia-smi-within-matlab-to-determine-which-gpus-are-in-current-use#answer_1330199

编辑：Walter Roberson 2023-10-10

No, there is no utility like that built into MATLAB.

My understanding is that the order of GPUs in the gpu device list depends upon the order in which the GPUs answered the probes at boot time, and so is potentially variable. However, if I understand correctly it is consistent during any particular boot session.

Therefore, if you can get the cooperation of all users, you could use simple resource management techniques such as creating a binary file with 10 entries in it that record whether a program is requesting to use the given GPU. If you were to write process IDs into the file, then you could have a management layer that checked whether the recorded process still existed and if not then freed up the entry.

2 个评论
显示无隐藏无

Sam Marshalik 2023-10-10

In combination with Walter's response, here is an idea.

Big disclaimer: Keep in mind this is not a perfect solution, as the information you get from nvidia-smi or gpuDevice will be stale after you retrieve it and may not be utilized in that momment in time but is being used by someone.

You can use the gpuDevice command in MATLAB to view GPU resources. You can use view memory usage to see if a GPU is being used in that momment (snippit from gpuDevice)

TotalMemory: 2147287040 (2.15 GB)

AvailableMemory: 1615209678 (1.62 GB)

You can also use the system command to call nvidia-smi via MATLAB. You can then use MATLAB text parsing to programmatically look at usage, get a handle to that GPU (have not played with this, but assuming it is possible), and use that GPU.

RLeighton 2023-10-10

Using the second approach, based on temperature. Didn't think of first approach. Thanks

请先登录，再进行评论。

Answer 2

Joss Knight 2023-10-11

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2031614-is-there-a-utility-like-nvidia-smi-within-matlab-to-determine-which-gpus-are-in-current-use#answer_1330674

gpuDeviceTable is the utility for listing your devices and their properties. You can also see all your devices in the Parallel menu on the Home toolstrip.

The menu entry will also tell you when the device was last used by your MATLAB, so you can check whether your code is successfully accessing the correct GPU.

The simplest way to control GPU access to multiple users is to put the devices into Exclusive Process compute mode, which means that only one user will be able to select each device at a time. You can use a command like "nvidia-smi -i 0 -c 3" to set the compute mode.

For a more sophisticated automated management of which user has access to which GPU, you will need a more sophisticated scheduling or resource management system. This can use environment variables like CUDA_VISIBLE_DEVICES to ensure that each user starting a process on a resource accesses a particular GPU only.