Is there a utility like nvidia-smi within matlab to determine which gpus are in current use?

11 次查看（过去 30 天）
RLeighton
RLeighton 2023-10-10
回答： Joss Knight 2023-10-11
10 GPUs, 4 users. Want to avoid usage conflict.

请先登录，再回答此问题。

采纳的回答

Walter Roberson
Walter Roberson 2023-10-10
编辑：Walter Roberson 2023-10-10
No, there is no utility like that built into MATLAB.
My understanding is that the order of GPUs in the gpu device list depends upon the order in which the GPUs answered the probes at boot time, and so is potentially variable. However, if I understand correctly it is consistent during any particular boot session.
Therefore, if you can get the cooperation of all users, you could use simple resource management techniques such as creating a binary file with 10 entries in it that record whether a program is requesting to use the given GPU. If you were to write process IDs into the file, then you could have a management layer that checked whether the recorded process still existed and if not then freed up the entry.
  2 个评论
Sam Marshalik
Sam Marshalik 2023-10-10
In combination with Walter's response, here is an idea.
Big disclaimer: Keep in mind this is not a perfect solution, as the information you get from nvidia-smi or gpuDevice will be stale after you retrieve it and may not be utilized in that momment in time but is being used by someone.
You can use the gpuDevice command in MATLAB to view GPU resources. You can use view memory usage to see if a GPU is being used in that momment (snippit from gpuDevice)
TotalMemory: 2147287040 (2.15 GB)
AvailableMemory: 1615209678 (1.62 GB)
You can also use the system command to call nvidia-smi via MATLAB. You can then use MATLAB text parsing to programmatically look at usage, get a handle to that GPU (have not played with this, but assuming it is possible), and use that GPU.

请先登录，再进行评论。

更多回答（1 个）

Joss Knight
Joss Knight 2023-10-11
gpuDeviceTable is the utility for listing your devices and their properties. You can also see all your devices in the Parallel menu on the Home toolstrip.
The menu entry will also tell you when the device was last used by your MATLAB, so you can check whether your code is successfully accessing the correct GPU.
The simplest way to control GPU access to multiple users is to put the devices into Exclusive Process compute mode, which means that only one user will be able to select each device at a time. You can use a command like "nvidia-smi -i 0 -c 3" to set the compute mode.
For a more sophisticated automated management of which user has access to which GPU, you will need a more sophisticated scheduling or resource management system. This can use environment variables like CUDA_VISIBLE_DEVICES to ensure that each user starting a process on a resource accesses a particular GPU only.

类别

在 Help Center 和 File Exchange 中查找有关 GPU Computing 的更多信息

标签

产品


版本

R2023b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by