GPU performance significantly slow when running multiple MATLAB

Question

Yu Li 2018-9-20

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/419899-gpu-performance-significantly-slow-when-running-multiple-matlab

评论： Walter Roberson 2018-9-22

Hi:

I have a code using GPU calculation, when it used by 1 MATLAB, it runs OK.

but when I open two MATLAB.exe in my computer, and run this GPU-relate function at the same time, the GPU performance becomes significantly slow.

for example, when run 1 MATLAB, it cost about 20 seconds, but when 2 MATLAB runs at the same time, it cost about 2000 seconds.

one thing I'm sure is that the GPU memory is enough for more than two tasks. therefore, the memory should not be the problem.

my GPU is GTX 980ti, which has 6GB memory.

Just want know if there is any mistake with my operation, or is there anyway to improve this performance.

Thanks!

Yu

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Joss Knight 2018-9-20

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/419899-gpu-performance-significantly-slow-when-running-multiple-matlab#answer_337674

编辑：Joss Knight 2018-9-20

I'd be very surprised if memory isn't the issue here, because contention between your two MATLABs will be forcing continual synchronization and raw allocation on both. I'd have to see what it is you're doing.

Ultimately you should explain why you are using two MATLABs if you only have one GPU. You may have a good reason of course but it's worth looking into.

10 个评论
显示 8更早的评论隐藏 8更早的评论

Joss Knight 2018-9-20

在 MATLAB Online 中打开

Wow, that's an interesting way to post your response!

Your peak memory usage isn't the amount of memory taken up by gpuArray variables at the end of your calculations. Each operation requires working memory. So for instance, to perform a matrix multiplication may require space for the input variables, space for the output variable, and space in which to perform the calculation which is as big as the input. The need for working memory varies from operation to operation. To monitor memory usage you can use the nvidia-smi utility and set it running in a command window or terminal:

nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader -lms 50

One way to check this might be to reduce the size of your data. If the unusual behaviour (performance when running on two GPUs) persists for small data then perhaps there is a real problem. I would expect it to take more than twice as long, because there is significant overhead due to the need for the GPU driver to continually switch context. 100x slower is unusual for that though.

I'm concerned that you have to reboot your computer, however if you are using this GPU to drive your display that is not surprising since it isn't being given any chance to render your desktop. If not, then perhaps the real issue here is CPU memory, since it tends to be virtual memory access that makes a system appear to hang due to continually thrashing of the disk.

Yu Li 2018-9-22

编辑：Yu Li 2018-9-22

Hi:

Thanks for your suggestion, I use GPU because it has about 2-3 times faster than vector calculation using MATLAB even when parallel situation. Now, in order to conduct multiple MATLAB calculation, I have changed the code back to normal MATLAB code. looks there is a trade-off relation.

However, since your answer does not solve the specific question for this topic, I could not accept your answer. but I really appreciate your suggestions and thanks for your time for discussion here.

Best!

Yu

Walter Roberson 2018-9-22

"Just want know if there is any mistake with my operation"

Yes, it is a mistake to share the GPU between multiple processes in Windows.

A document I was examining recently on how NVIDIA GPU handle multiple processing indicated that task synchronization is expensive in relative terms, and that moving tasks on / off of the GPU is the most expensive operation for their GPU.

When you run two different processes each trying to access the GPU, you are forcing it to run the most expensive operation often.

请先登录，再进行评论。

GPU performance significantly slow when running multiple MATLAB

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

10 个评论
显示 8更早的评论隐藏 8更早的评论

另请参阅

类别

标签

Community Treasure Hunt

GPU performance significantly slow when running multiple MATLAB

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

10 个评论 显示 8更早的评论隐藏 8更早的评论

另请参阅

类别

标签

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

10 个评论
显示 8更早的评论隐藏 8更早的评论