GPU computations slowing down

Question

Peter 2017-1-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/319156-gpu-computations-slowing-down

评论： Joss Knight 2017-1-8

Within a loop, my code carries out some iterative solving procedure on the GPU. With every loop iteration, the time taken by the solving procedure increases. There doesn't seem to be a memory leak: gpuDevice consistently reports approximately the following:

TotalMemory: 1.2885e+10
AvailableMemory: 5.6773e+09

However, when I pause the loop for a while by means of a debug stop, after relaunching again, computations are up to speed again, after which they start slowing down again...

I'm running matlab2016b on Windows 10, using a Nvidia Titan X (Pascal) GPU. Any suggestions for this problem?

3 个评论
显示 1更早的评论隐藏 1更早的评论

Jan 2017-1-5

Without seeing the code or knowing any details, it is hard to guess, what's going on. Did you control the temperature of the graphics board?

Peter 2017-1-6

thanks Jan, I already started to be suspicious about temperature and monitored it. After launching the matlab script, temperature starts rising, GPU performance starts being capped by Temperature. (See performance monitor screen shot below) After reaching 95(!) degrees, the crash occurred again. So we do have strong indications for a cooling issue... What does this mean?

a) I got myself a wrong system set-up, need better cooling ?
b) this GPU card doesn't suit my needs, perhaps go for one of the Keplers (more expensive for similar memory)?

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Peter 2017-1-6

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/319156-gpu-computations-slowing-down#answer_249595

Monitoring the GPU performance revealed that most probably the temperature is causing the issue: Slowing down of performance goes with rising of temperature and performance is capped by temperature.

Crash of the GPU occurred when GPU reached 95 degrees...

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Joss Knight 2017-1-8

Most people who report issues like this are simply seeing an issue of asynchronous execution. They see fast times until the kernel execution queue is full, and then they start seeing the actual complete computation times.

To avoid this try using wait(gpuDevice) to add synchronisation points to your code.

请先登录，再进行评论。