Matlab 2013a GPU memory leak

2 次查看（过去 30 天）
Michael
Michael 2013-5-27
I have been running some very long loops (millions of iterations) where , in each iteration, I call a few CUDA kernels via feval using pre-allocated arrays of fixed size. I noticed that the host memory grows linearly with the number of iterations and in the end matlab crashes. While I was trying to isolate the problem I found out the following: - Using feval to call a CUDA kernel , you have to have all the arguments of the function already cast as gpuArray's, even if you pass scalar variables. This also applies to functions like gpuArray.rand or randn:
n = 1e4;
for i = 1:1e6
out = gpuArray.rand(n,1,'single');
end
The above code causes the host memory to grow for the duration of the execution (about 100Mb per 250K iterations) If instead of n=1e4; you write n=gpuArray(1e4); the subsequent loop does not cause the memory to grow. I also found out the the above loop executes much faster when n is in the host memory vs. when n is a gpuArray (about 3 times faster).
-Even more puzzling is the following example:
x = gpuArray.rand(1e4,1,'single');
for i = 1:1e6
out = sqrt(x);
end
The above loop does not cause MATLAB's memory footprint to grow. However, if we change sqrt(x) with sqrt(1./x) then we get the memory blowup again. I am using MATLAB 2013a 64-bit on windows 7 professional. My video card is a gtx 650 2gb. Thanks in advance for any insights.
  3 个评论
显示 1更早的评论
Michael
Michael 2013-5-28
Hi Ben, I tried the following code
n = 1e3;
for i = 1:1e8
out = gpuArray.rand(n,1,'single);
disp(i)
end
After about 30 million iterations, MATLAB's memory footprint grew to about 14gb (my PC has 16gb of RAM). MATLAB started to pause at certain times because of hard drive activity. I didn't manage to crash it, memory usage was around 13.9-14.5GB but the system was almost unresponsive due to hard drive activity. I need to find again the code that produced the crash. I remember though that the error message was something about Java heap space.
If I try
n = gpuArray(1e3);
for i = 1:1e8
out = gpuArray.rand(n,1,'single);
disp(i)
end
then MATLAB's memory does not grow! However, it is quite a bit slower than the first loop (about 2-3 times)! That's what I meant when I said that I needed to cast scalar input variables as gpuArrays.
I used Windows' task manager to monitor MATLAB's memory usage. My driver version is 314.07. I installed the latest version (320.18); it didn't make a difference. Thanks for your help, Michael
Ben Tordoff
Ben Tordoff 2013-6-4
Thanks Michael, you are indeed right and this appears to be a bug introduced in R2013a. There is no realistic work-around I can provide right now, but I will post an update here once I have some more helpful suggestions.
The reason why the memory does not leak with certain calls is that they force a synchronisation event (in your first example, SQRT can error so has to wait to see if the error was hit; in the second the scalar parameter "n" has to be transferred back to host memory, which also causes a sync). You could achieve the same by inserting a "wait(gpu)" after every call:
gpu = gpuDevice();
for ii=1:1e8
out = gpuArray.rand(1e3,1,'single');
wait(gpu);
disp(i)
end
but that will also slow things down a lot and is hardly a practical solution.

请先登录，再进行评论。

请先登录，再回答此问题。

采纳的回答

Ben Tordoff
Ben Tordoff 2013-6-18
Hi Michael, could you read the following bug-report and try the workaround it contains (being careful about the backing-up step!):
http://www.mathworks.com/support/bugreports/954239
If this does not fix the problem, please let me know as soon as possible.
Ben
  1 个评论
Michael
Michael 2013-6-21
Hi Ben, I tried a piece of code that I have been working on; Without the patch, MATLAB ends up consuming 7gb of RAM (after 4mil iterations). With the patch MATLAB ends up consuming 1.3gb. At the beginning of the loop MATLAB was using 0.5gb. For simpler pieces of code I have (like the ones I posted on my question) there is no memory growth. Thanks for the patch, it basically solves completely the problem for me.
Michael

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 GPU Computing 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by