MEX memory leak - is this a bug?

Question

Ayal 2013-9-2

2
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/86194-mex-memory-leak-is-this-a-bug

评论： Mateusz Pieniak 2017-5-3

I tried reporting this as a bug, with no response so far. Before I try a little harder, I want to make sure this is indeed a bug. I asked a few people, here & other places, and no one can explain this:

mxGPUArray * tmp=mxGPUCopyFromMxArray(prhs[0]);
plhs[0]=mxGPUCreateMxArrayOnGPU(tmp);
mxGPUDestroyGPUArray(out);

This code causes memory leaks (at least in Matlab 2013a, compiled with Visual Studio 2010). I run the following on loop in matlab:

foo=gpuArray.randn(1000);
res=MyMex(foo);
gpuDevice

And I see that with every iteration, there is less free memory in the GPU device. Correct me if i'm wrong, but this SHOULD NOT be happening. Matlab should free the values previously stored by "res", but It doesn't seem to.

NOTE: This is not bug 954239.

I've been working on this problem for two weeks, with no luck. There's always some memory leak. Does anyone have any idea what's going on? Did I miss something?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Edric Ellis 2013-9-4

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/86194-mex-memory-leak-is-this-a-bug#answer_95833

在 MATLAB Online 中打开

It appears as though there's a problem with mxGPUCopyFromMxArray. Please could you try the following workaround:

/**
 * Replacement for mxGPUCopyFromMxArray
 */
mxGPUArray * mxGPUCopyFromMxArray2(mxArray const * mx) {
    if (mxIsGPUArray(mx)) {
        mxGPUArray const * tmp = mxGPUCreateFromMxArray(mx);
        mxGPUArray * returnValue = mxGPUCopyGPUArray(tmp);
        mxGPUDestroyGPUArray(tmp);
        return returnValue;
    } else {
        return const_cast<mxGPUArray *>(mxGPUCreateFromMxArray(mx));
    }
}

4 个评论
显示 2更早的评论隐藏 2更早的评论

Ayal 2013-9-8

在 MATLAB Online 中打开

This mostly works! I still see some odd behavior as far as the memory goes, but I guess it's just something I don't understand related to the memory managment.

using "clear dumdum" actually releases memory, and after the 2nd time I run the code, memory doesn't disappear anymore. However, using "clear foo2" doesn't release any memory. But all the memory returns after clearing all gpuArrays, but it seems like the major problems are resolved.

I also tried

foo=foo2;
testMyMex;
clear foo;

This won't release memory either. I guess there is more to be fixed. But your solution should be a big help to me. Thank you!

Edric Ellis 2013-9-9

It's possible that the remaining memory usage behaviour might simply be artefacts of the allocation pooling.

请先登录，再进行评论。

Answer 2

Edric Ellis 2013-9-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/86194-mex-memory-leak-is-this-a-bug#answer_95687

在 MATLAB Online 中打开

I think this is nothing to do with the MEX interface, and instead to do with the way MATLAB tries to recycle memory allocations on the device, as demonstrated by the following example:

% First, check the free memory on the device
>> dev = gpuDevice(); dev.FreeMemory
ans =
     5.5265e+09
% Next, make 100 allocations of 8MB
>> for idx = 1:100, c{idx} = gpuArray.ones(1000); end
>> dev.FreeMemory
ans =
     4.7138e+09
% Now, clear all but one of those and check the free memory
>> c(2:end) = [];
>> dev.FreeMemory
ans =
     4.7138e+09
% Clear all arrays, and check free memory again
>> clear c
>> dev.FreeMemory
ans =
     5.5265e+09

You can see here that freeing some arrays doesn't change the amount of free memory on the device, but when all arrays have been freed then all the memory is returned.

6 个评论
显示 4更早的评论隐藏 4更早的评论

Ayal 2013-9-3

编辑：Ayal 2013-9-3

在 MATLAB Online 中打开

first of all, WOW. 5.5 GB free memory? I'm jealous. not that I really need one, but It sure would help with the memory leaks.

On another note, It doesn't work quite the same for me (what version of MATLAB are you using?)

>> dev = gpuDevice(); dev.FreeMemory
ans =
     1.8480e+09
>> for idx = 1:100, c{idx} = gpuArray.ones(1000); end
>> dev.FreeMemory
ans =
     1.0354e+09
>> c(2:end) = []
c = 
      [1000x1000 gpuArray]
>> dev.FreeMemory
ans =
     1.2954e+09
>> clear c
>> dev.FreeMemory
ans =
     1.8480e+09

Also:

>> dev=gpuDevice;
>> dev.FreeMemory
ans =
     1.8477e+09
>> testMyMex
ans =
     1.6529e+09
>> testMyMex
ans =
     1.4582e+09

Not sure why I lose more memory the 2nd time too, but if I run the script again and again, it doesn't leak more memory. but let's continue:

>> foo4=foo;
>> testMyMex
ans =
     1.4581e+09
>> numel(foo4)
ans =
      14520000

no memory lost (well, a little bit. seems like a tenth of a megabyte - my guess is that it's actually less, you know - because of the exponent, we get approximate values, rounded). How come? was supposed to lose sizeof(double)*numel(foo4). Continuing:

>> clear foo4
>> dev.FreeMemory
ans =
     1.4581e+09

Where is my memory? I didn't get it back. >> clear foo4 >> dev.FreeMemory

ans =
     1.4581e+09
>> clear foo3
>> dev.FreeMemory
ans =
     1.4581e+09
>> clear foo
>> dev.FreeMemory
ans =
     1.4581e+09
>> clear foo2
>> dev.FreeMemory
ans =
     1.8476e+09

finally, I got my memory back after clearing all gpuArrays. But here's the thing. My code runs a lot of loops, and uses gpuArrays a lot. I use "clear" when i'm done with certain variables, but what use is it if it doesn't clear these certain variables? why do I need to clear ALL gpuArrays, even though most of them have different values from one another, and thus aren't used for lazy copies?

Correct me if i'm wrong (really, if I misunderstood something, please correct me), but that's not how it's supposed to be. (my script, tests the MEX on big chunks of data, so the memory loss is appropriate, but the fact that I don't get the memory back until I clear ALL gpuArrays isn't).

foo, foo2 and foo3 are variables used to test the MEX. I'm currently using a different code from the one posted here, but it's the same thing - create, destroy everything, still leaking memory.

EDIT: I've tried it with the code I posted here, and it's even worse. This is because the code I used for the example in this post doesn't return LHS, but changes the value of an input in-place (using const_cast<mxGPUArray *> to remove the const and overwrite values). The code I posted here is even worse. I clear ALL variables, and I still have missing memory. This seems to be a MEX issue. But additionally, from your post I've learned that indeed there is also a problem with the way MATLAB recycles memory. At least to me it seems like a problem - note that when I ran it, there were 550+ megabytes for a 1000*1000 array. meaning 550 million bytes for 1 million elements of type double, meaning 8,000,000 bytes. WAY too much. This makes no sense to me.

Ayal 2013-9-3

编辑：Ayal 2013-9-3

在 MATLAB Online 中打开

I don't want to use feval because I need streams, and (as I understand it), feval won't run CUDA code on the host, only on the device (only code begining with _global_). Using streams allows for a much faster code than using a Matlab loop with feval (I need the loop because some part of the computation has to be sequential)

Ignore the in-place editing. Here is the MEX for the code I posted in the question:

#include "mex.h"
#include "gpu/mxGPUArray.h"
void mexFunction(int nlhs, mxArray *plhs[],
                 int nrhs, mxArray const *prhs[])
{
    mxGPUArray * out=mxGPUCopyFromMxArray(prhs[0]);
    plhs[0] = mxGPUCreateMxArrayOnGPU(out);
mxGPUDestroyGPUArray(out);
}

and the script I use to run it:

foo2=gpuArray.randn([7 7 1000 1000]);
tic;
dumdum=MexMemoryTrack(foo2);
toc;

I run this a few times (for a more dramatic effect, increase the size of foo2). and the amount of memory decreases. when there isn't enough memory for the next time (according to gpuDevice), I run it again, and suddenly the amount of memory increases (a little). Then I run it one more time - and I get an error - not enough memory.

Ayal 2013-9-3

And thank you for your help. (Just to be sure: do you get the same problem, or am I just unlucky and need to update every software related to this in hopes that the problem is some corrupted file?)

Mateusz Pieniak 2017-5-3