parallel computing toolbox - memory issues - possibly leak
4 次查看(过去 30 天)
显示 更早的评论
I am seeing a problem - which has the appearance of a memory leak in PCT.
I have a function which I am running in parallel - I have tried using parfeval and spmd batching - both with the same result. the function runs fine when being computed serially 'in-process', but when it is executed using PCT the memory footprint of the worker nodes gradually increases until it fails.
The first set of tests was on a machine with 32GB RAM and 6 worker nodes. Before long all memory is consumed by the MatLab processes - I repeated on machine with 256 GB - with the same result - all 256 GB consumed. a back of the envelope calculation suggests that the memory required by the function directly should be more like 1% of this.
I looked in more detail at what is going on wrt memory. The JVM max memory is set to around 1GB (which is much greater than needed) - if I put tracing inside the worker nodes - getting info from the java runtime and also the matlab 'memory()' function. I also traced the contents of the function and base workspaces - nothing obviously big or wrong.
the tracing on the worker nodes confirms jvm max memory as 1GB, and the jvm used memory is around 50 MB (per worker) - but the 'MatLab used memory' grows to many GB per process (from MatLab 'memory' function - which agrees with task manager stats).
I took the function which is being evaluated apart - and found that this problem occurs when calling the MatLab inv() function in the function which is being parallellized using PCT (several functions deep) - and whatever is going on - the 'leaking' memory is apparently not being managed by the jvm. I have avoided using the inv() function using a workaround for test purposes - but that isn't a viable log-term solution.
Could anyone suggest what to do next.
Andrew
3 个评论
Walter Roberson
2018-6-4
inv() is generally recommended against for numeric stability purposes (quite apart from any memory leak.) Would it be possible to replace the inv() with using \ (mldivide) ?
回答(1 个)
Jan
2018-6-4
Without some code, which reproduces the problem it is hard to guess, what's going on. While it is recommended at all to avoid using inv, because there are really very few cases where this is needed for numerical computations, I'm sure that it does not interfere with the memory used by Java.
How do you determine the memory usage of Matlab? If you create a lot of variables, the operating system delivers memory to Matlab, but it would be a waste of time to release the memory, when it is not used anymore. Therefore the OS cleans up the memory only, if it is requested later to save time. This might look like Matlab "consumes" a lot of memory, although most of the memory is ready to be reassigned. So please explain exactly, which tool you are using and what you observe to get the impression, that memory is leaked.
另请参阅
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!