Parallel computing memory allocation

Question

CJ 2017-6-5

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/343432-parallel-computing-memory-allocation

评论： Nagarjuna Manchineni 2017-6-20

I am running code on an HPCC. I have a very large loop so I am trying to split it up. The problem is that I seem to have to give each parallel instance the entire memory reserve for the job leading to too much memory usage.

ClusterInfo.setMemUsage('10gb');  %command from our hpcc
parpool(100)
inside_code_file

Where inside_code_file generates a very large matrix [just under 10 gb] and runs 'parfor' through each cell independently.

When doing it this way, I have to give Matlab 10gb for each of the 100 parallel cores. I'm pretty sure I set up the parfor correctly as there are no `split' warnings.

Is this something I can fix through Matlab or is it on the HPCC side?

Or is this just a fundamental aspect of parallel computing and unavoidable?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Tushar Upadhyay 2017-6-16

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/343432-parallel-computing-memory-allocation#answer_270961

From the information you provided, It seems that you want to find a better memory efficient way to avoid allocating 10gb of memory to each worker.

Does each worker require a separate copy of the matrix to do their tasks or they can share the same matrix?

2 个评论
显示无隐藏无

CJ 2017-6-16

Each worker's input can be a shared matrix, but each worker outputs 1 cell into a new matrix that becomes very large. Does that answer your question?

Nagarjuna Manchineni 2017-6-20

If the input matrix can be shared, then MATLAB will distribute the array to the respective workers during the runtime and once the run for that particular input is completed, then MATLAB worker returns the data back to client (MATLAB).

So, once verify which kind of variable (sliced, broadcast, temporary, loop, etc..) you are trying to use in 'parfor' and let us know.

Also, see the following documentation link for more troubleshooting steps on variables that are shared across workers from MATLAB:

https://www.mathworks.com/help/distcomp/troubleshoot-variables-in-parfor-loops.html

请先登录，再进行评论。