Parallel computing memory allocation

25 次查看（过去 30 天）
CJ
CJ 2017-6-5
评论： Nagarjuna Manchineni 2017-6-20
I am running code on an HPCC. I have a very large loop so I am trying to split it up. The problem is that I seem to have to give each parallel instance the entire memory reserve for the job leading to too much memory usage.
ClusterInfo.setMemUsage('10gb'); %command from our hpcc
parpool(100)
inside_code_file
Where inside_code_file generates a very large matrix [just under 10 gb] and runs 'parfor' through each cell independently.
When doing it this way, I have to give Matlab 10gb for each of the 100 parallel cores. I'm pretty sure I set up the parfor correctly as there are no `split' warnings.
Is this something I can fix through Matlab or is it on the HPCC side?
Or is this just a fundamental aspect of parallel computing and unavoidable?

请先登录，再回答此问题。

回答（1 个）

Tushar Upadhyay
Tushar Upadhyay 2017-6-16
From the information you provided, It seems that you want to find a better memory efficient way to avoid allocating 10gb of memory to each worker.
Does each worker require a separate copy of the matrix to do their tasks or they can share the same matrix?
  2 个评论
CJ
CJ 2017-6-16
Each worker's input can be a shared matrix, but each worker outputs 1 cell into a new matrix that becomes very large. Does that answer your question?
Nagarjuna Manchineni
Nagarjuna Manchineni 2017-6-20
If the input matrix can be shared, then MATLAB will distribute the array to the respective workers during the runtime and once the run for that particular input is completed, then MATLAB worker returns the data back to client (MATLAB).
So, once verify which kind of variable (sliced, broadcast, temporary, loop, etc..) you are trying to use in 'parfor' and let us know.
Also, see the following documentation link for more troubleshooting steps on variables that are shared across workers from MATLAB:
https://www.mathworks.com/help/distcomp/troubleshoot-variables-in-parfor-loops.html

请先登录，再进行评论。

类别

在 Help Center 和 File Exchange 中查找有关 Parallel for-Loops (parfor) 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by