Use several pools of workers in a parfor loop

Question

gmhs 2019-9-9

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479599-use-several-pools-of-workers-in-a-parfor-loop

评论： gmhs 2019-10-29

Hi all,

I am currently trying to figure out if there is a way to use multiple pools of workers in a parfor loop. More precisely, I have 16 cores on my computer (8 physical cores) and until now I was using a classical for loop to perform 20 fminunc optimisations.

While fminunc already performs multiprocessing, I found out that the 16 cores of my cluster where only used to 30%, therefore I was thinking about using some kind of parfor loop over three pools of 5 workers in order to increase the efficiency of the computation and hopefully reduce the time of the computation.

I searched for a while now but didn't really find a way to do that...

Thanks a lot !

Gabriel

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Pravin Jagtap 2019-9-18

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479599-use-several-pools-of-workers-in-a-parfor-loop#answer_392383

Hello Gabriel,

Yes, most of the MATLAB in-built functions are multithreaded from 2008a Release. Following document may help you in accelerating the problem:

https://www.mathworks.com/help/optim/examples/minimizing-an-expensive-optimization-problem-using-parallel-computing-toolbox.html

A ‘parfor’ will likely to slow the performance since the inbuilt function of optimization is inherently multithreaded and make use of multiple cores calling. For further optimization/speedup we need to dig more into code which uses parfor.

Kind Regards

~Pravin

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Matt J 2019-10-16

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/479599-use-several-pools-of-workers-in-a-parfor-loop#answer_396692

编辑：Matt J 2019-10-16

在 MATLAB Online 中打开

Your terminology is a bit confusing. There is only ever 1 pool of workers, but each worker may use multiple cores and the entire pool will always try to make best use of all available processors. To loop with a pool of 3 workers (so roughly 5 cores per worker), you would do,

parfor (i=1:20,3)
    
    x{i}=fminunc(...);
    
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

gmhs 2019-10-29

Thanks for your answer. I managed to solve this issue some time ago now and indeed used a parfor loop with 3 workers. The problem came from the fact that the default cluster profile allows only one thread per worker, it is therefore necessary to change the value of NumThreads to allow fminunc to use all the CPUs.

请先登录，再进行评论。