Parallel Computing in Neural Networks is not using all the workers in 2018b?

Question

Eric Klinefelter 2019-1-8

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/438829-parallel-computing-in-neural-networks-is-not-using-all-the-workers-in-2018b

回答： D Hanish 2020-1-17

There was a similar question here, but I'm unable to get the parallel pool to use my CPU cores when using a GPU. My command is:

my_net = train(my_net,Xs,Ts,Xi,Ai,'useParallel','yes','useGPU','yes','showResources','yes');

Yet when starting the pool the response is:

NOTICE: Jacobian training not supported on GPU. Training function set to TRAINSCG.
 
Computing Resources:
Parallel Workers:
  Worker 1 on w541, GPU device #1, Quadro K1100M
  Worker 2 on w541, Unused
  Worker 3 on w541, Unused
  Worker 4 on w541, Unused
  Worker 5 on w541, Unused

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Joss Knight 2019-1-9

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/438829-parallel-computing-in-neural-networks-is-not-using-all-the-workers-in-2018b#answer_355736

I believe this is the designed behaviour. If multiple workers were to share the same GPU, you would get a performance reduction, not an improvement.

4 个评论
显示 2更早的评论隐藏 2更早的评论

Joss Knight 2019-1-12

I am not familiar with the implementation for shallow networks, but for deep learning, even if you filled the GPU memory and gave each CPU the minimum amount of work, the GPU would end up waiting for the CPUs to finish to synchronize each iteration, so the CPUs would just slow things down.

Walter Roberson 2019-1-13

I notice that there is no second GPU being allocated. That leads me to suspect that the Quadro K1100M might be the only GPU in the system. I wonder if it is driving a display? If it is then it would be in WDDM mode, in which case it would need to have short work timeouts, making it necessary to synchronize with the CPUs often compared to the likely total training time. If it is not driving a display and is in TCC mode then that factor is reduced... but of course the time it spends dedicated to processing work from one CPU would be time it was not processing work from a difference CPU.

请先登录，再进行评论。

Answer 2

D Hanish 2020-1-17

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/438829-parallel-computing-in-neural-networks-is-not-using-all-the-workers-in-2018b#answer_410606

You should set "useGPU" to off. It has been some time, you have probably figured that out already. Matlab uses uses one GPU per core and since there is only one GPU, will also only use one core. (pretty much as Walter Roberson said)

On my system (8 core Xeon + 1 GPU) It turns out to be much slower to use one core and GPU with 1 worker than to useParallel alone which gives me 8 workers on 8 real cores. For you, useParallel without useGPU will allow you to use 5 CPUs and Jacobian training. Be careful because Matlab (could be fixed in 2019b) must be restarted before it will use all the cores again.

[net,tr] = train(net,X,T,'UseParallel','yes','useGPU','no','showResources','yes','CheckpointFile','MyCheckpoint','CheckpointDelay',600);