Trying to include learning rate and momentum in sgdmupdate function under multiple GPUs

1 次查看（过去 30 天）

hyeonjin kim 2022-3-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1660885-trying-to-include-learning-rate-and-momentum-in-sgdmupdate-function-under-multiple-gpus

评论： hyeonjin kim 2022-4-1

I am working on modifying the example given in

"https://www.mathworks.com/help/deeplearning/ug/train-network-in-parallel-with-custom-training-loop.html?searchHighlight=sgdmupdate%20multiple%20gpu&s_tid=srchtitle_sgdmupdate%2520multiple%2520gpu_3".

The modification is that I am trying to incorporate learning rate and momentum into the sgdmupdate function.

That is,

[dlnet.Learnables,workerVelocity] = sgdmupdate(dlnet.Learnables,workerGradients,workerVelocity, learning rate, momentum);

instead of

[dlnet.Learnables,workerVelocity] = sgdmupdate(dlnet.Learnables,workerGradients,workerVelocity); (line 108 in the example).

However, that modification results in the following error in the case of using 4 GPUs (no error with a single GPU)

Would there be anyone who could help me on this ?

Thanks very much !!!

3 个评论
显示 1更早的评论隐藏 1更早的评论

Joss Knight 2022-3-10

Also make sure all your gradients are finite with something like assert(all(cellfun(@(x)all(isfinite(x(:))),workerGradients))). Sometimes when you mess with the learn rate you can get NaNs and infinities in the gradients.

hyeonjin kim 2022-4-1