Trying to include learning rate and momentum in sgdmupdate function under multiple GPUs

1 次查看（过去 30 天）
hyeonjin kim
hyeonjin kim 2022-3-1
评论： hyeonjin kim 2022-4-1
I am working on modifying the example given in
"https://www.mathworks.com/help/deeplearning/ug/train-network-in-parallel-with-custom-training-loop.html?searchHighlight=sgdmupdate%20multiple%20gpu&s_tid=srchtitle_sgdmupdate%2520multiple%2520gpu_3".
The modification is that I am trying to incorporate learning rate and momentum into the sgdmupdate function.
That is,
[dlnet.Learnables,workerVelocity] = sgdmupdate(dlnet.Learnables,workerGradients,workerVelocity, learning rate, momentum);
instead of
[dlnet.Learnables,workerVelocity] = sgdmupdate(dlnet.Learnables,workerGradients,workerVelocity); (line 108 in the example).
However, that modification results in the following error in the case of using 4 GPUs (no error with a single GPU)
Would there be anyone who could help me on this ?
Thanks very much !!!
  3 个评论
显示 1更早的评论
Joss Knight
Joss Knight 2022-3-10
Also make sure all your gradients are finite with something like assert(all(cellfun(@(x)all(isfinite(x(:))),workerGradients))). Sometimes when you mess with the learn rate you can get NaNs and infinities in the gradients.
hyeonjin kim
hyeonjin kim 2022-4-1
Dear Joss,
First of all, I am sorry to get back to you this late.
Thanks very much for your help. I will try to fix the problem according to your comments once our workstation with GPUs becomes available again.
Thanks very much !

请先登录，再进行评论。

请先登录，再回答此问题。

回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 GPU Computing 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by