Gradient clipping with custom feed-forward net

10 次查看（过去 30 天）
Christoph Aistleitner
回答： Artem Lensky 2022-12-4
Everytime I am training my custom feed-forward net with 2 inputs and one output( timeseries) with the train(net,....) function:
after ~10 training epochs the value of the gradient reaches the prestet value and the training stops.
Changing the networks architecture is not an option in my case.
Is there a way to implement "gradient clipping" with a feed-forward net?
Or is there any other workaround for the "exploding gradient"?

请先登录，再回答此问题。

采纳的回答

Vineet Joshi
Vineet Joshi 2021-9-1
Hi
The following documentation link will provide you suitable details regarding dealing with exploding gradients in MATLAB.
Gradient Clipping: Opions for training deep learning neural networks.
Gradient Clipping: Algorithm
Hope this helps.
Thanks
  1 个评论
Artem Lensky
Artem Lensky 2022-12-4
The answer you provided is not for a custom loop. See this example https://au.mathworks.com/help/deeplearning/ug/train-network-using-custom-training-loop.html there is the following line
[loss,gradients,state] = dlfeval(@modelLoss,net,X,T);
The question is how to apply clipping to gradients. Is there are standard Matlab function can do this for me or should I implement it myself.

请先登录，再进行评论。

更多回答（1 个）

Artem Lensky
Artem Lensky 2022-12-4
Please check this link that illustrates several examples on how to implement training options that you would usually define via trainingOptions() and use with trainNetwork() but for customs loops. Here is an L2 clipping example given in the link above
function gradients = thresholdL2Norm(gradients,gradientThreshold)
gradientNorm = sqrt(sum(gradients(:).^2));
if gradientNorm > gradientThreshold
gradients = gradients * (gradientThreshold / gradientNorm);
end
end
You might also find this link useful https://au.mathworks.com/help/deeplearning/ug/detect-vanishing-gradients-in-deep-neural-networks.html that discuss detection of vanishing gradients in deep neural networks.

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

产品


版本

R2021a

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by