The gradient of mini batches

4 次查看（过去 30 天）
MAHSA YOUSEFI
MAHSA YOUSEFI 2020-11-23
评论： Mahesh Taparia 2020-12-21
Hi there.
I need your confimation or rejection for this question...
In following code, if the minibatch size is h,
[grad,loss] = dlfeval(@modelGradients,dlnet,dlX_miniBatch,Y_miniBatch);
the grad is the average of gradients of loss over h samples? Does it calculate dradients automatically and at the end with:
grad = 1/h * sum_i=1:h (\nabla loss(y_i,yHat_i)) ??
Following this question, for computing the total loss and geadient (for a full batch), does we should take avarage of losses and averages of gradients (averaging with the number of batches, say 1000 batches each with h size)??

请先登录，再回答此问题。

采纳的回答

Mahesh Taparia
Mahesh Taparia 2020-12-14
Hi
The function dlfeval evaluate the custom deep learning models. The loss are calculated based on what has been defined in modelGradients function. So if you are calculating the average loss in this function, then it will return the averaged one. For example, consider this modelGradient function, it is calculating the average cross entropy loss, so it will return the average loss. The gradients are calculated with respect to the loss function defined in for the network.
  2 个评论
MAHSA YOUSEFI
MAHSA YOUSEFI 2020-12-19
In the example you mentioned, there is a mistake.
function [gradients, loss] = modelGradients(parameters, dlX, T)
% Forward data through the model function.
dlY = model(parameters,dlX);
% Compute loss.
loss = crossentropy(dlX,T);
% Compute gradients.
gradients = dlgradient(loss,parameters);
end
dlY must be feed to crossentropy!
Mahesh Taparia
Mahesh Taparia 2020-12-21
Yeah, crossentropy loss will be calculated between dlY and T. The documentation page will be updated.

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Sequence and Numeric Feature Data Workflows 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by