How to extract partial derivatives of some specific layer in the back-propagation of a deep learning model?

Question

SC 2019-11-22

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/492660-how-to-extract-partial-derivatives-of-some-specific-layer-in-the-back-propagation-of-a-deep-learning

评论： SC 2019-11-27

在 MATLAB Online 中打开

Say I have a deep learning model, and after training I call this model net.

When I input some images into net, I want to have the partial derivatives

, where h are the outputs of the relu1 layer (i.e.

) and θ are the parameters of all trainable weights of the layers before relu1.

You can see that h (i.e. the output of relu1) will have a size of

. I write the size of the training weights before relu1 as

, where

would be the set of all trainable parameters of the layers before relu1. Therefore

should have the size of

.

How can I get

in the code? Many thanks!

My current code

%% Load Data
digitDatasetPath = fullfile(matlabroot,'toolbox','nnet','nndemos', ...
    'nndatasets','DigitDataset');
imds = imageDatastore(digitDatasetPath, ...
    'IncludeSubfolders',true, ...
    'LabelSource','foldernames');
numTrainFiles = 50;
[imdsTrain,imdsValidation] = splitEachLabel(imds,numTrainFiles,'randomize');
%% Define Network Architecture
inputSize = [28 28 1];
numClasses = 10;
layers = [
    imageInputLayer(inputSize)
    convolution2dLayer(5,20,'Name','conv1')
    batchNormalizationLayer('Name','bn1')
    reluLayer('Name','relu1')
    fullyConnectedLayer(numClasses,'Name','fc2')
    softmaxLayer('Name','softmax')
    classificationLayer];
%% Train Network
options = trainingOptions('sgdm', ...
    'MaxEpochs',4, ...
    'ValidationData',imdsValidation, ...
    'ValidationFrequency',30, ...
    'Verbose',false, ...
    'Plots','training-progress');
net = trainNetwork(imdsTrain,layers,options);

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Dinesh Yadav 2019-11-26

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/492660-how-to-extract-partial-derivatives-of-some-specific-layer-in-the-back-propagation-of-a-deep-learning#answer_403385

Hi

Kindly go through the following link and examples in it.

https://www.mathworks.com/help/deeplearning/ref/dlarray.dlgradient.html

After the reluLayer command you can use dlgradient to compute partial derivatives on the outputs of relu layer.

Hope it helps.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Dinesh Yadav 2019-11-27

I dont think there is a way to do it with dlgradient without using loops . If you want to do it without using loops you will have to write your own custom gradient function.

SC 2019-11-27

I think something like the jacobian() would help.

jacobian() works for the simple rosenbrock() case, but I don't think it works for the deep learning objects...

I will use a for-loop with dlgradient() then (but I need to loop over 10000+ times since the output size of my desired layer is over 10000, so it will waste many computed re-usable values in the back-propagation and become much more time comsuming than the theorectical computational time). Thank you for your help.

请先登录，再进行评论。

How to extract partial derivatives of some specific layer in the back-propagation of a deep learning model?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

3 个评论
显示 1更早的评论隐藏 1更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

How to extract partial derivatives of some specific layer in the back-propagation of a deep learning model?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

回答（1 个）

3 个评论 显示 1更早的评论隐藏 1更早的评论

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

3 个评论
显示 1更早的评论隐藏 1更早的评论