How can I freeze layers for training a network with multiple outputs and reduce time for the training?

Question

Brian Park 2023-7-12

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/1994833-how-can-i-freeze-layers-for-training-a-network-with-multiple-outputs-and-reduce-time-for-the-trainin

评论： Brian Park 2023-7-15

Since I try to train the model with multiple outputs, I have an issue to use 'deepNetworkDesigner' and its Training option.

So, I follow the direct training method on this example with freezing net's layers using the code below

lgraph = net.layerGraph;
target = 290
for i = 1 : target
    try
        L = freezeWeights(lgraph.Layers(i));
        lgraph = replaceLayer(lgraph,lgraph.Layers(i).Name,L);
    catch
    end
end
net = dlnetwork(lgraph)

I checked WeightLearnRateFactor and BiasLearnRateFactor become zero, which means the layers are frozen.

However, it still takes too much time on the stage of training.

So,

Q1: is this the right way to freeze layers for the training multiple output network?

Q2: How can I reduce time for training, ignoring the layers which are frozen.

Here is the base code from example that I used for network training

[loss,gradients,state] = dlfeval(@modelLoss,net,X,T1,T2);

function [loss,gradients,state] = modelLoss(net,X,T1,T2)
[Y1,Y2,state] = forward(net,X,Outputs=["softmax" "fc_2"]);
lossLabels = crossentropy(Y1,T1);
lossAngles = mse(Y2,T2);
loss = lossLabels + 0.1*lossAngles;
gradients = dlgradient(loss,net.Learnables);
end

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Aniketh 2023-7-12

在 MATLAB Online 中打开

Yeah this a right way to freeze the layers, as long as you are not seeing any drastic changes in the output it shouldn't be any different for the case of predicting multiple outputs.

To further reduce training time and ignore the frozen layers, you can modify the modelLoss function to only compute and backpropagate gradients for the unfrozen layers using the find function in matlab. Here's a sample code for how this will work:

function [loss, gradients, state] = modelLoss(net, X, T1, T2)
    [Y1, Y2, state] = forward(net, X, Outputs = ["softmax" "fc_2"]);
    lossLabels = crossentropy(Y1, T1);
    lossAngles = mse(Y2, T2);
    loss = lossLabels + 0.1 * lossAngles;
    
    % Compute gradients only for unfrozen layers
    unfrozenLayers = find([net.Layers.WeightLearnRateFactor] > 0);
    gradients = dlgradient(loss, net.Learnables(unfrozenLayers));
end

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Brian Park 2023-7-15

在 MATLAB Online 中打开

to test it i freezed last layers (from 253 to 255)

unfrozen_Learnables = [net.Learnables(253,:); net.Learnables(254,:); net.Learnables(255,:)];
gradients = dlgradient(loss, unfrozen_Learnables);

and get the error when running the code in modelLoss function

gradients = dlgradient(loss, unfrozen_layer);

>>

Error using  + 
Arrays have incompatible sizes for this operation.
Error in  +  (line 39)
zdata = matlab.lang.internal.move(xdata) + matlab.lang.internal.move(ydata);
...
Error in adamupdate (line 152)
[p, avg_g, avg_gsq] = deep.internal.networkContainerFixedArgsFun(func, ...
    