Issue implementing custom fully connected layer - weights converging to 0

Question

Kenny Chou 2021-1-26

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/727013-issue-implementing-custom-fully-connected-layer-weights-converging-to-0

回答： Anshika Chaurasia 2021-2-11

I'd like to implement a custimized version of the fully connected layer. Before that, I'd like to make sure I can replicate the default FullyConnectedLayer that comes with Matlab. While my implementation did not return errors, the weights the network learned were unusually small. In my simulation, the default FullyConnectedLayer returned weights mostly between [1,0]. My implementation returned weights around 0 (e.g., 10^-40). Can someone help explain why this is happening?

classdef CustomFCLayer < nnet.layer.Layer
    % Custom fully connected layer, constrain weights to be positive.
    properties (Learnable)
        % Layer learnable parameters
        Weights
        Bias
    end
    
    methods
        function layer = CustomFCLayer(prev_units,num_units,name,initWeights) 
            % layer = CustomFCLayer(numInputs,name) creates a
            % fully connected layer 
            layer.Name = name;
            layer.Description = 'Fully connected layer';
            
            if ~exist('initWeights','var')
                std = sqrt(2/(num_units+prev_units));
                layer.Weights = std*rand([num_units prev_units]);
            else
                assert(size(initWeights,1)==num_units) 
                assert(size(initWeights,2)==prev_units) 
                layer.Weights = initWeights;
            end
            
            layer.Bias = zeros([num_units 1]);
        end
        
        function Z = predict(layer, X)
            % Forward input data through the layer at prediction time and
            % output the result
            Z = (layer.Weights)*X+layer.Bias;
        end
    end
end