Is there any layer defined in matlab for sine activation function? Or else can we define the layer using functionLayer?

Question

BIPIN SAMUEL 2023-12-1

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2054674-is-there-any-layer-defined-in-matlab-for-sine-activation-function-or-else-can-we-define-the-layer-u

评论： BIPIN SAMUEL 2023-12-8

How we can use a layer with sine activation function? Is it possible to use the functionLayer to define a sine activation function or should I define a class for creating the layer as shown in https://in.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-intermediate-layers.html?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Matt J 2023-12-1

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/2054674-is-there-any-layer-defined-in-matlab-for-sine-activation-function-or-else-can-we-define-the-layer-u#answer_1363544

If it will not have any learnable parameters, you can use a functionLayer.

8 个评论
显示 6更早的评论隐藏 6更早的评论

BIPIN SAMUEL 2023-12-8

编辑：Matt J 2023-12-8

在 MATLAB Online 中打开

ok @Matt J, This is the custom layer that I have created:

classdef CoAtten < nnet.layer.Layer  ...
        & nnet.layer.Formattable ... 
        %   & nnet.layer.Acceleratable 
    % properties
    %     % (Optional) Layer properties.
    % 
    %     % Declare layer properties here.
    % end
    properties (Learnable)
        % (Optional) Layer learnable parameters.
        alpha
    end
    % properties (State)
    %     % (Optional) Layer state parameters.
    % 
    %     % Declare state parameters here.
    % end
    % 
    % properties (Learnable, State)
    %     % (Optional) Nested dlnetwork objects with both learnable
    %     % parameters and state parameters.
    % 
    %     % Declare nested networks with learnable and state parameters here.
    % end
    methods
        function layer = CoAtten(NameValueArgs)
            % (Optional) Create a myLayer.
            % This function must have the same name as the class.
            arguments
                NameValueArgs.Name = '';
            end
            name=NameValueArgs.Name;
            layer.Name=name;
            layer.Description="Attention Mechanism based on correlation";
            layer.Type="Correlation Attention";
        end
        function layer = initialize(layer,layout)
            % (Optional) Initialize layer learnable and state parameters.
            if isempty(layer.alpha)
                idx=finddim(layout,"C");
                numChannels = layout.Size(idx);
                layer.alpha=dlarray(zeros(numChannels));
            end
        end
        
        function Z = predict(layer,varargin)
            % Forward input data through the layer at prediction time and
            % output the result and updated state.
            %
            X=varargin;
            filtersize=ones(1);
            nc=ones(1);
            numfilters=ones(1);
            sz=[filtersize,nc,numfilters];
            numout=prod(filtersize)*numfilters;
            numin=prod(filtersize)*numfilters;
            weights=initializeGlorot(sz,numout,numin);
            bias=dlarray(zeros(nc));
            query=dlconv(X,weights,bias,WeightsFormat='CUT');
            weights=initializeGlorot(sz,numout,numin);
            key=dlconv(X,weights,bias,WeightsFormat='CUT');
            weights=initializeGlorot(sz,numout,numin);
            value=dlconv(X,weights,bias,WeightsFormat='CUT');
            X=stripdims(X);
            query=stripdims(query);
            key=stripdims(key);
            value=stripdims(value);
            X_zscore=zscore(X);
            X_transpose=permute(X_zscore,[1 3 2]);
            X_flatten=permute(X_transpose,[2 3 1]);
            query_flatten=permute(query,[2 4 1 3]);
            query_energy=pagemtimes(X_flatten,query_flatten);
            query_energy=permute(query_energy, [3 1 4 2]);
            key_flatten=permute(key,[2 4 1 3]);
            key_energy=pagemtimes(X_flatten,key_flatten);
            key_energy=permute(key_energy,[3 1 4 2]);
            query_energy=permute(query_energy,[2 3 1]);
            key_energy=permute(key_energy,[2 4 1 3]);
            energy=pagemtimes(query_energy,key_energy);
            energy=permute(energy,[3 1 4 2]);
            energy=permute(energy, [2 1 3]);
            attention=softmax(energy,'DataFormat',"UCU");
            value_flatten=permute(value,[2 3 1]);
            out=pagemtimes(value_flatten,attention);
            out=permute(out,[2 1 3]);
            Alpha=layer.alpha;
            Z=Alpha*out+X;
            Z=dlarray(Z,"CBT");
        end
    end
end

This is the layer that I have used for the validity check using "checkLayer".

Matt J 2023-12-8

在 MATLAB Online 中打开

CoAtten.m

Does this make any more sense?

layer=CoAtten(Name="atten");
validInputSize = [1 14 1024];
layout = networkDataLayout(validInputSize,"CBT");
layer = initialize(layer,layout);
checkLayer(layer,layout,ObservationDimension=2)
Skipping GPU tests. No compatible GPU device found.
 
Skipping code generation compatibility tests. To check validity of the layer for code generation, specify the CheckCodegenCompatibility and ObservationDimension options.
 
Running nnet.checklayer.TestLayerWithoutBackward
..
================================================================================
nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=one) was filtered.
    Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
================================================================================
nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=multiple) was filtered.
    Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
....
================================================================================
Verification failed in nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=one).
    ----------------
    Test Diagnostic:
    ----------------
    Test failure may be due to the layer not being initialized. If the layer is not initialized, then initialize it by calling its initialize method.
    ---------------------
    Framework Diagnostic:
    ---------------------
    The function 'predict' threw an error:
    Undefined function 'initializeGlorot' for input arguments of type 'double'.
    
    Error in CoAtten/predict (line 57)
                weights=initializeGlorot(sz,numout,numin);
    ------------------
    Stack Information:
    ------------------
    In /MATLAB/toolbox/nnet/cnn/+nnet/+checklayer/TestLayerWithoutBackward.m (TestLayerWithoutBackward.predictDoesNotError) at 26
================================================================================
.
================================================================================
Verification failed in nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=multiple).
    ----------------
    Test Diagnostic:
    ----------------
    Test failure may be due to the layer not being initialized. If the layer is not initialized, then initialize it by calling its initialize method.
    ---------------------
    Framework Diagnostic:
    ---------------------
    The function 'predict' threw an error:
    Undefined function 'initializeGlorot' for input arguments of type 'double'.
    
    Error in CoAtten/predict (line 57)
                weights=initializeGlorot(sz,numout,numin);
    ------------------
    Stack Information:
    ------------------
    In /MATLAB/toolbox/nnet/cnn/+nnet/+checklayer/TestLayerWithoutBackward.m (TestLayerWithoutBackward.predictDoesNotError) at 26
================================================================================
.. ....
================================================================================
nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=single,Device=cpu) was filtered.
    Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
================================================================================
nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=double,Device=cpu) was filtered.
    Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
..... ..
================================================================================
nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=one) was filtered.
    Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
================================================================================
nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=multiple) was filtered.
    Test Diagnostic: Test did not run because 'predict' threw an error.
================================================================================
.
Done nnet.checklayer.TestLayerWithoutBackward
__________

Failure Summary:

     Name                                                                                                Failed  Incomplete  Reason(s)
    =================================================================================================================================================
     nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=one)                   X       Filtered by assumption.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/formattableLayerPredictIsFormatted(Observations=multiple)              X       Filtered by assumption.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=one)                        X                 Failed by verification.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/predictDoesNotError(Observations=multiple)                   X                 Failed by verification.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=single,Device=cpu)                 X       Filtered by assumption.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/predictIsConsistentInType(Precision=double,Device=cpu)                 X       Filtered by assumption.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=one)                      X       Filtered by assumption.
    -------------------------------------------------------------------------------------------------------------------------------------------------
     nnet.checklayer.TestLayerWithoutBackward/backwardPropagationDoesNotError(Observations=multiple)                 X       Filtered by assumption.
Test Summary:
	 16 Passed, 2 Failed, 6 Incomplete, 10 Skipped.
	 Time elapsed: 1.3986 seconds.

BIPIN SAMUEL 2023-12-8

Rectified Thank You very much @Matt J

请先登录，再进行评论。

Is there any layer defined in matlab for sine activation function? Or else can we define the layer using functionLayer?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

8 个评论
显示 6更早的评论隐藏 6更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

Is there any layer defined in matlab for sine activation function? Or else can we define the layer using functionLayer?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

8 个评论 显示 6更早的评论隐藏 6更早的评论

更多回答（0 个）

另请参阅

类别

标签

产品

版本

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

8 个评论
显示 6更早的评论隐藏 6更早的评论