how to adjust derivatives of backpropagation according to custom error function

Question

Sergey Ohanyan 2019-1-14

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/439650-how-to-adjust-derivatives-of-backpropagation-according-to-custom-error-function

编辑： Sergey Ohanyan 2019-2-14

I want to implement custom error function to backpropagation algorithm, and enforce to update weights with taking into account output layer outputs.

Network consists of one hidden layer. The inputs are (n,1) vector, outputs have the same dimensions.

Lets say we have given input and target, I want to calculate error function as

E = ((input*output-target)^2)/2

so at each iteration target updates to reach initial target.

updatedTarget = input*output

According to this, derivatives are modified to

dE_dOutput(j) = sum(input .* netOutputLayerOutputs - initialTarget(j)) * input(j);

the code part of backpropagation algorithm is provided below

 
for i = 1:100      
    for j = 1 : length(input)        
        % forward propagation
        netHiddenLayerValues(j) = sum(netHiddenLayerWeights(:,j) .* input(j))  + netBiasValue1 * 1;
        netHiddenLayerOutputs(j) = 1/(1 + exp(-netHiddenLayerValues(j)));
        netOutputLayerValues(j) = sum(netOutputLayerWeights(:,j) * netHiddenLayerOutputs(j)) + netBiasValue2 * 1;
        netOutputLayerOutputs(j) = 1/(1 + exp(-netOutputLayerValues(j)));    
        % Custom target function
        target(j) = input(j) * netOutputLayerOutputs(j);    
        % back propagation for output layer        
        % customized error derivative with respect to output layer outputs
        dE_dOutput(j) = sum(input .* netOutputLayerOutputs - initialTarget(j)) * input(j);
        
        %dE_dOutput(j) = -(target(j) - netOutputLayerOutputs(j));  % this is for standart mse
        % partial derivative of logistic function with respect to network output value
        dOutput_dNetout(j) = netOutputLayerOutputs(j) * (1 - netOutputLayerOutputs(j));    
        % partial derivative of network output with respect to weight
        dNetout_dw(j) = netHiddenLayerOutputs(j);    
        % calculate total error for output layer     
        d_EtotalOut(j) = dE_dOutput(j) * dOutput_dNetout(j) * netHiddenLayerOutputs(j);
        
        % calculate back propagation error for hidden layer
        d_EtotalHidden_dOut(j) = dE_dOutput(j) * dOutput_dNetout(j) * netOutputLayerWeights(j);
        dOut_dNetHidden(j) = netHiddenLayerOutputs(j) * (1 - netHiddenLayerOutputs(j));
        dNetHidden_dw(j) = input(j);        
        d_EtotalHiddenOut(j) = d_EtotalHidden_dOut(j) * dOut_dNetHidden(j) * dNetHidden_dw(j);    
    
        % update weights for hidden layer
        netHiddenLayerWeights(:,j) = netHiddenLayerWeights(:,j) - eta * d_EtotalHiddenOut(j);
    
        % update weights for output layer 
        netOutputLayerWeights(:,j) = netOutputLayerWeights(:,j) - eta * d_EtotalOut(j);
    end
end

The figures above shows how are changes during iterations (5,10,20,50,100 iterations respectively), outputs of net(orange) and updated target (blue). The static curves are inputs (yellow) and initial or real target (magenta).

The issue is that updated at each iteration target doesnot reach real initial target.

Please write if you notice error in algorithm implementation or logic.

The backpropagation algorithm with mse error is described here step-by-step-backpropagation-example.

Thanks.