Hi Chris,
As you have mentioned that one possible workaround is to use leaky ReLU activation function as NARX network's hidden layer transfer function. Consider following steps to replace tansig transfer function with leaky ReLU:
- Generate a MATLAB code of the NARX net. Refer command-line function.
- Define leaky ReLU function in the generated MATLAB code as:
% ===== MODULE FUNCTIONS ========
% Leaky ReLU Transfer Function
function a = leakyrelu_apply(n,scale,~)
if n>=0
a = n;
else
a = scale*n;
end
end
- In the script within myNeuralNetworkFunction(X,Xi,~) do following changes:
% Time loop
for ts=1:TS
...
% Layer 1
a1 = leakyrelu_apply(repmat(b1,1,Q) + IW1_1*tapdelay1 + IW1_2*tapdelay2,0.01);
....
end
You could also try Glorot (or Xavier) or He Initialization to initialize weights and bias. Refer to link for implementation of Glorot Initialization.
