Hi Ilias,
If I understood the query correctly, you're trying to apply a custom function between layers in a neural network. You can achieve this using MATLAB's 'functionLayer': https://www.mathworks.com/help/deeplearning/ref/nnet.cnn.layer.functionlayer.html which can be used to apply specific function to the layer input.
You can refer to the below example script which defines a simple CNN with an intermediate 'functionLayer' that applies the softsign operation "f(x) = x / (1 + |x|)".
% Network with a custom softsign activation
layers = [
imageInputLayer([28 28 1], 'Name', 'input')
convolution2dLayer(5, 20, 'Name', 'conv')
%%
% custom function
functionLayer(@(X) X ./ (1 + abs(X)), 'Name', 'softsign', 'Description', 'softsign')
%%
maxPooling2dLayer(2, 'Stride', 2, 'Name', 'maxpool')
fullyConnectedLayer(10, 'Name', 'fc')
softmaxLayer('Name', 'softmax')
];
net = dlnetwork(layers);
sampleInput = rand(28, 28, 1, 1);
dlX = dlarray(sampleInput, 'SSCB');
dlY = predict(net, dlX);
disp("Output of the network:");
disp(extractdata(dlY));
Hope this helps!
