How to customize Neural Networks' activation function

93 次查看（过去 30 天）
Shahar Hochma
Shahar Hochma 2016-11-18
回答： km y 2023-9-8
Hi, I would like to implement, using Matlab, a neural network with 3 hidden layers, each using ReLU activation function. How can i do this?
Currently, I know i can set the activation function using:
net.layers{i}.transferFcn = reluLayer();
But this only allows to set a specific type of function that is predefined (like logsig), but ReLU is not one of those functions.
Is there a way to change the layer to the ReLU layer? Thanks

请先登录，再回答此问题。

回答（6 个）

Darío Pérez
Darío Pérez 2017-10-24
As far as I am concern, you can use the predefined function 'poslin' (which is a ReLU):
net.layers{i}.transferFcn = 'poslin';
but "other differentiable transfer functions can be created and used if desired": Multilayer Neural Network Architecture.
Not sure how discontinuity at x=0 would affect training stage. In addition, recent articles state that ReLU should be used for regression problems but it achieves worst results than 'tansig' or 'logsig' in one of my examples. Has anyone any thoughts/conclusions in this regard?
Regards!
daniel
daniel 2017-11-30
its useful for deeper nets so depends on your # layers, it's mostly for minimizing vanishing/exploding gradients
  1 个评论
Jan
Jan 2019-1-21
编辑：Jan 2019-1-21
[MOVED from flags] Flagged by Amine Bendali on 18 Jan 2019 at 22:31.
i want to check it later
[Please use flags only to inform admins and editors about messages, which should be reviewed. Thanks]

请先登录，再进行评论。

peter chevo
peter chevo 2018-8-31
编辑：peter chevo 2018-8-31
1. Copy folder and file of C:\Program Files\MATLAB\MATLAB Production Server\R2015a\toolbox\nnet\nnet\nntransfer\ such as +tansig and tansig.m to current path 2. edit file name such as tansig.m is my_transfer.m 3. edit folders name such as +tansig is +my_transfer 4. edit last line in apply.m to your formula equation
  1 个评论
Abdelwahab Afifi
Abdelwahab Afifi 2021-3-3
This method doesn't work. Because each Activation function has its own files with its own sturcture/values/equation.

请先登录，再进行评论。

Maria Duarte Rosa
For Deep Learning networks one can create a custom activation layer using:
https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layers.html
David Willingham
David Willingham 2022-5-19
Hi,
@Maria Duarte Rosa gave a good answer on how to create a custom activation layer by visiting this page: Define Custom Deep Learning Layers
This extended answer is aimed at addressing how to define a "Relu" function in MATLAB.
With MATLAB's current Deep Learning framework, ReLu function is a standard layer you can define.
Here is an example:
Create a ReLU layer with the name 'relu1'.
layer = reluLayer('Name','relu1')
layer =
ReLULayer with properties:
Name: 'relu1'
Include a ReLU layer in a Layer array.
layers = [ ...
imageInputLayer([28 28 1])
convolution2dLayer(5,20)
reluLayer
maxPooling2dLayer(2,'Stride',2)
fullyConnectedLayer(10)
softmaxLayer
classificationLayer]
layers =
7x1 Layer array with layers:
1 '' Image Input 28x28x1 images with 'zerocenter' normalization
2 '' Convolution 20 5x5 convolutions with stride [1 1] and padding [0 0 0 0]
3 '' ReLU ReLU
4 '' Max Pooling 2x2 max pooling with stride [2 2] and padding [0 0 0 0]
5 '' Fully Connected 10 fully connected layer
6 '' Softmax softmax
7 '' Classification Output crossentropyex
For more information on ReLu layer see this page: reluLayer
  3 个评论
显示 1更早的评论
Antoni Woss
Antoni Woss 2022-9-13
The reluLayer will apply the rectified linear unit operation to the outputs of the preceding layer. As you mentioned the reluLayer is exactly a layer of activation functions. For example, if the reluLayer follows a 2D convolutional layer, where the output of the convolution layer is say 10x10x5 (5 filters each of 10 pixels by 10 pixels), then the reluLayer will apply the rectified linear operation to each of the 10x10x5 values.
As the reluLayer operates elementwise, you do not need to specify architecture, such as specifying the outputSize in the fullyConnectedLayer. The reluLayer infers the correct architecture from the previous layer.
To specify the number of output activations (number of neurons as you are referring to) for MATLABs built in layers, you can take a look at the documentation for the particular layer. For example:
  • For the lstmLayer specify the number of hidden units, numHiddenUnits, to specify the number of output activations.
  • For the fullyConnectedLayer you can specify the outputSize.
Here are some documentation links to the layers discussed in this post for further reference:
Alexander Krauss
Alexander Krauss 2022-9-14
Thanks a lot Antoni, great and insightful answer. :)
I would just like to know how dropout layers are to be understood.
I mean I know what dropout and dropout rates respectively are and how they work.
But again my question refers to the implementation of dropout layers in MATLAB.
If I want to achieve a dropout rate of say 15 % (for each hidden layer), do I have to add a 15 % dropout layer after every activation layer or will it be enough to just add one dropout layer which then applies to the entire network of hidden layers.
Would be great if you could answer this one as well.
Best regards
Alex

请先登录，再进行评论。

km y
km y 2023-9-8
可以看一下我写的博客，这里有如何调用和修改激活函数内容：如何在matlab工具箱中自定义激活函数及其使用_web，gui编写爱好者的博客-CSDN博客

类别

在 Help Center 和 File Exchange 中查找有关 Image Data Workflows 的更多信息

标签

产品

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by