Correct weight Initialization in CNN

2018 7 29
3 个回答
回答已采纳
更新时间：2019 7 5
1 次查看（30 天）

0 个投票

When a very deep DAG network is built from scratch, the initialization of the weights made by matlab is not very good since it presents a vanishing gradient problem which causes the CNN not to learn.
What is the function with which Matlab does the initiation of CNN weights?
Why do you implement initialization functions in Matlab such as XAVIER or RELU AWARE SCALALED?
Thank you for your answers.

2 个评论
显示 无 隐藏 无

Greg Heath
Greg Heath 2018-7-31
I do not understand
"Why do you implement initialization functions in Matlab such as XAVIER or RELU AWARE SCALALED?"
Please explain.
Greg
Yuze Zou
Yuze Zou 2019-7-3
I guess this issuse has been solved in the latest release (R2019a) via new default weights initialization method (i.e., Xavier/Glorot) for `fullyConnectedLayer`. You can find more details here.

请先登录，再进行评论。

请先登录，再回答此问题。

 采纳的回答

Maria Duarte Rosa
编辑：Maria Duarte Rosa 2019-7-5

1 个投票

In R2019a, the following weight initializers are available (including a custom initializer via a function handle):
'glorot' (default) | 'he' | 'orthogonal' | 'narrow-normal' | 'zeros' | 'ones' | function handle
Glorot is also know as Xavier initializer.
Here is a page comparing 3 initializers when training LSTMs:
https://www.mathworks.com/help/deeplearning/examples/compare-layer-weight-initializers-example.html
I hope this helps,
Maria

更多回答（2 个）

Andres Ramirez
Andres Ramirez 2018-7-31

0 个投票

Hello Gerg, thanks for answering ... I'll explain:
I have built very deep networks such as: Googlenet, Resnet and VGG19, and I want to train them from scratch with my databases; However, when I do the training of any of these networks, the network does not learn and only reaches a maximum performance of 12 or 15%. I think the low performance is mainly due to the random initialization of the weights made by defaul in matlab does not work for very deep networks, according to the literature, random initialization causes a vanishing gradient problem which causes the network do not learn.
For the above, my questions are:
Why do not implement in matlab more appropriate weight initialization functions for the training of deep DAG networks, for example, XAVIER or RELA AWARE SCALALED?
I hope to have been clear the explanation of the problem I have ...
Thank you.
Greetings.

1 个评论
显示 -1更早的评论 隐藏 -1更早的评论

Greg Heath
Greg Heath 2018-8-1
编辑：Greg Heath 2018-8-1
Do you have a reference for
RELA AWARE SCALALED
I have no idea what this is.
Thanks
Greg

请先登录，再进行评论。

0 个投票

I think you can take a look at this example https://www.mathworks.com/help/images/single-image-super-resolution-using-deep-learning.html
I am also looking for a way on weight initialization options, you can see in the example it create the initialization with He method for every conv layer.

类别

在 帮助中心 和 File Exchange 中查找有关 Deep Learning Toolbox 的更多信息

产品

版本

R2018a

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by