Correct weight Initialization in CNN

Question

Andres Ramirez 2018-7-29

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/412549-correct-weight-initialization-in-cnn

编辑： Maria Duarte Rosa 2019-7-5

When a very deep DAG network is built from scratch, the initialization of the weights made by matlab is not very good since it presents a vanishing gradient problem which causes the CNN not to learn.

What is the function with which Matlab does the initiation of CNN weights?

Why do you implement initialization functions in Matlab such as XAVIER or RELU AWARE SCALALED?

Thank you for your answers.

2 个评论
显示无隐藏无

Greg Heath 2018-7-31

I do not understand

"Why do you implement initialization functions in Matlab such as XAVIER or RELU AWARE SCALALED?"

Please explain.

Greg

Yuze Zou 2019-7-3

I guess this issuse has been solved in the latest release (R2019a) via new default weights initialization method (i.e., Xavier/Glorot) for `fullyConnectedLayer`. You can find more details here.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Maria Duarte Rosa 2019-7-5

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/412549-correct-weight-initialization-in-cnn#answer_382152

编辑：Maria Duarte Rosa 2019-7-5

In R2019a, the following weight initializers are available (including a custom initializer via a function handle):

Glorot is also know as Xavier initializer.

Here is a page comparing 3 initializers when training LSTMs:

https://www.mathworks.com/help/deeplearning/examples/compare-layer-weight-initializers-example.html

I hope this helps,

Maria

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

Answer 2

Andres Ramirez 2018-7-31

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/412549-correct-weight-initialization-in-cnn#answer_331085

Hello Gerg, thanks for answering ... I'll explain:

I have built very deep networks such as: Googlenet, Resnet and VGG19, and I want to train them from scratch with my databases; However, when I do the training of any of these networks, the network does not learn and only reaches a maximum performance of 12 or 15%. I think the low performance is mainly due to the random initialization of the weights made by defaul in matlab does not work for very deep networks, according to the literature, random initialization causes a vanishing gradient problem which causes the network do not learn.

For the above, my questions are:

Why do not implement in matlab more appropriate weight initialization functions for the training of deep DAG networks, for example, XAVIER or RELA AWARE SCALALED?

I hope to have been clear the explanation of the problem I have ...

Thank you.

Greetings.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Greg Heath 2018-8-1

编辑：Greg Heath 2018-8-1

Do you have a reference for

RELA AWARE SCALALED

I have no idea what this is.

Thanks

Greg

请先登录，再进行评论。

Answer 3

fareed jamaluddin 2018-8-4

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/412549-correct-weight-initialization-in-cnn#answer_331600

I think you can take a look at this example https://www.mathworks.com/help/images/single-image-super-resolution-using-deep-learning.html

I am also looking for a way on weight initialization options, you can see in the example it create the initialization with He method for every conv layer.