How can I train a neural network manually choosing a specific set of initial weights and biases?

Question

Roberto 2016-2-3

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/266341-how-can-i-train-a-neural-network-manually-choosing-a-specific-set-of-initial-weights-and-biases

评论： Nga Dao Thi 2017-1-4

My goal is to reproduce many times exactly the same training in order to do some tests about some training parameters variations. So what i tried first was to define a neural network, configure it, initialize it. Then I "switched off" the init function in order to keep fixed the initial values of weights and biases and use them again later in another training. I succeeded in this by setting

     net.initFcn = 'initlay';
     net.layers{1,1}.initFcn = 'initwb';
     net.layers{2,1}.initFcn = 'initwb';
     net.inputWeights{1,1}.initFcn = '';
     net.layerWeights{2,1}.initFcn = '';
     net.biases{1,1}.initFcn = '';
     net.biases{2,1}.initFcn = '';

and it actually worked. In this way even calling the function "init" or "configure" does not affect in any way the weights and biases of the net. So I thought that if I manually set two nets in this way and with the same initial values, training them would result in exactly the same final weights/biases...but it doesn't... My question is why?

Does "trainlm" re-initialize the net in some way different than calling "init"? Has it some randomness other than initialization?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Greg Heath 2016-2-3

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/266341-how-can-i-train-a-neural-network-manually-choosing-a-specific-set-of-initial-weights-and-biases#answer_208414

If you want to reproduce initial weights, reinitialize the rng to the same state before using configure.

If any non-zero weights are present, TRAIN will treat all weights as initial weights. Otherwise it will self-initialize using random weights.

Hope this helps.

Thank you for formally accepting my answer

Greg

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

Roberto 2016-2-3

在 MATLAB Online 中打开

I'm sorry, I don't understand. If train doesn't initialize the net if a non-zero weight is present, then two networks initialized to the same state will have exactly the same training and should result in exactly the same final weights/biases right? But look:

    >> net2 = net;
>> net.IW{1,1}

ans =

-4.199999999999999
-4.199999999999999
-4.199999999999998

>> net2.IW{1,1}

ans =

-4.199999999999999
-4.199999999999999
-4.199999999999998

>> net = train(net,Input,Output); >> net2 = train(net2,Input,Output); >> net.IW{1,1}

ans =

-5.175970179589142
-2.034413485697540
-2.720340992563584

>> net2.IW{1,1}

ans =

-6.053550714637582
 0.375020219358226
-3.628899940303217

the 2 nets are identical and both have clearly non-zero weights before training...so how is possible that final weights are different? Maybe trainlm has some other "randomness" somewhere? I know Levenberg-Marquardt algorithm and I thought that it was purely deterministic...no randomness at all...am I wrong?

请先登录，再进行评论。

Answer 2

Greg Heath 2016-2-3

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/266341-how-can-i-train-a-neural-network-manually-choosing-a-specific-set-of-initial-weights-and-biases#answer_208465

在 MATLAB Online 中打开

After net is trained the state of the rng is changed. Therefore the weight initialization of net2 is different.

 % help fitnet
 close all, clear all, clc
 [ x, t ] = simplefit_dataset;
 net1 = fitnet; net2 = fitnet; net3 = fitnet;
 isequal(net1,net2), isequal(net1,net3)% 1,1
 WB0 = [getwb(net1) getwb(net2) getwb(net3) ] % zeros(10,3)
 rng(0),  net1 = train(net1,x,t); net2 = train(net2,x,t);
 rng(0), net3 = train(net3,x,t);
 WB = [getwb(net1) getwb(net2) getwb(net3)]
 WB =
          -16.568      -11.025      -16.568
          -5.5931      -7.8365      -5.5931
           3.6185      -3.8022       3.6185
          -4.0666       4.3355      -4.0666
            -1.85       1.6122        -1.85
          0.07357    -0.094195      0.07357
           -2.834       3.1095       -2.834
           4.3672      -5.1322       4.3672
           8.4144       7.4463       8.4144
            12.07        13.83        12.07
           12.118       12.197       12.118
           6.6852       9.9585       6.6852
          -6.9822       7.4495      -6.9822
           10.482      -11.442       10.482
           12.836      -11.914       12.836
          -12.557      -14.228      -12.557
          -7.3149       7.7502      -7.3149
           7.6948       -8.879       7.6948
           9.7115       8.1667       9.7115
           12.196       12.392       12.196
         -0.84428      -1.9359     -0.84428
         -0.71237      0.25416     -0.71237
          0.72471      0.36081      0.72471
          0.81622       -0.766      0.81622
         -0.18894      0.17421     -0.18894
         0.090501    -0.095302     0.090501
        -0.051973    -0.040525    -0.051973
          0.11345     -0.13442      0.11345
         -0.21845      0.19223     -0.21845
          0.31931      0.53145      0.31931
          0.41655        1.956      0.41655

2 个评论
显示无隐藏无

Roberto 2016-2-4

It really does the trick. Thank you.

Nga Dao Thi 2017-1-4

Thank you! I experienced the similar situation. I define a neural network (NN) with initial weights are 0s. However, when this NN trains the same dataset differently, i.e., the results are different when I run program many times. I also don't know why it happens. Later, I find that the training function trainscg will update weight and bias values according to the scaled conjugate gradient method. If I set biases are 0s, my NN has the same result. In contrast, If I set biases are not 0s, trainscg will update biases and weights. So that, the NN trains differently.

请先登录，再进行评论。

How can I train a neural network manually choosing a specific set of initial weights and biases?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

更多回答（1 个）

2 个评论
显示无隐藏无

另请参阅

类别

标签

Community Treasure Hunt

How can I train a neural network manually choosing a specific set of initial weights and biases?

0 个评论 显示 -2更早的评论隐藏 -2更早的评论

采纳的回答

1 个评论 显示 -1更早的评论隐藏 -1更早的评论

更多回答（1 个）

2 个评论 显示 无隐藏 无

另请参阅

类别

标签

Community Treasure Hunt

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

2 个评论
显示无隐藏无