Why does my neural network always output 0.05?

Question

mctaff 2017-2-24

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/326804-why-does-my-neural-network-always-output-0-05

评论： Vandana Rajan 2017-3-1

Hi,

I'm trying to implement a neural network, replicating that from a research paper.

There are 150 input variables, with values in the range [0,1]. The net has 4 hidden layers, the first 3 have 128 nodes, the final layer 20 nodes.

As per the paper, I'm creating the network by sequentially training auto encoders, then stacking them to create the final network.

However, whatever input I provide, every value is mapped to ~0.05 in the final output.

Is this an issue anyone has seen, and are there any suggestions as to why it may be occurring?

Code below, with some random numbers added as input so it will execute.

    X = rand(150, 3000);
    noNodes = 128;
    ae1   = trainAutoencoder(X, noNodes);
    f1  = encode(ae1, X);
    ae2   = trainAutoencoder(f1, noNodes, 'ScaleData', false);
    f2  = encode(ae2, f1);
    ae3   = trainAutoencoder(f2, noNodes, 'ScaleData', false);
    f3  = encode(ae3, f2);
    ae4   = trainAutoencoder(f3, 20, 'ScaleData', false);
    dnn = stack(ae1, ae2, ae3, ae4);
    Y = dnn(X);

Thanks,

David.

1 个评论
显示 -1更早的评论隐藏 -1更早的评论

mctaff 2017-2-25

编辑：mctaff 2017-2-25

Ok, so I've established that the 0.05 is the default value for the Sparsity Proportion attribute of the trainAutoencoder function. If I amend the value of this attribute, then my network will output the amended value.

Further, I notice all the layer weights are similar, which would cause all my nodes to give a similar output.

Can anyone explain why the Matlab trainAutoencoder function is making all the weights similar, and how I can avoid this?

Thanks.

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Vandana Rajan 2017-2-28

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/326804-why-does-my-neural-network-always-output-0-05#answer_256705

Hi,

Could you please try the code by reducing the value of 'SparsityRegularization' (not 'SparsityProportion') or just setting it off to 0?

2 个评论
显示无隐藏无

mctaff 2017-2-28

Thanks Vandana,

I've been playing around with other input data sets over the weekend - with the above parameter set to 0 as you suggest - and have found some data that lead to a network with multiple output values... so I'm wondering if its something to do with a combination of the initial input data I was using, and the sparsity regularisation setting

Vandana Rajan 2017-3-1

Hi Mctaff,

When training a sparse autoencoder, we optimize a loss function that has two terms:

1) A mean squared error term.

2) A sparsity regulariser which tries to force the average activation value to the value defined by ‘SparsityProportion’.

The parameter ‘SparsityRegularization’ controls the weighting of the second term in the loss function. If it is too high for a particular problem, then the training procedure can collapse into a pathological solution to a problem, where it simply forces the value of the activations to match the value of ‘SparsityProportion’. This is what appears to be happening when running your code.

You can prevent this by reducing the value of ‘SparsityRegularization’, or switching it off entirely by setting it to 0.

请先登录，再进行评论。