Alternative to softmax function for Neural Network predicting fractions of a whole

Question

F S 2019-8-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/477382-alternative-to-softmax-function-for-neural-network-predicting-fractions-of-a-whole

评论： Ajay Pattassery 2019-8-29

Hi, I created a feed forward Regression Neural Network to predict variables which are fractions of a whole (i.e. they sum up to 1). In order to have the network fullfil this criterion perfectly, I am using the softmax transfer function. Unfortunately, I realize that the network predicts smaller fractions very poorly, and I think this is due to the fact that the softmax transfer function normalizes my target fractions by dividing exponent of the fractions minus the largest fraction by its sum (exp(n-nmax)/sum(exp(n-nmax))), which results in much larger values for very small fractions. It wouldn't have to do that, since my fractions are already between 0 and 1. Can I change that somehow in the softmax transfer function, or is there an alternative to it that doesn't do this normailzation?

6 个评论
显示 4更早的评论隐藏 4更早的评论

F S 2019-8-27

编辑：F S 2019-8-27

Of course! Thank you for your comment. Firstly I have noticed that this is happening from simply observing the prediction of my data. Secondly I digged a bit deeper into the code that is doing the regularization in the softmax function, and I found this, where n is my input targets and a the normalized output (this is in the 'apply' function in the +softmax folder from 2017 Version. In the 2019 Version it's still the same, but looks somewhat more complicated because there was some paralell computing code built into that function, from what I can judge):

nmax = max(n,[],1);

n = bsxfun(@minus,n,nmax);

numer = exp(n);

denom = sum(numer,1);

denom(denom == 0) = 1;

a = bsxfun(@rdivide,numer,denom);

What I realize is happening, is that if one fraction is much larger than the others, say 80% and the others split the remaining 20% unequally, e.g. one fraction is exponetially smaller, say 0.1%, the neural network cannot really catch that, because after the normalization from this code, the 19.9% and 0.1% fraction would be much closer together between 0 and 1 than what they actually are because of the exponent (you can see by simply creating a matrix n of fractions and run it eg n=[80 18.9 1 0.1; 70 29 0.9 0.1; 0.01 0.989 0.001 99, 25 25 25 25]'), in respect to the larger fraction. I am not really sure that that is the problem, since I do not at all see the big picture of the neural netoworks toolbox. I would be very glad if somebody could clarify, and if my instinct is right, suggest me a better solution where that problem didn't arise.

F S 2019-8-28

It is not multinomial logistic regression in a classical sence, as there are no classes to be predicted, although it is mathematically equal to it as far as I can judge. It is fractions of constituents of a mass, of which I know the weight and weight distribution, and I now the density of the constituents. So now I want to predict how much of which constituent I have, knowing that their weights ALWAYS sum up to the total weight. It can't be physically solved unfortunately, because there is several combinations that can lead to the same total weight, but the weight distribution holds information that can help solve the problem.

If there is a neural network or other method that can solve such a problem other than softmax, I would be happy if somebody can point that out to me.

I am also happy to send you more information in a private email, containing some data that I cannot disclose publicly. My licence and support subscription are up to date.

Ajay Pattassery 2019-8-29

Did you tried creating a custom layer which can force your output to one like the one I mentioned above instead of having a softmax layer.

Please refer the following link for creating custom layers.

https://www.mathworks.com/help/deeplearning/ug/define-custom-deep-learning-layer.html

请先登录，再进行评论。

请先登录，再回答此问题。

Alternative to softmax function for Neural Network predicting fractions of a whole

6 个评论
显示 4更早的评论隐藏 4更早的评论

回答（0 个）

另请参阅

类别

标签

Community Treasure Hunt

Alternative to softmax function for Neural Network predicting fractions of a whole

6 个评论 显示 4更早的评论隐藏 4更早的评论

回答（0 个）

另请参阅

类别

标签

Community Treasure Hunt

6 个评论
显示 4更早的评论隐藏 4更早的评论