How is RMSE calculated on Multivariate Regression Neural Networks?

14 次查看（过去 30 天）
Raphael Rottshäfer
评论： Raphael Rottshäfer 2020-8-24
In short: The multivariate regression network ends with validation RMSE of e.g. ~72.6 while my own calculation ends up to be 14.8. How is this network calculating the RMSE on multivariate regression? How can I trust this RMSE / Loss optimization? Did I miss some configuration?
At first I thought those 72.8 is the RMSE for a whole day and 72.8/24 = 3.0 will be my RMSE for a single hour. But that does not fit on my own RMSE calculation afterwards, which states 14.8 hourly RMSE. I use the following formula:
RMSE = sqrt(mean((v1-v2).^2));
Before my RMSE calculation, I reshape the prediction/validation data to Kx1 matrix, where K represents the hours of each day in order.
Some facts on my approach:
  • My feature array has the shape of NxM, where N is the number of features and M is the corresponding day (in order).
  • My label set look similar: 24xM, where 24 represents the hourly energy price and M is the corresponding day (in order).
  • I split the data beforehand into training & validation data (while cut out some random days as testing data beforehand).
As far as I can tell, that seems to work with 24 responses as regression output. But the RMSE is far away from being reasonable.
Any help will be appreciated! :)
Network:
numFeatures = size(features.input_data, 1);
numHiddenUnits1 = 250;
numResponses = 24;
layers = [ ...
sequenceInputLayer(numFeatures)
lstmLayer(numHiddenUnits1,'OutputMode','sequence')
fullyConnectedLayer(100)
fullyConnectedLayer(50)
fullyConnectedLayer(numResponses)
regressionLayer];
options = trainingOptions('sgdm', ...
'MaxEpochs',200, ...
'ValidationData',{features.val, labels.val},...
'ValidationFrequency',5,...
'InitialLearnRate', 0.05,...
'LearnRateSchedule', 'piecewise', ...
'LearnRateDropFactor',0.8, ...
'LearnRateDropPeriod',15, ...
'Plots',"training-progress", ...
'Verbose', false, ...
'GradientThreshold', 0.25);
[net scores] = trainNetwork(features.train, labels.train, layers, options);

请先登录，再回答此问题。

采纳的回答

Prasanth
Prasanth 2020-8-20
I am assuming that you are using MATLAB R2020a and you tried to design a neural network with a 'regressionLayer' as output. You computed the validation error from your neural network as ~72.6 (example).
By default, the 'regressionLayer' calculates the half mean squared error. So, when you are referring to the validation loss given by your neural network, it is actually the half mean squared error, but not RMSE, as you are assuming.
You can refer to the documentation link for the regressionLayer for more details.
Currently, there is no option to choose the loss function in the 'regressionLayer'.
If you want to use any other loss function, you can create a custom regression layer with your own loss function.
You can refer to Defne Custom Regression Layer documentation link to get more information and template code for creating a custom regression layer. The example in the above link creates a custom regression layer with Mean Absolute Error (MAE) as the loss function.
I hope this helps in resolving your issue.
  1 个评论
Raphael Rottshäfer
Raphael Rottshäfer 2020-8-24
Hello Prasanth,
thank you very much for your detailed explanation on my issue! You are totally right: I use R2020a and a regressionLayer. And now I know what is happening there. As I assume, that the vanilla regression layer's loss function scales as expected, I skip the Cutom Regression Layer for now, since now I can comprehend the vanilla version. :)
Thank you!

请先登录，再进行评论。

更多回答（0 个）

类别

在 Help Center 和 File Exchange 中查找有关 Sequence and Numeric Feature Data Workflows 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by