As I understand it, ‘RMSE’ is the square-root of the MSE value.
I am not certain what you want to do. Here are two options (using the example from the plotperform documentation):
[x,t] = bodyfat_dataset;
net = feedforwardnet(10);
[net,tr] = train(net,x,t);
plotperform(tr)
yt = get(gca, 'YTick');
set(gca, 'YTick', yt, 'YTickLabel',compose('%.1f',sqrt(yt))) % Convert ‘YTickLabel’ Values To RMSE
Ls = findobj(gca, 'Type','line'); % Calculate RMSE For All ‘Line’ Object Y-Values
for k1 = 1:numel(Ls)
yval{k1} = Ls(k1).YData;
rmse{k1} = sqrt(yval{k1});
end
The entire code converts the Y-tick values and calculates the RMSE.
Experiment to get the result you want.