I understand that you wanted to use “RMSE” as the performance function in "nntool". In the "nntool", the Root Mean Square Error (RMSE) is not available as a built-in performance function.
You can create a custom performance function to achieve your task. The easiest way to create a custom performance function is to use the +MSE package as a template.
You can refer to the link below, which includes all the steps in detail, in order to create the custom performance function:
I hope this helps!