The RMS formula of gwyddion concerns the term "z-zmean", while MATLAB's RMS concerns u. If u=z-zmean, both formulas reply the same value.
The difference is, that gwyddion considers real values only, while MATLAB cares for complex values by using CONJ in addition. For real values u==conj(u) and:
u .* conj(u) == u .* u == u .^ 2
