For the matlab function rms, I suggest you see
help rms
since it listed how to use that function and apparently it can do an rms calculation column-wise or row-wise, so it should be suitable for you.
Regarding the code from your colleague, as you said with A as the matrix with dB values:
rmslevel=10*log10(mean(10.^(A/10)));
So I would break it down to 3 steps of calculation: The exponent, the mean, and the log operations, respectively.
x = 10.^(A/10); %the exponent
y = mean(x); %the mean
z = 10*log10(y); %the log
Refering to this site, the exponent is a formula to convert dB values back to power ratio (so, gain or attenuation). From this piece of code, you should get all values of A from dB back to the power ratio, which is why it is appropriate to do the second part, the mean. As it is, it computes the mean of the values column-wise, unless A is a vector (only 1 row) then the computation would be row-wise. Finally the log brings that averaged value of power ratio back to dB.
Therefore, I think this code only computes the average of a vector of dB values, not the rms.
I would recommend you to do something like
rms_A = 10*log10(rms(10.^(A/10),2));
which would convert dB values back to power ratio, find the rms of that, and turn the rms to a dB value.
As to why the dB values should be converted to power ratios first, I think this discussion presents good explanations.
