Your formula is wrong. when you say var(var(A)) you are actually computing the variances of each column and then variances of these variances. Is there a reason you are doing this? However if you are interested in reproducing the result, then you have to follow the same steps manually as follows:
A = magic(3);
B = sum((A-repmat(mean(A),3,1)).^2)/2; % Variance of each column
var_magic = sum((B-mean(B)).^2)/2 % Variance of the variance computed above.
var_magic =
27
