A is a 4x9 array, so A' is a 9x4 array.
A.*A'
This tries to use element-wise multiplication on A and A', but they aren't the same size (or compatible), so you get an error.
Maybe you mean this?
a=A*A'\A*y';
This performs matrix multiplicaton on A and A', resulting in a 4x4 array which you can then pass to mldivide along with A*y' (a 4x1 array).
