b=inv(A'*A)*A'*y;
Multiply both sides on the left by A'*A :
(A'*A) * b = (A'*A) * inv(A'*A) * A' * y
and for any invertable matrix, X * inv(X) is the identity matrix so (A'*A) * inv(A'*A) cancels out to identity, so
(A'*A) * b = A' * y
Multiply both sides on the left by inv(A'):
inv(A') * A' * A * b = inv(A') * A' * y
and inv(A') * A' cancels to the identity on both sides, so
A * b = y
Multiply by inv(A) on the left on both sides:
inv(A) * A * b = inv(A) * y
inverse cancels to identity, so
b = inv(A) * y
Now use the \ operator:
b = A \ y;
The above mathematics might not strictly apply if A is not a square matrix.
