For the give dimensions and given equation it cannot be solved. But for given dimensions the following equation can be solved.
% A'*X*A = B
To solve above A can be (3x4) , X is (3,3) abd B is (4x4).
A = rand(3,4) ;
B = rand(4,4) ;
X = pinv(A')*B*pinv(A)
If you want to solve the given equation:
% A*X*A' = B
The dimensions of A should be (4x3), X is (3x3) and B is (4X4)
A = rand(4,3) ;
B = rand(4,4) ;
X = pinv(A)*B*pinv(A')
Check the dimensions once for your quesion. Read about pinv function and it's properties.
