You are trying to solve A=C*B+D. Have you tried the solve function?
syms a b c d e f g h i
A=[a*cos(h)+d,b*sin(g)+e,c*sin(i)+f;a*sin(g)+f,b*cos(i)+e,c*sin(h)+d;a*cos(g)+d,b*sin(h)+f,c*sin(i)+e];
B=[a;b;c];
C=[cos(h),sin(g),sin(i);sin(g),cos(i),sin(h);cos(g),sin(h),sin(i)];
D=[d,e,f;f,e,d;d,f,e];
sol=solve(A==C*B+D,[d e f g h i]);
This returns only empty syms, meaning that Matlab can't find a general solution to your question.
