You are not telling the Symbolic Toolbox everything you know about your system. See if this code does something similar to what you want:
syms x a
assume(a, 'real')
assumeAlso(x, 'real')
f = [ [ cos(conj(x))*cos(x), cos(conj(x))*exp(a*i)*sin(x)]; [ sin(conj(x))*exp(-conj(a)*i)*cos(x), sin(conj(x))*exp(a*i)*exp(-conj(a)*i)*sin(x)] ]
fs = simplify(collect(f))
Note that when it knows that x is real, it eliminates conj(x).
‘Also exp(a*i)*exp(-conj(a)*i) would normally have cancelled out to give 1 but the resulted expression doesn't simplify to it’
It does now that it knows what you know about a:
g = exp(a*i)*exp(-conj(a)*i)
Experiment with it and the various functions in the Symbolic Toolbox to get the result in the form you want.
If you want it as a function you can execute as regular MATLAB code (outside of the Symbolic Math Toolbox), see matlabFunction.
