As you have defined them, X and A are SCALAR symbolic objects, not general matrices. So the trace operation is a no-op, essentially ignored as the trace of a scalar, and the differentiation does not see them as matrices, since A and X are indeed scalars.
However, if A and X are symbolic matrices, it appears that diff does not allow differentiation with respect to a symbolic matrix. So:
>> A = sym('A',[2,2])
A =
[ A1_1, A1_2]
[ A2_1, A2_2]
>> X = sym('X',[2,2])
X =
[ X1_1, X1_2]
[ X2_1, X2_2]
>> diff(trace(A*X),X)
Error using sym/diff (line 69)
The second argument must be a variable or a nonnegative integer specifying the number of differentiations.
So as you can see, this produces an error as you wish to do the operation. Yet, if you compute the derivatives wrt the scalar elements of X, we see the proper elements of the transpose of A are indeed produced.
>> diff(trace(A*X),X(1,2))
ans =
A2_1
>> diff(trace(A*X),X(2,1))
ans =
A1_2
