With your line syms alpha, you have overwritten the previous value of the MATLAB(!) variable alpha, so the alpha(t) in your subs call no longer does what you expected it to do.
You could substitute alpha(t) with a new variable like this:
syms alpha(t)
x = sin(alpha(t))
x_dot = diff(x,t)
syms alpha_t
x_dot = subs(x_dot,alpha(t),alpha_t)
diff(x_dot, alpha_t)
But I doubt that is what you want, since x_dot contains a diff(alpha(t),t) term and if you now make alpha independent of t, that gives 0. I have to guess, but maybe you want this?
syms alpha(t)
x = sin(alpha(t))
x_dot = diff(x,t)
syms alpha_t alpha_dot
x_dot = subs(x_dot, diff(alpha(t),t), alpha_dot)
x_dot = subs(x_dot,alpha(t),alpha_t)
diff(x_dot, alpha_t)
