Your question is ambiguous as to what you are trying to retain, whether you are trying to retain terms that involve either thetadot1 or thetadot2, or only terms that involve multiplying by both of those variables.
You start by asking for terms that involve thetadot1. There is a single such term in the original expression, namely -2*m2*l1*l2*sin(theta2)*(thetadot1*thetadot2) . The coefficient associated with thetadot1 in that term is -2*m2*l1*l2*sin(theta2)*thetadot2 which is exactly the output MATLAB returned. You did not ask it for the coefficients for thetadot2.
You are also asking only about the first coefficient, power 1, but one of the terms you want in the result involves power 2
coeffn = @(poly, var, n) feval(symengine, 'coeff', poly, var, n);
syms m1 m2 l1 l2 thetadotdot1 thetadotdot2 thetadot1 thetadot2 theta1 theta2 g
t = m2*l2^2*(thetadotdot1+thetadotdot2)+m2*l1*l2*cos(theta2)*(2*thetadotdot1+thetadotdot2)+(m1+m2)*l1^2*thetadotdot1-m2*l1*l2*sin(theta2)*(thetadot2^2)-2*m2*l1*l2*sin(theta2)*(thetadot1*thetadot2)+m2*l2*g*cos(theta1+theta2)+(m1+m2)*l1*g*cos(theta1);
c10 = coeffn(t, thetadot1, 0);
c20 = c10 - coeffn(c10, thetadot2, 0);
result = t - c10 + c20;
disp(char(result))
That is, we look in t for all the terms that do not involve the first variable. t minus that would leave you with terms that only involve any power of the first variable. We fish through the terms that did not involve the first variable to find any terms that involve the second variable: those would be terms that involve the second variable but not the first variable. Terms that involved both variables would not have been selected by coeff 0 of t. When we take the terms that did not involve the first variable and subtract out the terms of that that did not involve the second variable either, then we are left with the terms that involve only the second but not the first in c20. So now we take the original expression, subtract out the terms that did not involve the first variable, add back in the terms that involved the second variable but not the first, and we get our result.
This process can obviously be extended to any number of variables with a small loop.
