Part I
You are using SYM() while using SYMS style of command. They are different. So change sym F x to syms F X or if you insist on using sym() then you need to use it properly such as F=sym('F')'
Part II
You can use that error to determine both the coefficient and if it was polynomial or not. You can use try/catch as follows:
try
PolyCoef = sym2poly(F);
catch
disp('that was not a polynomial')
PolyCoef=[];
end
In this case the program is not halted. If the statement(s) between catch and end (catch block) is only executed if, the statements between try and catch (try block), i.e. PolyCoef = sym2poly(F) in this case, produces any error. Refer to try/catch for further info on that.
