xold is not defined in your function before the first time you use it. However, your use of
xo = [x1 x2 ]';
hints that xold is likely to be 2 x 1. If so then when you do
xnew = xold - J\f;
then because J\f would be 1 x 2, you would be getting a 2 x 1 minus a 1 x 2. In R2016a or earlier that would be an error.
In R2016b or later, it is well defined, and would give you a 2 x 2 array as a result. That would probably trigger problems further down the road.
