STOP.
Is this nonlinear? Or is it linear? A*X = b is still LINEAR, even if the array is not square.
Even in the linear case, thus A*X = b, even if the matrix is not square, you should never be using inv anyway.
x = A\b;
solves the problem in MATLAB. If the matrix A is rank deficient, then you can use tools like
x = pinv(A)*b;
or
x = lsqminnorm(A,b);
HOWEVER, if this truly is nonlinear? Then you have some problem in the form of a nonlinear regression, where you need to use tools like lsqnonlin or lsqcurvefit. So in either case, there exist tools to solve the problem (NONE of them should ever be inv) but you need to know which problem you want to solve. At least, you need to explain to someone clearly what problem you want to solve.
One thing you can only very rarely do, if your problem is of the form
f(x) == y
for a nonlinear function f, is to solve it by computing an inverse function that might be thought of as inv(f).
