Probably not applicable to (or too complicated for) this problem, but if f(x) has a linear dependence on the unknowns,
y=p1*f1(x)+p2*f2(x)+...+pn*fn(x)
where p1...pn are the unknown parameters, some of which are integers, then instead of doing a least squares fit, you can do a max-norm fit
min r
subject to
-r<=y-( p1*f1(x)+p2*f2(x)+...+pn*fn(x) )<=r
which is a mixed integer linear program in the variables [r,p1,...,p3] and can be solved with intlinprog . An L1-norm fit can be done in a similar way.
