You are describing a nonlinear constraint. This is a type of constraint that simulannealbnd cannot handle.
In fact, your constraint is not differentiable. This can be a problem for solvers. However, if your underlying variables are integer-valued, then you can use some techniques from integer linear programming to create the constraint. See Optimal Dispatch of Power Generators for an example.
Solvers that can handle nonlinear constraints include fmincon, patternsearch, ga, and the recent version of surrogateopt. Solvers that handle integer variables include intlinprog, ga, and surrogateopt.
Alan Weiss
MATLAB mathematical toolbox documentation