fmincon() is never suitable for discrete parameters.
If you have mixed integer work that has a scalar output then your options are:
- ga -- supports anonymous functions; supports nonlinear constraints
- abusing patternsearch() -- supports anonymous functions; supports nonlinear constraints
- intlinprog() -- supports matrix objectives only; does not support nonlinear
- surrogate optimization -- supports anonymous functions; supports nonlinear constraints; https://www.mathworks.com/help/gads/surrogateopt.html
In order to use integer constraints with ga(), you have to write your own functions to handle cross-over and mutation and initial population -- functions that just happen to obey the required integer constraints. The more obvious integer constraints for ga() are incompatible with providing nonlinear contraints.
Surrogate optimization provides obvious integer constraints, but does not provide obvious nonlinear constraints. However, instead of your objective function returning a numeric scalar, it can instead return a struct with a particular form, and surrogate optimization will follow the nonlinear constraints expressed in the structure.
