The error message indicates that the given set of constraints are impossible to solve due to an inconsistency. To troubleshoot this, it is helpful to remove these constraints one at a time to get a general idea of where the problem may originate.
Some other things to check:
- If the local optimum (supplied by the first optimization) is actually a feasible point according to the set of constraints
- The number of iterations does fmincon runs in both cases. You may need to increase the number of iterations.
- The tolerance values might need to be changed using optimoptions
Have you considered global searching algorithms like "ga" or "patternsearch", considering that the problem you wish to solve is non-convex?
