It is very rarely easy to know why a solver fails to converge on a messy thing like this. But you should never let it come to that point.
BEFORE you ever use a solver on a problem like this, first test your objective function. Evaluate it at several points in the domain of your parameter space, points where you expect it should be well-behaved. Make several observations at this stage.
Does the objective function produce a REAL result, and is it SCALAR valued?
If you perturb the parameters by a small amount, does the objective change?
Are there any issues you observe that might be cause for a problem? That is, if the function is essentially constant, then you should not expect to see fmincon fail. If the function returns complex results, then again, fail. If the function returns a vector or array, then again, fail.
Look carefully at the variation you see in the objective function. Remember that a tolerance will be applied, and if the function is seen to be changing not at all, or less than the tolerance applied, then what do you expect to happen?
Think about your code. Is there any point where you form a discretized operation inside? That, do you ever use floor, fix, round, ceil, etc.? If so, then your function will not e differentiable, or even continuous. Again, FAIL.
Is there any random aspect of the code inide the objective? Fmincon assumes the function is absolutely deterministic. If not, then fail.
The above are some basic tests you should ALWAYS perform. (Given some time, I could probably think of a few more, but the above tests would obviate most of the desperate questions we see here, about why some optimzer has failed to produce a viable result.)
NEVER just throw together a pile of code, and then throw it at an optimizer. Instead, think about what it does, and if it makes sense to employ fmincon there. Test the objective, and verify it does what it should do. (Do I do all of the above tests when I use a solver? Well, usually not. But I've been writing code and using optimizers since probably before you were born. Until you get to that point, then follow my advice.)
When you do see a failure to converge, first, go back and perform EXACTLY those same tests I just told you to run.
Next, look at the return flag. What does the exitflag tell you? If you don't understand what the various returns are for exitflag, then tell us what you got, and ask for help on interpreting that result.
Finally, if everything above seems reasonable, look at the final point fmincon returns. What is the objective value at that location? Is it just that your function has no minimum? For example, if I told fmincon to minimize exp(x), then I would expect a divergent result at x==-inf, or at least as close as it could get to -inf before it gives up.
