Very strange. (Not what fminsearch did, but what you are doing.)
You have one function in there called tota, which is identically zero. You add it to err(x). err is something strange looking too. But it appears that err is doing something highly non-differentiable, and certainly discontinuous.
Why is that a problem? Because fminsearch is not designed to solve such a problem.
And, of course, you do not tell us what img and adam are. IMG is some arbitrary cell array, containing only cell. adam is some unknown vector, of length 5. So we cannot even test your code out to show you what it does, and why fminsearch fails.
So it is totally impossible to help you, to explain why fminseach goes where it did go.
As for why it might have done so, fminseach has NO constraints upon it. So it can freely go wherever it decides might improve the objective function.
