fmincon uses standard algorithms. You can find references to the algorithms in the Constrained Nonlinear Optimization Algorithms section of the documentation. In general, it uses time-tested, robust algorithms for the solution of general nonlinear problems with general smooth nonlinear constraints.
If you are asking "Is fmincon the best available nonlinear optimization solver?" then I don't think that anyone can give a definitive answer. For problems having special structures, it is entirely possible that a specialized solver could work better, but for general problems fmincon works pretty well. It's like asking if a Toyota is the best car. The best car for what purpose? Fastest for the price? Longest lasting? Able to haul the most people? Bottom line: Toyota is a reliable manufacturer that satisfies many people's transportation needs.
Hans Mittleman has a comparison of nonlinear solvers that shows fmincon is overall not as fast as Knitro, but holds up well compared to many other solvers. Is his set of benchmarking problems similar to yours? I don't know, and it would take quite a bit of work for you to find out.
Bottom line: fmincon is the main solver for constrained nonlinear problems in Optimization Toolbox, a widely-used optimization program. Optimization Toolbox is a reliable choice that satisfies many people's optimization needs.
Alan Weiss
MATLAB mathematical toolbox documentation
