There are links to the bibliography throughout the fmincon algorithm descriptions. Perhaps the most useful general reference is Nocedal, J. and S. J. Wright. Numerical Optimization, Second Edition. Springer Series in Operations Research, Springer Verlag, 2006. However, many details are in the referenced papers.
The fminsearch algorithm is described completely in the documentation. I cannot imagine that you would need any further detail, but you can read the entire code by entering edit fminsearch at the MATLAB command line.
lsqnonlin and lsqcurvefit use exactly the same algorithms. As for fmincon, the algorithms are described in the documentation and associated references.
Alan Weiss
MATLAB mathematical toolbox documentation
