If you're trying to minimize or maximize Y, and Y is a scalar, I'd recommend trying FMINCON from the optimization toolbox first. This is a gradient-based method, so it will travel downhill to find the local minimum value of Y. You can constrain the variables with bounds, linear equalities/inequalities, and nonlinear equalities/inequalities.
If Y is a vector, and you're interested in a least square minimization of Y, then the LSQNONLIN solver is probably more along the lines of what you're looking for.
Depending on what your function for Y looks like, you might also want to try some of the solvers from the global optimization toolbox. These solvers were designed to search for global minimums of nonlinear problems. I recommend starting with the PATTERNSEARCH solver if you're interested in using global optimization techniques.
