Nelder-Mead optimization with equality constraints

2016 5 6
1 个回答
回答已采纳
更新时间：2016 5 6
5 次查看（30 天）

0 个投票

Hello
I would like to use the function fminsearch of matlab to search for the best hyperparameters of my SVM with a weighted RBF kernel classifier. fminsearch uses the Nelder-Meader simplex method.
Let's say I have the following hyperparameters: C, gamma, w1....wn where wi are the weights of the kernel.
Additionally, I have the constraint that sum(wi) = 1, i.e. all weights must sum up to one.
Is there a possibility to use Nelder-Mead with this equality constraint? I know that there is the fminsearchbnd method for Matlab but I think it can handle only boundary inequality constraints.

请先登录，再回答此问题。

 采纳的回答

Alan Weiss
Alan Weiss 2016-5-6

0 个投票

I would imagine that you also have the constraints that several or all of your parameters must be positive.
It might be difficult to have all those constraints satisfied at the same time using just fminsearch, or even fminsearchbnd. You might want to use fmincon or patternsearch to ease your search for the best hyperparameters.
It is likely that you would do best by setting up a rough grid of hyperparameters, search them, and then start a local search using patternsearch or fmincon starting at the best grid point.
Good luck,
Alan Weiss
MATLAB mathematical toolbox documentation

1 个评论
显示 -1更早的评论 隐藏 -1更早的评论

Sepp
Sepp 2016-5-6
编辑：Sepp 2016-5-6
Hi Alan
Thank you very much for your answer, I really appreciate it.
Yes, you are right, all parameters must be positive, i.e. 0 <= w(i) <= 1
I have to clarify a bit. I'm using a SVM classifier and the weights are used in a weighted RBF kernel (one weight for each feature). The parameters to estimate are thus C, gamma and the weights. The cost function is the accuracy.
An option would be to allow w(i) to go above 1, i.e. only having the constraint 0 <= w(i). Then I could just normalize the w(i) to sum to one inside my cost function. Would this work with Nelder-Mead or would the method get stuck?
Does fmincon only work for smooth and linear cost function? Because I think my cost function (accuracy) is not linear.
Regarding your suggestion to first using a grid, I think I could also using random search or simulated annealing as a start.

请先登录，再进行评论。

更多回答（0 个）

类别

在 帮助中心 和 File Exchange 中查找有关 Surrogate Optimization 的更多信息

标签

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by