bayesopt: how do I know when the hyperparameter optimization has converged?

Question

Roland W. Fleming 2018-8-21

1
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/415657-bayesopt-how-do-i-know-when-the-hyperparameter-optimization-has-converged

评论： Alan Weiss 2019-12-12

This is a conceptual question. We are using Bayesian hyperparameter search to find optimal hyper parameters for our neural networks. Over time (i.e. with increasing iterations of bayesopt), the estimates of the optimal hyperparameter values change, and the objective improves (obviously). But how do we know when it has converged on a final set of values? What criterion can we use to determine whether more iterations of the optimization are needed?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Alan Weiss 2018-8-21

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/415657-bayesopt-how-do-i-know-when-the-hyperparameter-optimization-has-converged#answer_333577

Sorry to say, this is almost impossible to determine. It is a very difficult problem to determine stopping conditions for any optimization, and when your problem includes large data, neural networks and the like, it is even harder.

For some examples why I say this, see Isolated Global Minimum, Global Vs. Local Minimum Using ga, When the Solver Succeeds, and When the Solver Might Have Succeeded.

Alan Weiss

MATLAB mathematical toolbox documentation

4 个评论
显示 2更早的评论隐藏 2更早的评论

Chandrakanth Reddy Kancharla 2019-12-12

在 MATLAB Online 中打开

Hey Alan,

I saw the documentation that you have mentioned. But is it the same when using bayesopt with fit models like fitcknn, fitcsvm, etc.?

Context: I have few different datasets, I have written script that loops through these datasets and tries to fit a classifier model for each dataset using fitcknn that uses bayesopt as optimizer. Sometimes it takes more iterations and sometimes less to get to a MinObjective of '0'. What I am trying to do is stop the iterations once the MinObjective goes to '0'.

I tried using it as a part of the HyperparameterOptimizationOptions input argument. But that definitely seems to be wrong as it says there is no 'OutputFunc' as input. Here is the part of the function that uses the fitcknn model. Could you please point me in the right direction for the MinObjective based stop function implementation.

Thank you very much.

CrossValPart = cvpartition(FaultIndTrain, 'KFold',10);
    
opts = struct('Optimizer','bayesopt','CVPartition',CrossValPart, 'AcquisitionFunctionName','expected-improvement-plus',...
        'MaxObjectiveEvaluations',60, 'Verbose', 1, 'ShowPlots', false);
Classifier = fitcknn(ExtFeatTrain, FaultIndTrain,...
        'OptimizeHyperparameters','all','HyperparameterOptimizationOptions',opts);

Alan Weiss 2019-12-12

To use an output function you will have to use bayesopt directly, not indirectly in a fitcknn as you have done. Write your objective function to call fitcknn, and then use the classification error as the objective function (or whatever objective suits you). For a similar example, see Optimize a Cross-Validated SVM Classifier Using bayesopt.

Good luck,

Alan Weiss

MATLAB mathematical toolbox documentation

请先登录，再进行评论。