bayesopt: how do I know when the hyperparameter optimization has converged?
显示 更早的评论
This is a conceptual question. We are using Bayesian hyperparameter search to find optimal hyper parameters for our neural networks. Over time (i.e. with increasing iterations of bayesopt), the estimates of the optimal hyperparameter values change, and the objective improves (obviously). But how do we know when it has converged on a final set of values? What criterion can we use to determine whether more iterations of the optimization are needed?
采纳的回答
Alan Weiss
2018-8-21
1 个投票
Sorry to say, this is almost impossible to determine. It is a very difficult problem to determine stopping conditions for any optimization, and when your problem includes large data, neural networks and the like, it is even harder.
For some examples why I say this, see Isolated Global Minimum, Global Vs. Local Minimum Using ga, When the Solver Succeeds, and When the Solver Might Have Succeeded.
Alan Weiss
MATLAB mathematical toolbox documentation
4 个评论
Alexander
2019-8-14
Sorry for resurrecting the thread, but my question would have exactly the same topic.
Is there a way to provide a custom stopping function? Like
result = bayesopt(..., 'StoppingFunction', @customCriteriaFunction); ?
If not, I'm forced to do something like
results = bayesopt(..., 'MaxObjectiveEvaluations', N);
while % customCriteria not fulfilled
results = resume(results);
end
with N set to a number suitable to wait for, after setting a debug breakpoint to stop optimization deliberately.
Alan Weiss
2019-8-14
To stop an optimization early, use an output function. Set the OutputFcn name-value pair. For an example, see Bayesian Optimization Output Functions.
Alan Weiss
MATLAB mathematical toolbox documentation
Chandrakanth Reddy Kancharla
2019-12-12
Hey Alan,
I saw the documentation that you have mentioned. But is it the same when using bayesopt with fit models like fitcknn, fitcsvm, etc.?
Context: I have few different datasets, I have written script that loops through these datasets and tries to fit a classifier model for each dataset using fitcknn that uses bayesopt as optimizer. Sometimes it takes more iterations and sometimes less to get to a MinObjective of '0'. What I am trying to do is stop the iterations once the MinObjective goes to '0'.
I tried using it as a part of the HyperparameterOptimizationOptions input argument. But that definitely seems to be wrong as it says there is no 'OutputFunc' as input. Here is the part of the function that uses the fitcknn model. Could you please point me in the right direction for the MinObjective based stop function implementation.
Thank you very much.
CrossValPart = cvpartition(FaultIndTrain, 'KFold',10);
opts = struct('Optimizer','bayesopt','CVPartition',CrossValPart, 'AcquisitionFunctionName','expected-improvement-plus',...
'MaxObjectiveEvaluations',60, 'Verbose', 1, 'ShowPlots', false);
Classifier = fitcknn(ExtFeatTrain, FaultIndTrain,...
'OptimizeHyperparameters','all','HyperparameterOptimizationOptions',opts);
Alan Weiss
2019-12-12
To use an output function you will have to use bayesopt directly, not indirectly in a fitcknn as you have done. Write your objective function to call fitcknn, and then use the classification error as the objective function (or whatever objective suits you). For a similar example, see Optimize a Cross-Validated SVM Classifier Using bayesopt.
Good luck,
Alan Weiss
MATLAB mathematical toolbox documentation
更多回答（1 个）
Greg Heath
2018-8-22
For good estimators I tend to be satisfied when the squared error is at least 100 times smaller than that of the reference naive guess
outref = mean(target)
with
MSEref = mean(var(target',0))
i.e.,
MSEgoal = 0.01*MSEref
I have hundreds of examples in both comp.soft-sys.matlab and ANSWERS.
Hope this helps.
Thank you for formally accepting my answer
Greg
类别
在 帮助中心 和 File Exchange 中查找有关 Gaussian Process Regression 的更多信息
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!
Translated by 
