How does bayesian optimization and cross-validation work?

Question

Dimitri 2019-8-23

0
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/477302-how-does-bayesian-optimization-and-cross-validation-work

评论： Don Mathis 2021-1-25

Hello,

I was wondering how exactly the hyperparameter optimization works in this example: Example. The default setting is 5-fold cross-validation, but the output is a normal RegressionSVM and not a RegressionPartitionedSVM. That's how I understand the process, please give me feedback.

Let´s consider the first step of the hyperparameter optimization. The algorithm choses a initial hyperparameter setting and learns a model with 4/5 of the data. Now it evaluates the performance on the 1/5 of the data. What happens next? Is this hyperparameter setting used again and one model learned on another of the 4/5 data? After 5 iterations you now have 5 objectiv function values which are used for the calculation of the loss? This loss is the final loss for the first hyperparameter setting. This procedure is now repeated 30 times?

0 个评论
显示 -2更早的评论隐藏 -2更早的评论

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Don Mathis 2019-8-23

1
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/477302-how-does-bayesian-optimization-and-cross-validation-work#answer_388783

编辑：Don Mathis 2019-8-23

In each iteration of the optimization, fitrsvm is called with 5-fold crossvalidation, using a particular vector of hyperparameters. This results in a RegressionPartitionedSVM. Then the kfoldLoss method is called on that object, obtaining the Loss for that vector of hyperparameters. That Loss value is printed in the command line display in the "Objective" column for that iteration.

In the next iteration, a new vector of hyperparameters is chosen, and the process is repeated.

Finally, after 30 (by default) iterations, the "best" hyperparameter vector is chosen, and a final model is trained on the entire dataset using those hyperparameters, without crossvalidation. That final RegressionSVM model is returned.

3 个评论
显示 1更早的评论隐藏 1更早的评论

Sinan Islam 2021-1-23

@Don Mathis I optimized model with cross validation, now I dont know how to print the cross validation error. Using kfoldloss on optimized model is not working.

Don Mathis 2021-1-25