Bayesian Optimization tries to find the best parameter values(eg. Network section depth) that minimizes the value of the objective function (neural network loss function) in a bounded domain (parameter range). Bayesian Optimization consists of two main components: a Bayesian statistical model and an acquisition function. The Bayesian statistical model, also referred to as the surrogate model, is used for approximating the objective function: it provides a Bayesian posterior probability distribution that describes potential values for the objective function at any candidate point. This posterior distribution is updated each time we evaluate the objective function at a hyperparameter candidate(eg. network depth). The surrogate model for Bayesian optimization in MATLAB is a Gaussian process, which defines a prior over functions that are updated in every iteration.
At every iteration we find the new hyperparameter value (optimal network depth) that maximizes the acquisition function a(x). We can simultaneously optimize multiple hyperparameters in every iteration, for eg. Network depth, l2 regularization, etc.
If you would like to understand more about the Bayesian optimization, please refer to the following links:
1. https://www.mathworks.com/help/releases/R2020b/stats/bayesian-optimization-algorithm.html
