Parallel Bayesian Optimization
Optimize in Parallel
Running Bayesian optimization in parallel can save time. Running in parallel
requires Parallel Computing Toolbox™.
bayesopt performs parallel objective function
evaluations concurrently on parallel workers.
To optimize in parallel:
bayesopt— Set the
UseParallelname-value argument to
true. For example,
results = bayesopt(fun,vars,'UseParallel',true);
Fit functions — Set the
UseParallelfield of the
true. For example,
Mdl = fitcsvm(X,Y,'OptimizeHyperparameters','auto',... 'HyperparameterOptimizationOptions',struct('UseParallel',true))
Parallel Bayesian Algorithm
The parallel Bayesian optimization algorithm is similar to the serial algorithm, which is described in Bayesian Optimization Algorithm. The differences are:
bayesoptassigns points to evaluate to the parallel workers, generally one point at a time.
bayesoptcalculates on the client to determine which point to assign.
bayesoptevaluates the initial random points, it chooses points to evaluate by fitting a Gaussian process (GP) model. To fit a GP model while some workers are still evaluating points,
bayesoptimputes a value to each point that is still on a worker. The imputed value is the mean of the GP model value at the points it is evaluating, or some other value as specified by the
'ParallelMethod'name-value argument. For parallel optimization of fit functions,
bayesoptuses the default
bayesoptassigns a point to evaluate, and before it computes a new point to assign, it checks whether too many workers are idle. The threshold for active workers is determined by the
MinWorkerUtilizationname-value argument. If too many workers are idle, then
bayesoptassigns random points, chosen uniformly within bounds, to all idle workers. This step causes the workers to be active more quickly, but the workers have random points rather than fitted points. If the number of idle workers does not exceed the threshold, then
bayesoptchooses a point to evaluate as usual, by fitting a GP model and maximizing the acquisition function.
Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results.
Settings for Best Parallel Performance
Fit functions have no special settings for better parallel performance. In
bayesopt settings can help to speed an
GPActiveSetSize option to a smaller value than
the default (
300) can speed the solution. The cost is
potential inaccuracy in the points that
bayesopt chooses to
evaluate, because the GP model of the objective function can be less accurate
than with a larger value. Setting the option to a larger value can result in a
more accurate GP model, but requires more time to create the model.
ParallelMethod option to
'max-observed' can lead
search more widely for a global optimum. This choice can lead to a better
solution in less time. However, the default value of
'clipped-model-prediction' is often best.
MinWorkerUtilization option to a large value
can result in higher parallel utilization. However, this setting causes more
completely random points to be evaluated, which can lead to less accurate
solutions. A large value, in this context, depends on how many workers you have.
The default is
the number of parallel workers. Setting the option to a lower value can give
lower parallel utilization, but with the benefit of higher quality
Placing the Objective Function on Workers
You can place an objective function on the parallel workers in one of three ways. Some have better performance, but require a more complex setup.
1. Automatic If you give a function handle as
the objective function,
bayesopt sends the handle to all
the parallel workers at the beginning of its run. For example,
load ionosphere splits = optimizableVariable('splits',[1,100],'Type','integer'); minleaf = optimizableVariable('minleaf',[1,100],'Type','integer'); fun = @(params)kfoldLoss(fitctree(X,Y,'Kfold',5,... 'MaxNumSplits',params.splits,'MinLeaf',params.minleaf)); results = bayesopt(fun,[splits,minleaf],'UseParallel',true);
This method is effective if the handle is small, or if you run the optimization only once. However, if you plan to run the optimization several times, you can save time by using one of the other two techniques.
2. Parallel constant If you plan to run an
optimization several times, save time by transferring the objective function to
the workers only once. This technique is especially effective when the function
handle incorporates a large amount of data. Transfer the objective once by
setting the function handle to a
parallel.pool.Constant (Parallel Computing Toolbox)
construct, as in this example.
load ionosphere splits = optimizableVariable('splits',[1,100],'Type','integer'); minleaf = optimizableVariable('minleaf',[1,100],'Type','integer'); fun = @(params)kfoldLoss(fitctree(X,Y,'Kfold',5,... 'MaxNumSplits',params.splits,'MinLeaf',params.minleaf)); C = copyFunctionHandleToWorkers(fun); results1 = bayesopt(C,[splits,minleaf],'UseParallel',true); results2 = bayesopt(C,[splits,minleaf],'UseParallel',true,... 'MaxObjectiveEvaluations',50); results3 = bayesopt(C,[splits,minleaf],'UseParallel',true,... 'AcquisitionFunction','expected-improvement');
In this example,
copyFunctionHandleToWorkers sends the
function handle to the workers only once.
3. Create objective function on workers If
you have a great deal of data to send to the workers, you can avoid loading the
data in the client by using
spmd (Parallel Computing Toolbox) to load the data on the
workers. Use a
Composite (Parallel Computing Toolbox) with
parallel.pool.Constant to access the distributed
% makeFun is at the end of this script spmd fun = makeFun(); end % ObjectiveFunction is now a Composite. Get a parallel.pool.Constant % that refers to it, without copying it to the client: C = parallel.pool.Constant(fun); % You could also use the line % C = parallel.pool.Constant(@MakeFun); % In this case, you do not use spmd % Call bayesopt, passing the Constant splits = optimizableVariable('splits', [1 100]); minleaf = optimizableVariable('minleaf', [1 100]); bo = bayesopt(C,[splits minleaf],'UseParallel',true); function f = makeFun() load('ionosphere','X','Y'); f = @fun; function L = fun(Params) L = kfoldLoss(fitctree(X,Y, ... 'KFold', 5,... 'MaxNumSplits',Params.splits, ... 'MinLeaf', Params.minleaf)); end end
In this example, the function handle exists only on the workers. The handle never appears on the client.
Differences in Parallel Bayesian Optimization Output
bayesopt runs in parallel, the Bayesian optimization
output includes these differences.
Iterative Display — Iterative display includes a column showing the number of active workers. This is the number after
bayesoptassigns a job to the next worker.
Objective Function Model plot (
@plotObjectiveModel) shows the pending points (those points executing on parallel workers). The height of the points depends on the
Elapsed Time plot (
@plotElapsedTime) shows the total elapsed time with the label Real time and the total objective function evaluation time, summed over all workers, with the label Objective evaluation time (all workers). Objective evaluation time includes the time to start a worker on a job.