Extremum Seeking Control

Extremum seeking control (ESC) is a model-free, real-time adaptive control algorithm that is useful for adapting parameters to unknown system dynamics and unknown mappings from control parameters to an objective function. You can use extremum seeking to solve static optimization problems and to optimize parameters of dynamic systems.

The extremum seeking algorithm uses the following stages to tune a parameter value.

Modulation — Perturb the value of the parameter being optimized using a low-amplitude sinusoidal signal.
System Response — The system being optimized reacts to the parameter perturbations. This reaction causes a corresponding change in the objective function value.
Demodulation — Multiply the objective function signal by a sinusoid with the same frequency as the modulation signal. This stage includes an optional high-pass filter to remove bias from the objective function signal.
Parameter Update — Update the parameter value by integrating the demodulated signal. The parameter value corresponds to the state of the integrator. This stage includes an optional low-pass filter to remove high-frequency noise from the demodulated signal.

Simulink^® Control Design™ software implements this algorithm using the Extremum Seeking Control block. For examples of extremum-seeking control, see:

Time Domain

Using the Extremum Seeking Control block, you can implement both continuous-time and discrete-time controllers. The ESC algorithm is the same in both cases. Changing the time-domain of the controller affects the time domain of the high-pass filters, low-pass filters, and integrators used in the tuning loops.

To generate hardware-deployable code for the Extremum Seeking Control block, use a discrete-time controller.

The following table shows the continuous-time and discrete-time transfer functions for the filters and integrators in the Extremum Seeking Control block.

Controller Element	Continuous-Time Transfer Function	Discrete-Time Transfer Function
High-pass filter	$\frac{s}{s + ω_{h}}$	$\frac{1 - z^{- 1}}{1 - ω_{h} z^{- 1}}$
Low-pass Filter	$\frac{1}{s + ω_{l}}$	$\frac{1}{1 - ω_{l} z^{- 1}}$
Integrator	$\frac{1}{s}$	Forward Euler: $\frac{T_{s}}{z - 1}$ Backward Euler: $\frac{T_{s} z}{z - 1}$ Trapezoidal: $\frac{T_{s}}{2} \frac{z + 1}{z - 1}$

Controller Element

Continuous-Time Transfer Function

Discrete-Time Transfer Function

High-pass filter

$\frac{s}{s + ω_{h}}$

$\frac{1 - z^{- 1}}{1 - ω_{h} z^{- 1}}$

Low-pass Filter

$\frac{1}{s + ω_{l}}$

$\frac{1}{1 - ω_{l} z^{- 1}}$

Integrator

$\frac{1}{s}$

Forward Euler:

$\frac{T_{s}}{z - 1}$

Backward Euler:

$\frac{T_{s} z}{z - 1}$

Trapezoidal:

$\frac{T_{s}}{2} \frac{z + 1}{z - 1}$

Here:

ω_l is the low-pass filter cutoff frequency.
ω_h is the high-pass filter cutoff frequency.
T_s is the sample time of the discrete-time controller learning rate.

Static Optimization

To demonstrate extremum seeking, consider the following static optimization problem.

Extremum seeking control diagram showing the modulation, demodulation, and parameter update stages.

Here:

$\hat{θ}$ is the estimated parameter value.
θ is the modulation signal
y = f(θ) is the function output being maximized, that is, the objective function.
ω is the forcing frequency of the modulation and demodulation signals.
b·sin(ωt) is the modulation signal.
a·sin(ωt) is the demodulation signal.
k is the learning rate.

The optimum parameter value, θ*, occurs at the maximum value of f(θ).

To optimize multiple parameters, you use a separate tuning loop for each parameter.

The following figure demonstrates extremum seeking for an increasing portion of the objective function curve. The modulated signal θ is the sum of the current estimated parameter and the modulation signal. Applying f(θ) produces a perturbed objective function with the same phase as the modulation signal. Multiplying the perturbed objective function by the demodulation signal produces a positive signal. Integrating this signal increases the value of θ, which moves it closer to the peak of the objective function.

Graph of an objective function with modulation and demodulation demonstrated on an increasing portion of the curve.

The following figure demonstrates extremum seeking for a decreasing portion of the objective function curve. In this case, applying f(θ) produces a perturbed objective function that is 180 degrees out of phase from the modulation signal. Multiplying by the demodulation signal produces a negative signal. Integrating this signal decreases the value of θ, which moves it closer to the peak of the objective function.

Graph of an objective function with modulation and demodulation demonstrated on a decreasing portion of the curve

The following figure demonstrates extremum seeking for a flat portion of the objective function curve, that is, a portion of the curve near the maximum. In this case, applying f(θ) produces a near-zero perturbed objective function. Multiplying by the demodulation signal and integrating this signal does not significantly change the value of θ, which is already near its optimum value θ*.

Graph of an objective function with modulation and demodulation demonstrated on a nearly flat portion of the curve

Dynamic System Optimization

Extremum seeking optimization of a dynamic system occurs in a similar fashion as static optimization. However, in this case, the parameter θ affects the output of a time-dependent dynamic system. The objective function to be maximized is computed from the system output. The following figure shows the general tuning loop for a dynamic system.

Extremum seeking for a time-dependent dynamic system.

Here:

$\dot{x} = f (x, α (x, θ))$ is the state function of the dynamic system.
z = h(x) is the output of the dynamic system.
y = g(z) is the objective function derived from the output of the dynamic system.
ϕ₁ is the phase of the demodulation signal.
ϕ₂ is the phase of the modulation signal.

ESC Design Guidelines

When designing an extremum-seeking controller, consider the following guidelines.

Ensure that the system dynamics are on the fastest time scale, the forcing frequencies are on the medium time scale, and the filter cutoff frequencies are on the slowest time scale.
Specify an amplitude for the demodulation signal that is much greater than the modulation signal amplitude (a ≫ b).
Select phase angles for the modulation and demodulation signals such that cos(ϕ₁ – ϕ₂) > 0.
When tuning multiple parameters, the forcing frequency for each tuning loop must be different.
Try designing your system without high-pass and low-pass filters. If the performance is not satisfactory, you can then consider adding one or both filters.

More About Extremum Seeking Control

For more information on extremum seeking control, play the video. This video is part of the Learning-Based Control video series.

References

[1] Ariyur, Kartik B., and Miroslav Krstić. Real Time Optimization by Extremum Seeking Control. Hoboken, NJ: Wiley Interscience, 2003.