Hampel Filter

Filter outliers using Hampel identifier

Libraries:
DSP System Toolbox / Filtering / Filter Designs

Description

The Hampel Filter block detects and removes the outliers of the input signal by using the Hampel identifier. The Hampel identifier is a variation of the three-sigma rule of statistics, which is robust against outliers. For each sample of the input signal, the block computes the median of a window composed of the current sample and $\frac{L e n - 1}{2}$ adjacent samples on each side of the current sample. Len is the window length you specify through the Window length parameter. The block also estimates the standard deviation of each sample about its window median by using the median absolute deviation. If a sample differs from the median by more than the threshold multiplied by the standard deviation, the filter replaces the sample with the median. For more information, see Algorithms.

Ports

Input

expand all

x — Data input
vector | matrix

The block accepts multichannel inputs, that is, m-by-n size inputs, where m ≥ 1, and n ≥ 1. m is the number of samples in each frame (channel), and n is the number of channels. The block also accepts variable-size inputs. That is, you can change the size of each input channel during simulation. However, the number of channels cannot change.

This port is unnamed until you select the Specify threshold from input port parameter.

Data Types: single | double

T — Threshold
real scalar greater than or equal to 0

Threshold for outlier detection, specified as a real scalar greater than or equal to 0. For information on how this parameter is used to detect the outlier, see Algorithms.

Dependencies

This port appears when you select the Specify threshold from input port parameter.

Data Types: single | double

Output

expand all

y — Filtered output
vector | matrix

The size and data type of this output matches the size and data type of the input.

This port is unnamed until you select the Output outlier status check box.

Data Types: single | double

outlier — Output the outlier status
vector | matrix

A value of 1 in this output indicates that the corresponding element in the input is an outlier. This output has the same size as the input.

Dependencies

To enable this port, select the Output outlier status check box.

Data Types: Boolean

Parameters

expand all

If a parameter is listed as tunable, then you can change its value during simulation.

Window length — Length of sliding window
`11` (default) | positive odd scalar integer

Length of the sliding window, specified as a positive odd scalar integer. The window of finite length slides over the data, and the block computes the median and median absolute deviation of the data in the window.

Specify threshold from input port — Flag to specify threshold
off (default) | on

When you select this check box, the threshold is input through the T port. When you clear this check box, the threshold is specified on the block dialog through the Threshold for outlier detection (standard deviations) parameter.

Threshold for outlier detection (standard deviations) — Threshold
3 (default) | real scalar greater than or equal to 0

Threshold for outlier detection, specified as a real scalar greater than or equal to 0. For information on how this parameter is used to detect the outlier, see Algorithms.

Tunable: Yes

Dependencies

This parameter appears when you clear the Specify threshold from input port check box.

Output outlier status — Flag to output the outlier status
off (default) | on

Select this parameter to output a matrix of boolean values that has the same size as the input. Each element in this matrix indicates whether the corresponding element in the input is an outlier. A value of 1 indicates an outlier.

Simulate using — Type of simulation to run
`Interpreted execution` (default) | `Code generation`

Specify the type of simulation to run. You can set this parameter to:

Interpreted execution –– Simulate model using the MATLAB^® interpreter. This option shortens startup time.
Code generation –– Simulate model using generated C code. The first time you run a simulation, Simulink^® generates C code for the block. The C code is reused for subsequent simulations as long as the model does not change. This option requires additional startup time but provides faster subsequent simulations.

Block Characteristics

Data Types	`double` \| `single`
Direct Feedthrough	`no`
Multidimensional Signals	`no`
Variable-Size Signals	`yes`
Zero-Crossing Detection	`no`

More About

expand all

Hampel Identifier

The Hampel identifier is a variation of the three-sigma rule of statistics that is robust against outliers.

Given a sequence x₁, x₂, x₃, …, x_n and a sliding window of length k, define point-to-point median and standard-deviation estimates using:

Local median — $m_{i} = median (x_{i - k}, x_{i - k + 1}, x_{i - k + 2}, \dots, x_{i}, \dots, x_{i + k - 2}, x_{i + k - 1}, x_{i + k})$
Standard deviation — $σ_{i} = κ median (| x_{i - k} - m_{i} |, \dots, | x_{i + k} - m_{i} |),$ where $κ = \frac{1}{\sqrt{2} {erfc}^{- 1} (1 / 2)} \approx 1.4826$

The quantity σ_i /κ is known as the median absolute deviation (MAD).

If a sample x_i is such that

$| x_{i} - m_{i} | > n_{σ} σ_{i}$

for a given threshold n_σ, then the Hampel identifier declares x_i an outlier and replaces it with m_i. If n_σ is 0, then the Hampel filter behaves as a regular median filter.

Algorithms

For a given sample of data, x_s, the algorithm:

Centers the window of odd length at the current sample.
Computes the local median, m_i, and standard deviation, σ_i, over the current window of data.
Compares the current sample with n_σ × σ_i, where n_σ is the threshold value. If $| x_{s} - m_{i} | > n_{σ} \times σ_{i}$ , the filter identifies the current sample, x_s, as an outlier and replaces it with the median value, m_i.

Consider a frame of data that is passed into the Hampel filter.

In this example, the Hampel filter slides a window of length 5 (Len) over the data. The filter has a threshold value of 2 (n_σ). To have a complete window at the beginning of the frame, the filter algorithm prepends the frame with Len – 1 zeros. To compute the first sample of the output, the window centers on the ${[\frac{L e n - 1}{2} + 1]}^{th}$ sample in the appended frame, the third zero in this case. The filter computes the median, median absolute deviation, and the standard deviation over the data in the local window.

Current sample: x_s = 0.
Window of data: win = [0 0 0 0 1].
Local median: m_i = median([0 0 0 0 1]) = 0.
Median absolute deviation: $m a d_{i} = median (| x_{i - k} - m_{i} |, \dots, | x_{i + k} - m_{i} |)$ . For this window of data, $m a d = median (| 0 - 0 |, \dots, | 1 - 0 |) = 0$ .
Standard deviation: σ_i = κ × mad_i = 0, where $κ = \frac{1}{\sqrt{2} {erfc}^{- 1} (1 / 2)} \approx 1.4826$ .
The current sample, x_s = 0, does not obey the relation for outlier detection.

$[| x_{s} - m_{i} | = 0] > [(n_{σ} \times σ_{i}) = 0]$
Therefore, the Hampel filter outputs the current input sample, x_s = 0.

Repeat this procedure for every succeeding sample until the algorithm centers the window on the ${[E n d - \frac{L e n - 1}{2}]}^{th}$ sample, marked as End. Because the window centered on the last $\frac{L e n - 1}{2}$ samples cannot be full, these samples are processed with the next frame of input data.

Here is the first output frame the Hampel filter generates:

The seventh sample of the appended input frame, 23, is an outlier. The Hampel filter replaces this sample with the median over the local window [4 9 23 8 12].

References

[1] Bodenham, Dean. “Adaptive Filtering and Change Detection for Streaming Data.” Ph.D. Thesis. Imperial College, London, 2012.

[2] Liu, Hancong, Sirish Shah, and Wei Jiang. “On-line outlier detection and data cleaning.” Computers and Chemical Engineering. Vol. 28, March 2004, pp. 1635–1647.

[3] Suomela, Jukka. Median Filtering Is Equivalent to Sorting, 2014.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using Simulink® Coder™.

Version History

Introduced in R2017a

Hampel Filter

Description

Ports

Input

x — Data input
vector | matrix

T — Threshold
real scalar greater than or equal to 0

Dependencies

Output

y — Filtered output
vector | matrix

outlier — Output the outlier status
vector | matrix

Dependencies

Parameters

Window length — Length of sliding window
`11` (default) | positive odd scalar integer

Specify threshold from input port — Flag to specify threshold
off (default) | on

Threshold for outlier detection (standard deviations) — Threshold
3 (default) | real scalar greater than or equal to 0

Dependencies

Output outlier status — Flag to output the outlier status
off (default) | on

Simulate using — Type of simulation to run
`Interpreted execution` (default) | `Code generation`

Block Characteristics

More About

Hampel Identifier

Algorithms

References

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using Simulink® Coder™.

Version History

See Also

Functions

Objects

Blocks

Hampel Filter

Description

Ports

Input

x — Data input vector | matrix

T — Threshold real scalar greater than or equal to 0

Dependencies

Output

y — Filtered output vector | matrix

outlier — Output the outlier status vector | matrix

Dependencies

Parameters

Window length — Length of sliding window 11 (default) | positive odd scalar integer

Specify threshold from input port — Flag to specify threshold off (default) | on

Threshold for outlier detection (standard deviations) — Threshold 3 (default) | real scalar greater than or equal to 0

Dependencies

Output outlier status — Flag to output the outlier status off (default) | on

Simulate using — Type of simulation to run Interpreted execution (default) | Code generation

Block Characteristics

More About

Hampel Identifier

Algorithms

References

Extended Capabilities

C/C++ Code Generation Generate C and C++ code using Simulink® Coder™.

Version History

See Also

Functions

Objects

Blocks

x — Data input
vector | matrix

T — Threshold
real scalar greater than or equal to 0

y — Filtered output
vector | matrix

outlier — Output the outlier status
vector | matrix

Window length — Length of sliding window
`11` (default) | positive odd scalar integer

Specify threshold from input port — Flag to specify threshold
off (default) | on

Threshold for outlier detection (standard deviations) — Threshold
3 (default) | real scalar greater than or equal to 0

Output outlier status — Flag to output the outlier status
off (default) | on

Simulate using — Type of simulation to run
`Interpreted execution` (default) | `Code generation`

C/C++ Code Generation
Generate C and C++ code using Simulink® Coder™.