Demosaic Interpolator

Construct RGB pixel data from Bayer pattern pixel stream

Libraries:
Vision HDL Toolbox / Conversions

Description

The Demosaic Interpolator block provides a Bayer pattern interpolation filter for streaming video data. The block implements the calculations using hardware-efficient, multiplier-free algorithms for HDL code generation. You can select a low-complexity bilinear interpolation, or a moderate-complexity gradient-corrected bilinear interpolation.

When you select bilinear interpolation, the block operates on a 3-by-3 pixel window using only additions and bit shifts.
When you select gradient correction, the block operates on a 5-by-5 pixel window. The calculation is performed using bit shift, addition, and low-order canonical signed digit (CSD) multiplication.

Ports

This block uses a streaming pixel interface with a pixelcontrol bus for frame control signals. This interface enables the block to operate independently of image size and format. All Vision HDL Toolbox™ blocks use the same streaming interface. The block accepts and returns a scalar pixel value and a bus that contains five control signals. The control signals indicate the validity of each pixel and its location in the frame. To convert a frame (pixel matrix) into a serial pixel stream and control signals, use the Frame To Pixels block. For a full description of the interface, see Streaming Pixel Interface.

This block also supports multipixel streams. In that case, the pixel input is a vector of M-by-1 values, where M is number of pixels. The pixel output is a matrix of M-by-3 values. The M value corresponds to the Number of pixels parameter of the Frame To Pixels block, and each output pixel has three components in RGB color space.

Input

expand all

pixel — Input pixel stream
scalar | vector

For single pixel streams, specify pixel as a scalar. For multipixel streams, specify pixel as a vector of Number of pixels-by-1 pixel values. Number of pixels can be two, four, or eight. Images in the Bayer format have one color component for each pixel location. Select the sequence of R, G, and B pixels by using the Sensor alignment parameter.

The software supports double and single data types for simulation, but not for HDL code generation.

Data Types: uint | fixdt(0,W,F) | single | double

ctrl — Control signals associated with pixel stream
`pixelcontrol` bus

The pixelcontrol bus contains five signals. The signals describe the validity of the pixel and its location in the frame. For more information, see Pixel Control Bus.

For multipixel streaming, each vector of pixel values has one set of control signals. Because the vector has only one valid signal, the pixels in the vector must be either all valid or all invalid. The hStart and vStart signals apply to the pixel with the lowest index in the vector. The hEnd and vEnd signals apply to the pixel with the highest index in the vector.

Data Types: bus

Output

expand all

pixel — Output pixel stream
vector | matrix

For single pixel streams, the output pixel is a three-element vector of RGB values. For multipixel streams, the output pixel is a matrix of Number of pixels-by-3 RGB values. The block calculates the values of the missing color components for each pixel, using the method you specify in the Interpolation algorithm parameter.

The software supports double and single data types for simulation, but not for HDL code generation.

Data Types: uint | fixdt(0,W,F) | single | double

ctrl — Control signals associated with pixel stream
`pixelcontrol` bus

The pixelcontrol bus contains five signals. The signals describe the validity of the pixel and its location in the frame. For more information, see Pixel Control Bus.

Data Types: bus

Parameters

expand all

Interpolation algorithm — Algorithm used to calculate missing pixel values
`Gradient-corrected linear` (default) | `Bilinear`

Algorithm used to calculate missing pixel values, specified as one of the following:

Gradient-corrected linear — Bilinear average, corrected for intensity gradient
Bilinear — Average of the pixel values in the surrounding 3-by-3 neighborhood

Sensor alignment — Color sequence of the pixels in the input stream
`RGGB` (default) | `GBRG` | `GRBG` | `BGGR`

Select the sequence of R, G, and B pixels that correspond to the 2-by-2 block of pixels in the top-left corner of the input image. Specify the sequence in left-to-right, top-to-bottom order. For instance, the default sequence of RGGB represents an image with this pattern.

Two-by-two grid with R and G across the top and G and B across the bottom

Line buffer size — Size of line memory buffer
`2048` (default) | integer

Size of the line memory buffer, specified as a positive integer. Choose a power of two that accommodates the number of active pixels in a horizontal line. If you specify a value that is not a power of two, the buffer uses the next largest power of two.

When you use multipixel input, this value must accommodate (Active pixels per line)/(Number of pixels) + 2.

The total memory size allocated depends on your selection of Interpolation algorithm:

Bilinear: 2-by-Line buffer size memory locations
Gradient-corrected linear: 4-by-Line buffer size memory locations

Tips

When you use a block with an internal line buffer inside an Enabled Subsystem (Simulink), the enable signal pattern must maintain the timing of the pixel stream, including the minimum blanking intervals. If the enable pattern corrupts the timing of the pixel stream, you might see partial output frames, corrupted pixel stream control signals, or mismatches between Simulink^® and HDL simulation results. You may need to extend the blanking intervals to accommodate for cycles when the enable is low. For more information, see Configure Blanking Intervals.

Algorithms

expand all

The block pads the edges of the image with symmetric pixel values. For more details, see Edge Padding.

When you use multipixel streaming, the block uses a single line memory and implements a demosiac interpolation algorithm for each of the M input pixels, in parallel. This increase in hardware resources is a trade off for increasing throughput compared to single-pixel streaming.

Interpolation

Bilinear Interpolation

The block interpolates the missing color values by using a 3-by-3 neighborhood. The average is calculated over the adjacent two pixels or four pixels, depending on the sensor color pattern. The block implements this algorithm using only add and shift operations.

Gradient-Corrected Linear Interpolation

Gradient correction improves interpolation performance across edges by taking advantage of the correlation between the color channels. The block calculates the missing color values using bilinear interpolation, and then modifies the value corresponding to the intensity gradient calculated over a 5-by-5 neighborhood. The block applies the gradient correction using a fixed set of filter kernels. The filter coefficients were designed empirically to perform well over a wide range of image data. To enable an efficient hardware implementation, the coefficients are multiples of powers of two. For details of this interpolation algorithm, see [1].

Latency

The block buffers one line of input pixels before starting bilinear interpolation calculations. The gradient correction calculation starts after the block buffers two lines.

The latency of the block is the line buffer latency plus the latency of the kernel calculation. The line buffer latency includes edge padding by default. The latency of the padding operation depends on the size of the kernel. If edge padding is not necessary for your design, you can reduce the latency by setting the Padding method parameter to None. When you use this option, the block latency does not depend on your kernel size. To determine the exact latency for any configuration of the block, measure the number of time steps between the input and output control signals.

Note

When you use edge padding, use a horizontal blanking interval of at least twice the kernel width. This interval lets the block finish processing one line before it starts processing the next one, including adding padding pixels before and after the active pixels in the line.

The horizontal blanking interval is equal to TotalPixelsPerLine – ActivePixelsPerLine or, equivalently, FrontPorch + BackPorch. Standard streaming video formats use a horizontal blanking interval of about 25% of the frame width. This interval is much larger than the filters applied to each frame.

The horizontal blanking interval must also meet these minimums:

If the kernel size is less than 4, and you use edge padding, the total porch must be at least 8 pixels.
When you disable edge padding, the horizontal blanking interval must be at least 12 cycles and is independent of the kernel size.
The BackPorch must be at least 6 pixels. This parameter is the number of inactive pixels before the first valid pixel in a frame.

For more information, see Configure Blanking Intervals.

Performance

This table shows the resource use after synthesis of the block for the Xilinx^® Zynq^®-7000 SoC ZC706 Evaluation Kit with single-pixel uint8 input and the default parameter settings. The design achieves a clock frequency of 348 MHz.

Resource	Usage
Slice LUTs	1126
Slice Registers	1849
DSP48	1
Block RAM	2

References

[1] Malvar, Henrique S., Li-wei He, and Ross Cutler. “High-Quality Linear Interpolation for Demosaicing of Bayer-Patterned Color Images.” Microsoft Research, May 2004. http://research.microsoft.com/pubs/102068/Demosaicing_ICASSP04.pdf.

Extended Capabilities

C/C++ Code Generation
Generate C and C++ code using Simulink® Coder™.

This block supports C/C++ code generation for Simulink accelerator and rapid accelerator modes and for DPI component generation.

HDL Code Generation
Generate VHDL, Verilog and SystemVerilog code for FPGA and ASIC designs using HDL Coder™.

HDL Coder™ provides additional configuration options that affect HDL implementation and synthesized logic.

HDL Architecture

This block has one default HDL architecture.

HDL Block Properties

ConstrainedOutputPipeline	Number of registers to place at the outputs by moving existing delays within your design. Distributed pipelining does not redistribute these registers. The default is `0`. For more details, see ConstrainedOutputPipeline (HDL Coder).
InputPipeline	Number of input pipeline stages to insert in the generated code. Distributed pipelining and constrained output pipelining can move these registers. The default is `0`. For more details, see InputPipeline (HDL Coder).
OutputPipeline	Number of output pipeline stages to insert in the generated code. Distributed pipelining and constrained output pipelining can move these registers. The default is `0`. For more details, see OutputPipeline (HDL Coder).

Restrictions

You cannot generate HDL for this block inside a Resettable Synchronous Subsystem (HDL Coder).

Version History

Introduced in R2015a

expand all

R2022a: Two pixels-per-clock streaming

The block now supports multipixel streams that have 2 pixels per clock cycle.

R2021b: Multipixel streaming

The Demosaic Interpolator block now supports multipixel streams. The HDL implementation replicates the algorithm for each pixel in parallel.

The block accepts an input vector of NumPixels-by-1 values and returns an output matrix of NumPixels-by-3 values. The ctrl ports remain scalar, and the control signals in the pixelcontrol bus apply to all pixels in the matrix.

R2018b: Improved line buffer

The internal line buffer in this block now handles bursty data, that is, noncontiguous valid signals within a pixel line. This implementation uses fewer hardware resources due to improved padding logic and native support for kernel sizes with an even number of lines. This change affects the Line Buffer block and blocks that use an internal line buffer.

The latency of the line buffer is now reduced by a few cycles for some configurations. You might need to rebalance parallel path delays in your models. A best practice is to synchronize parallel paths in your models by using the pixel stream control signals rather than by inserting a specific number of delays.

Demosaic Interpolator

Description