Lane Detection with Zynq-Based Hardware

Setup Prerequisites

This example follows the algorithm development workflow that is detailed in the Developing Vision Algorithms for Zynq-Based Hardware example. If you have not already done so, please work through that example to gain the better understanding of the required workflow.

This algorithm corresponds to the Vision HDL Toolbox™ example, Lane Detection (Vision HDL Toolbox). With the SoC Blockset Support Package for AMD FPGA and SoC Devices, you get a hardware reference design that allows for easy integration of your targeted algorithm in the context of a vision system. The support package also provides a Video Capture HDMI block that, when deployed to the Zynq board, routes the video from FPGA output to the ARM processor for further computation.

If you have not yet done so, run through the guided setup wizard portion of the SoC Blockset Support Package for AMD FPGA and SoC Devices installation. You might have already completed this step when you installed this support package.

On the MATLAB Home tab, in the Environment section of the Toolstrip, click Add-Ons > Manage Add-Ons. Locate SoC Blockset Support Package for AMD FPGA and SoC Devices, and click Setup.

The guided setup wizard performs a number of initial setup steps, and confirms that the target can boot and that the host and target can communicate.

For more information, see Set Up Xilinx Devices.

Pixel-Stream Model

This model contains two major subsystems: Lane Detection Algorithm which performs inverse perspective mapping and lane-marking candidate extraction, and Lane Fit and Overlay Algorithm which does lane fit and overlays the lanes onto the birds-eye view.

open_system('vzLaneDetection_01_PixelStream')

The Lane Detection Algorithm subsystem is targeted to the FPGA, and the Lane Fit and Overlay Algorithm subsystem is targeted to the ARM processor. The birds-eye view and lane coordinates outputs of the Lane Detection Algorithm subsystem are passed to the Lane Fit and Overlay Algorithm subsystem. The result is displayed in Simulink by the BirdsEyeLaneOverlay block.

Hardware/Software Synchronization: Some logic is required to synchronize communication between the FPGA design and the ARM processor portion of the design. The Lane Detection Algorithm subsystem contains two shift registers that accumulate the lane coordinates for each birds-eye view frame. The Lane Detection Algorithm and Lane Fit and Overlay Algorithm subsystems use state machines based on the dataReady and swStart signals to keep the two subsystems synchronized. The state machines must see dataReady before exchanging the lane coordinates, and then wait for swStart before accumulating the next set of coordinates.

The inverse perspective mapping algorithm takes 2.5 frames to complete, so the lane fitting must be done in 0.5 frames. If the lane fitting takes longer, it would miss the next set of lane coordinates. To meet this constraint, the Simulink rates of the signals dataReady, swStart, LeftLane and RightLane must be: 1/720 (swStart) + 1/720 (dataReady) + 1/180 (LeftLane and RightLane) = 1/120 (0.5 frames at 60fps).

Instead of working on full images, the HDL-ready lane detection algorithm works on streaming pixel data. The blocks in the shaded areas convert to and from pixel stream signals in preparation for targeting.

Video Source The source video for this example comes from either the From Multimedia File block, that reads video data from a multimedia file, or from the Video Capture HDMI block, that captures live video frames from an HDMI source connected to the Zynq-based hardware. To configure the source, right-click on the variant selection icon in the lower-left corner of the Image Source block, choose Label mode active choice, and select either File or HW.

The model features a Video Frame Buffer block that provides a simplified simulation model of a frame buffer implemented in external memory. The Video Frame Buffer block is configured for a pixel format of RGB, and a video resolution of 640*480p. In the targeted design, the frame buffer connections will interface with the external memory on the chosen Zynq platform.

The Video Frame Buffer block input interface features the video pixel ports {R, G, B}, corresponding pixel control bus port {pixelCtrl}, and a frame buffer trigger {pop} port. The video stream that is to be stored in the frame buffer, and the corresponding video timing signals, are provided on the pixel and control bus ports. The frame buffer pop port is used to schedule the release of the stored video frame from the frame buffer. The release of the frame from the frame buffer is controlled from within the Lane Detection Algorithm subsystem, and should be asserted high for a single clock cycle. Once triggered, the frame buffer will release the stored frame on the video pixel output ports {R, G, B} with corresponding video timing signals on the pixel control bus port {pixelCtrl}.

For this algorithm, the model must be configured with a pixel format of RGB, and frame size of 640x480. Both the From Multimedia File and Video Capture blocks are configured to deliver video frames in this format.

During the first two frames of simulation output, the To Video Display shows a black image. This condition indicates that no image data is available. This behavior is because the inverse perspective mapping algorithm takes 2.5 frames to complete.

Target the Algorithm

After you are satisfied with the pixel streaming algorithm simulation, you can target the pixel algorithm to the FPGA on the Zynq board.

Start the targeting workflow by right clicking the Lane Detection Algorithm subsystem and selecting HDL Code > HDL Workflow Advisor.

The video stream from the algorithm subsystem to the frame buffer is mapped to the Frame Buffer Master interface, and the video stream from the frame buffer to the algorithm subsystem is mapped to the Frame Buffer Slave interface. The frame buffer pop signal is also be mapped as part of the Frame Buffer Master interface.

With reference to Target platform interface table, map the RFromFrameBuf port to the Frame Buffer Slave interface, and select R from the dropdown menu in the Bit Range / Address / FPGA Pin column. Similarly, select Frame Buffer Slave as the interface for the GFromFrameBuf, BFromFrameBuf and ctrlFromFrameBuf ports, and G, B and Pixel Control Bus from the Bit Range / Address / FPGA Pin column respectively.

Map the RToFrameBuf, GToFrameBuf, BToFrameBuf, ctrlToFrameBuf, and framePop ports to the Frame Buffer Master interface, and select R, G, B, Pixel Control Bus, and Frame Trigger from the dropdown menu in the Bit Range / Address / FPGA Pin column respectively.

Also, map the swStart, dataReady, LeftLane, and RightLane to AXI4-Lite for software interaction.

Execute each step in sequence to experience the full workflow, or, if you are already familiar with preparation and HDL code generation phases, right-click Step 4.1 in the table of contents on the left hand side and select Run to selected task.

Steps 4.3 and 4.4

The rest of the workflow generates a bitstream for the FPGA, downloads it to the target, and reboots the board.

Because this process can take 20-40 minutes, you can choose to bypass this step by using a pre-generated bitstream for this example that ships with product and was placed on the SDCard during setup.

To use this pre-generated bitstream execute the following:

>> vz = visionzynq();
>> changeFPGAImage(vz,'visionzynq-zc706-hdmicam-lane_detection.bit');

To use a bitstream for another platform, replace 'zc706' with the platform name.

Alternatively, you can continue with Steps 4.3 and 4.4.

Using the Generated Models from the HDL Workflow Advisor

Step 4.2 generated two, or four, models depending on whether Embedded Coder® is installed: A 'targeted hardware interface' model and associated library model, and a 'software interface' model and associated library model. The 'targeted hardware interface' model can be used to run Simulink algorithms that interact with the pixel-streaming design running on the FPGA. These algorithms work on video captured from the output of the FPGA, and also can read and write AXI-Lite registers. The 'software interface' model supports full software targeting to the ARM processor when Embedded Coder and the Embedded Coder Support Package for AMD SoC Devices are installed, enabling External mode simulation, Processor-in-the-loop, and full deployment.

The library models are created so that any changes to the hardware generation model are propagated to any custom targeted hardware simulation or software interface models that exist.

Setup Video Playback

When running either of the generated models, which run the targeted portions of the algorithm on the board, you must provide an HDMI input source. For instance, replay the provided 640x480 front-facing vehicle camera source video, vzLaneDetection640.mp4, by connecting the HDMI input of the board as the secondary display of the Simulink host computer. To configure your secondary display to use 640x480 resolution, see Configure Display for VGA Playback.

In this example, the video timing after inverse perspective mapping is not compliant with the HDMI standard. The output of the inverse perspective is enabled by using external memory as an output display frame buffer. By default, the frame buffer is configured in YCbCr 4:2:2 format. To view the output of the inverse perspective mapping on the HDMI output, the frame buffer must be configured for RGB pixel format. To configure the frame buffer, open the Getting Started example model.

In the Getting Started model, configure these Video Capture HDMI block parameters:

On the To Video Display block, set these parameters:

Run the Getting Started model to configure the frame buffer. Once you can see output on the external monitor, play the source video using full-screen mode and set to repeat.

Alternatively, check the Bypass FPGA user logic option on the Video Capture block. This option reroutes the input video directly to the output HDMI display.

Leave the source video running and close the Getting Started model.

Targeted Hardware Interface Model

open_system('vzLaneDetection_02_HardwareInterface')

The Video Capture HDMI block in this model returns the birds-eye view video from the output of the FPGA to Simulink. The FPGA processes the HDMI input video that you set up in the previous step. The Lane Detection Algorithm block is the generated interface to the FPGA. It returns lane coordinates to Simulink corresponding to each birds-eye view frame. Using this captured data, the lane fit and overlay algorithm runs in Simulink.

Software Interface Model

You can run this model in External mode on the ARM processor, or you can use this model to fully deploy a software design. (This model is generated only if Embedded Coder and the Embedded Coder Support Package for AMD SoC Devices are installed.)

open_system('vzLaneDetection_03_SoftwareInterface')

Before running this model, you must perform additional setup steps to configure the AMD cross-compiling tools. For more information, see Setup for ARM Targeting with IP Core Generation Workflow.

The software interface model included with this example has some setting changes to enable running the algorithm on the ARM processor. When you generate your own software interface model, make the following changes.

To avoid buffering errors when running the Video Viewer in External mode, reduce the duration of the External mode trigger. In the Code menu, select External Mode Control Panel. Click the Signal & Triggering button. In the Trigger options section, set Duration to 1.

Change these Configuration Parameters:

Run the model in External mode. This mode runs the algorithm on the ARM processor on the Zynq board.

The Video Capture HDMI and Lane Detection Algorithm blocks in this model work the same as in the Targeted Hardware Interface Model. When you run in External mode, the entire model runs on the ARM processor. The birds-eye view output video from the FPGA is captured to the ARM processor. The lane coordinates from the FPGA are passed to the ARM processor using an AXI-Lite interface. The result of the ARM processing is displayed in Simulink by the Video Viewer.