Bicyclist and Pedestrian Classification by Using FPGA

This example uses:

Deep Learning HDL Toolbox Deep Learning HDL Toolbox
Deep Learning Toolbox Deep Learning Toolbox
Deep Learning HDL Toolbox Support Package for Xilinx FPGA and SoC Devices Deep Learning HDL Toolbox Support Package for Xilinx FPGA and SoC Devices

This example shows how to deploy a custom trained series network to detect pedestrians and bicyclists based on their micro-Doppler signatures. This network is taken from the Pedestrian and Bicyclist Classification Using Deep Learning example from the Phased Array Toolbox. For more details on network training and input data, see Pedestrian and Bicyclist Classification Using Deep Learning (Radar Toolbox).

Prerequisites

Xilinx™ Vivado™ Design Suite 2020.2
Zynq® UltraScale+™ MPSoC ZCU102 Evaluation Kit
Deep learning HDL Toolbox™ Support Package for Xilinx® FPGA and SoC Devices
Deep Learning Toolbox™
Deep Learning HDL Toolbox™

The data files used in this example are:

The MAT File trainedNetBicPed.mat contains a model trained on training data set trainDataNoCar and its label set trainLabelNoCar.
The MAT File testDataBicPed.mat contains the test data set testDataNoCar and its label set testLabelNoCar.

Load Data and Network

Load a pretrained network. Load test data and its labels.

load('trainedNetBicPed.mat','trainedNetNoCar')
load('testDataBicPed.mat')

View the layers of the pre-trained series network:

analyzeNetwork(trainedNetNoCar);

Set up HDL Toolpath

Set up the path to your installed Xilinx™ Vivado™ Design Suite 2023.1 executable if it is not already set up. For example, to set the toolpath, enter:

% hdlsetuptoolpath('ToolName', 'Xilinx Vivado','ToolPath', 'C:\Vivado\2023.1\bin');

Create Target Object

Create a target object for your target device with a vendor name and an interface to connect your target device to the host computer. Interface options are JTAG (default) and Ethernet. Vendor options are Intel or Xilinx. Use the installed Xilinx Vivado Design Suite over an Ethernet connection to program the device.

hT = dlhdl.Target('Xilinx', 'Interface', 'Ethernet');

Create Workflow Object

Create an object of the dlhdl.Workflow class. When you create the object, specify the network and the bitstream name. Specify the saved pre-trained series network, trainedNetNoCar, as the network. Make sure the bitstream name matches the data type and the FPGA board that you are targeting. In this example, the target FPGA board is the Zynq UltraScale+ MPSoC ZCU102 board. The bitstream uses a single data type.

hW = dlhdl.Workflow('Network', trainedNetNoCar, 'Bitstream', 'zcu102_single', 'Target', hT);

Compile `trainedNetNoCar` Series Network

To compile the trainedNetNoCar series network, run the compile function of the dlhdl.Workflow object.

dn = hW.compile;

### Optimizing series network: Fused 'nnet.cnn.layer.BatchNormalizationLayer' into 'nnet.cnn.layer.Convolution2DLayer'
          offset_name          offset_address    allocated_space 
    _______________________    ______________    ________________

    "InputDataOffset"           "0x00000000"     "28.0 MB"       
    "OutputResultOffset"        "0x01c00000"     "4.0 MB"        
    "SystemBufferOffset"        "0x02000000"     "28.0 MB"       
    "InstructionDataOffset"     "0x03c00000"     "4.0 MB"        
    "ConvWeightDataOffset"      "0x04000000"     "4.0 MB"        
    "FCWeightDataOffset"        "0x04400000"     "4.0 MB"        
    "EndOffset"                 "0x04800000"     "Total: 72.0 MB"

Program the Bitstream onto FPGA and Download Network Weights

To deploy the network on the Zynq® UltraScale+™ MPSoC ZCU102 hardware, run the deploy function of the dlhdl.Workflow object. This function uses the output of the compile function to program the FPGA board by using the programming file.The function also downloads the network weights and biases. The deploy function checks for the Xilinx Vivado tool and the supported tool version. It then starts programming the FPGA device by using the bitstream, displays progress messages and the time it takes to deploy the network.

hW.deploy;

### FPGA bitstream programming has been skipped as the same bitstream is already loaded on the target FPGA.
### Deep learning network programming has been skipped as the same network is already loaded on the target FPGA.

Run Predictions on Micro-Doppler Signatures

Classify one input from the sample test data set by using the predict function of the dlhdl.Workflow object and display the label. The inputs to the network correspond to the sonograms of the micro-Doppler signatures for a pedestrian or a bicyclist or a combination of both.

testImg = single(testDataNoCar(:, :, :, 1));
testLabel = testLabelNoCar(1);
classnames = trainedNetNoCar.Layers(end).Classes;

% Get predictions from network on single test input
score = hW.predict(testImg, 'Profile', 'On')

### Finished writing input activations.
### Running single input activations.


              Deep Learning Processor Profiler Performance Results

                   LastLayerLatency(cycles)   LastLayerLatency(seconds)       FramesNum      Total Latency     Frames/s
                         -------------             -------------              ---------        ---------       ---------
Network                    9430692                  0.04287                       1            9430707             23.3
    conv_module            9411355                  0.04278 
        conv_1             4178753                  0.01899 
        maxpool_1          1394883                  0.00634 
        conv_2             1975197                  0.00898 
        maxpool_2           706156                  0.00321 
        conv_3              813598                  0.00370 
        maxpool_3           121790                  0.00055 
        conv_4              148165                  0.00067 
        maxpool_4            22255                  0.00010 
        conv_5               41999                  0.00019 
        avgpool2d             8674                  0.00004 
    fc_module                19337                  0.00009 
        fc                   19337                  0.00009 
 * The clock frequency of the DL processor is: 220MHz

score = 1×5 single row vector

    0.9956    0.0000    0.0000    0.0044    0.0000

[~, idx1] = max(score);
predTestLabel = classnames(idx1)

predTestLabel = categorical
     ped

Load five random images from the sample test data set and execute the predict function of the dlhdl.Workflow object to display the labels alongside the signatures. The predictions will happen at once since the input is concatenated along the fourth dimension.

numTestFrames = size(testDataNoCar, 4);
numView = 5;
listIndex = randperm(numTestFrames, numView);
testImgBatch = single(testDataNoCar(:, :, :, listIndex));
testLabelBatch = testLabelNoCar(listIndex);

% Get predictions from network using DL HDL Toolbox on FPGA
[scores, speed] = hW.predict(testImgBatch, 'Profile', 'On');

### Finished writing input activations.
### Running single input activations.


              Deep Learning Processor Profiler Performance Results

                   LastLayerLatency(cycles)   LastLayerLatency(seconds)       FramesNum      Total Latency     Frames/s
                         -------------             -------------              ---------        ---------       ---------
Network                    9446929                  0.04294                       5           47138869             23.3
    conv_module            9427488                  0.04285 
        conv_1             4195175                  0.01907 
        maxpool_1          1394705                  0.00634 
        conv_2             1975204                  0.00898 
        maxpool_2           706332                  0.00321 
        conv_3              813499                  0.00370 
        maxpool_3           121869                  0.00055 
        conv_4              148063                  0.00067 
        maxpool_4            22019                  0.00010 
        conv_5               42053                  0.00019 
        avgpool2d             8684                  0.00004 
    fc_module                19441                  0.00009 
        fc                   19441                  0.00009 
 * The clock frequency of the DL processor is: 220MHz

[~, idx2] = max(scores, [], 2);
predTestLabelBatch = classnames(idx2);

% Display the micro-doppler signatures along with the ground truth and
% predictions.
for k = 1:numView
    index = listIndex(k);
    imagesc(testDataNoCar(:, :, :, index));
    axis xy
    xlabel('Time (s)')
    ylabel('Frequency (Hz)')
    title('Ground Truth: '+string(testLabelNoCar(index))+', Prediction FPGA: '+string(predTestLabelBatch(k)))
    drawnow;
    pause(3);
end

The image shows the micro-Doppler signatures of two bicyclists (bic+bic) which is the ground truth. The ground truth is the classification of the image against which the network prediction is compared. The network prediction retrieved from the FPGA correctly predicts that the image has two bicyclists.