Main Content

Code Generation for Lidar Object Detection Using PointPillars Deep Learning

This example uses:

Open Live Script

This example shows how to generate CUDA® MEX for a PointPillars object detector. For more information, see Lidar 3-D Object Detection Using PointPillars Deep Learning (Lidar Toolbox) example from the Lidar Toolbox™.

Third-Party Prerequisites

Required

  • CUDA enabled NVIDIA® GPU and compatible driver.

Optional

For non-MEX builds such as static and dynamic libraries or executables, this example has the following additional requirements.

Verify GPU Environment

To verify that the compilers and libraries for running this example are set up correctly, use the coder.checkGpuInstall function.

envCfg = coder.gpuEnvConfig('host');
envCfg.DeepLibTarget = 'cudnn';
envCfg.DeepCodegen = 1;
envCfg.Quiet = 1;
coder.checkGpuInstall(envCfg);

Pretrained PointPillars Network

Load the pretrained pointPillarsObjectDetector (Lidar Toolbox) trained in the Lidar 3-D Object Detection Using PointPillars Deep Learning example. To train the detector yourself, see Lidar 3-D Object Detection Using PointPillars Deep Learning (Lidar Toolbox). 

matFile = 'pretrainedPointPillarsDetector.mat';
pretrainedDetector = load('pretrainedPointPillarsDetector.mat','detector');
detector = pretrainedDetector.detector;

pointpillarsDetect Entry-Point Function

The pointpillarsDetect entry-point function takes in the point cloud and confidence threshold as input and passes them to a trained pointPillarsObjectDetector (Lidar Toolbox) for prediction through the pointpillarDetect function. The pointpillarsDetect function loads the detector object from the MAT file into a persistent variable and reuses the persistent object for subsequent prediction calls.

type('pointpillarsDetect.m')
function [bboxes,scores,labels] = pointpillarsDetect(matFile,dataLoc,dataInt,threshold)
% Predict the output of network and extract the confidence, x, y,
% width, height, and class.

% load the deep learning network for prediction
persistent pointPillarObj;

if isempty(pointPillarObj)
    pointPillarObj = coder.loadDeepLearningNetwork(matFile);
end

ptCloud = pointCloud(dataLoc,'Intensity',dataInt);

[bboxes,scores,labels] = pointPillarObj.detect(ptCloud,'Threshold',threshold);
end

Evaluate the Detector for Object Detection

Read the point cloud.

pc = pcread('pandasetDrivingData.pcd');

Use the detect method on the pretrained detector.

confidenceThreshold = 0.7;
[bboxes,~,labels] = detect(detector,pc,'Threshold',confidenceThreshold);
bboxesCar = bboxes(labels == 'Car',:);
bboxesTruck = bboxes(labels == 'Truck',:);

Display the detections on the point cloud.

helperDisplay3DBoxesOverlaidPointCloud(pc.Location,bboxesCar,'green',...
                      bboxesTruck,'magenta','Predicted bounding boxes');

Figure contains an axes object. The axes object with title Predicted bounding boxes contains an object of type scatter.

Generate CUDA MEX

To generate CUDA® code for the pointpillarsDetect entry-point function, create a GPU code configuration object for a MEX target and set the target language to C++. Use the coder.DeepLearningConfig function to create a cuDNN deep learning configuration object and assign it to the DeepLearningConfig property of the GPU code configuration object. 

cfg = coder.gpuConfig('mex');
cfg.TargetLang = 'C++';
cfg.DeepLearningConfig = coder.DeepLearningConfig(TargetLibrary='cudnn');

dataLoc = pc.Location;
dataInt = pc.Intensity;

args = {coder.Constant(matFile) coder.typeof(dataLoc,[Inf,3],[1 0]) coder.typeof(dataInt,[Inf,1],[1 0]) coder.typeof(confidenceThreshold)};

codegen -config cfg pointpillarsDetect -args args -report
Code generation successful: View report

Run the Generated MEX

Call the generated CUDA MEX with the point cloud. Display the results.

[bboxes,~,labels] = pointpillarsDetect_mex(matFile,dataLoc,dataInt,confidenceThreshold);
bboxesCar = bboxes(labels == 'Car',:);
bboxesTruck = bboxes(labels == 'Truck',:);

helperDisplay3DBoxesOverlaidPointCloud(pc.Location,bboxesCar,'green',...
                      bboxesTruck,'magenta','Predicted bounding boxes');

Figure contains an axes object. The axes object with title Predicted bounding boxes contains an object of type scatter.

Helper Functions

function helperDisplay3DBoxesOverlaidPointCloud(ptCld,labelsCar,carColor,...
    labelsTruck,truckColor,titleForFigure)
% Display the point cloud with different colored bounding boxes for different
% classes
    figure;
    ax = pcshow(ptCld);
    showShape('cuboid',labelsCar,'Parent',ax,'Opacity',0.1,'Color',...
        carColor,'LineWidth',0.5);
    hold on;
    showShape('cuboid',labelsTruck,'Parent',ax,'Opacity',0.1,'Color',...
        truckColor,'LineWidth',0.5);
    title(titleForFigure);
    zoom(ax,1.5);
end

References

[1] Lang, Alex H., Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. "PointPillars: Fast Encoders for Object Detection From Point Clouds." In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12689-12697. Long Beach, CA, USA: IEEE, 2019. https://doi.org/10.1109/CVPR.2019.01298.

[2] Hesai and Scale. PandaSet. https://scale.com/open-datasets/pandaset.