Main Content

Ground Plane Segmentation and Obstacle Detection on NVIDIA Jetson Xavier™ NX Embedded platform

This example shows ground plane segmentation of 3-D lidar data from a vehicle on NVIDIA® embedded platforms to find nearby obstacles. The example uses ground plane segmentation and obstacle detection application to illustrate:

  • C++ and CUDA® code generation for the ground plane segmentation and obstacle detection algorithm by using MATLAB® Coder™ and GPU Coder™.

  • Verify behavior of the generated code on the target platform by using processor-in-the-loop (PIL) simulation.

  • Compare of the performance of the application on the CPU (C++) and the GPU (CUDA).

Third-Party Prerequisites

Target Board Requirements

  • NVIDIA Jetson Xavier™ NX Embedded platform.

  • NVIDIA CUDA toolkit installed on the board.

  • Environment variables on the target board for the compilers and libraries. For more information, see Install and Setup Prerequisites for NVIDIA Boards (MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms).

Development Host Requirements

  • NVIDIA CUDA toolkit installed on the host.

  • Environment variables for the compilers and libraries. For information on the supported versions of the compilers and libraries, see Third-Party Hardware. For setting up the environment variables, see Setting Up the Prerequisite Products.

Configure and Verify NVIDIA Target Platform

Connect to NVIDIA Hardware

The MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE™ Platforms uses SSH connection over TCP/IP to execute commands while building and running the generated code on the Jetson platforms. Connect the target platform to the same network as the host computer.

To communicate with the NVIDIA hardware, create a live hardware connection object by using the jetson (MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms) function. This example uses the device address, user name, and password settings from the most recent successful connection to the Jetson hardware.

hwobj = jetson;

Configure PIL Simulation

This example uses processor-in-the-loop (PIL) simulation to test the generated C++ and CUDA code on the Jetson board. Because the input data transfer and algorithm computations consumes time, change the default higher PIL timeout value to prevent time-out errors.

setPILTimeout(hwobj,100);

Verify GPU Environment

To verify that the compilers and libraries necessary for running this example are set up correctly, use the coder.checkGpuInstall function.

envCfg = coder.gpuEnvConfig('jetson');
envCfg.BasicCodegen = 1;
envCfg.Quiet = 1;
envCfg.HardwareObject = hwobj;
coder.checkGpuInstall(envCfg);

Configure Code Generation Parameters

To generate a PIL executable that runs on the ARM® CPU of the Jetson board, create a coder.EmbeddedCodeConfig object for a static library.

cfgCpu = coder.config('lib');

Set the target language for the generated code to C++ and enable PIL execution in the code configuration object. Then, enable execution-time profiling during PIL execution. Execution-time profiling generates metrics for tasks and functions in the generated code. For more information, see Create Execution-Time Profile for Generated Code (Embedded Coder). Finally, create a coder.hardware object for the Jetson platform and assign it to the Hardware property of the code configuration object.

cfgCpu.TargetLang = 'C++';
cfgCpu.VerificationMode = 'PIL';
cfgCpu.CodeExecutionProfiling = true;
cfgCpu.Hardware = coder.hardware('NVIDIA Jetson');

Similarly, create configuration parameters for the CUDA GPU on the Jetson board by using coder.gpuConfig.

cfgGpu = coder.gpuConfig('lib');
cfgGpu.VerificationMode = 'PIL';
cfgGpu.CodeExecutionProfiling = true;
cfgGpu.Hardware = coder.hardware('NVIDIA Jetson');

The segmentGroundAndObstacles Entry-Point Function

The segmentGroundAndObstacles entry-point function segments points belonging to the ground plane, the ego vehicle, and nearby obstacles from the input point cloud locations. The following diagram illustrates the algorithm implemented in the entry-point function. For more information, see Ground Plane and Obstacle Detection Using Lidar (Automated Driving Toolbox).

segmentGroundAndObstacles.png

type segmentGroundAndObstacles

Generate and Run Executables on the Target

The point cloud data from the lidar sensor is of size 32-by-1100-by-3. Due to signal misses and noise, a few points may be dropped from this point cloud data. So, the second dimension might vary with an upper bound of 1100. Create the input type for the entry-point function segmentGroundAndObstacles with varying dimensions for the second argument using the coder.typeof function.

codegenArgs = {coder.typeof(single(0),[32,1100,3],[0,1,0])};

Generate and Run C++ Executable

Generate C++ code with the CPU code configuration object cfgCpu.

codegen -config cfgCpu -args codegenArgs segmentGroundAndObstacles -report

The obstacleDetectionWrapper is an execution wrapper function that processess the streaming lidar input data frame-by-frame, calls the PIL executable, and displays the 3-D point cloud with segmenting points belonging to the ground plane, the ego vehicle, and nearby obstacles. The lidar data used in this example was recorded using a Velodyne HDL32E sensor mounted on a vehicle. For an explanation of the processing performed by the obstacleDetectionWrapper function, see Ground Plane and Obstacle Detection Using Lidar (Automated Driving Toolbox).

obstacleDetectionWrapper();

cpuFrame.png

Execution Profile of the C++ Executable

Clear the PIL executable and collect the execution time profile by using the getCoderExecutionProfile function.

clear segmentGroundAndObstacles_pil;
cpuExecutionProfile = getCoderExecutionProfile('segmentGroundAndObstacles');

Generate and Run CUDA Executable

Generate CUDA code with the GPU code configuration object cfgGpu.

codegen -config cfgGpu -args codegenArgs segmentGroundAndObstacles -report

To maximize the GPU performance, use the jetson_clocks.sh script on the board. For more information, see NVIDIA Xavier - Maximizing Performance (RidgeRun wiki).

ClockFileStatus = system(hwobj, 'test -f l4t_dfs.conf && echo "1" || echo "0"');
if ~str2double(ClockFileStatus)
    system(hwobj,'echo "ubuntu" | sudo -S jetson_clocks --store');
end
system(hwobj,'echo "ubuntu" | sudo -S jetson_clocks');

Use the obstacleDetectionWrapper to process the streaming lidar input data, calls the PIL executable, and display the 3-D point cloud with segmenting points belonging to the ground plane, the ego vehicle, and nearby obstacles.

obstacleDetectionWrapper();

gpuFrame.png

Execution Profile of the CUDA Executable

Disable Jetson clock settings, clear the PIL executable and collect the execution time profile by using the getCoderExecutionProfile function.

system(hwobj,'echo "ubuntu" | sudo -S jetson_clocks --restore');
clear segmentGroundAndObstacles_pil;
gpuExecutionProfile = getCoderExecutionProfile('segmentGroundAndObstacles');

Analysis of CPU and GPU Execution Profiles

Get the per frame execution times of CPU and GPU from their execution profiles by using the ExecutionTimeInSeconds property.

[~,cpuExecTimePerFrame,~] = cpuExecutionProfile.Sections.ExecutionTimeInSeconds;
[~,gpuExecTimePerFrame,~] = gpuExecutionProfile.Sections.ExecutionTimeInSeconds;

To plot the per frame execution times use the code mentioned below:

figure;
% Plot CPU execution times.
plot(cpuExecTimePerFrame(2:end)*1000,'r');
hold on;
% Plot GPU execution times.
plot(gpuExecTimePerFrame(2:end)*1000,'b');
grid on;
% Set the title, legend and labels.
title('CPU vs GPU Per-frame Execution times (in ms)');
legend('GPU Timings', 'CPU Timings');
axis([0,1240,0,40]);
xlabel('Frame Number');
ylabel('Execution Time (in ms)');

Supporting Functions

obstacleDetectionWrapper processess the lidar data, calls the PIL executable, and visualize the results. For an explanation of the processing performed by the obstacleDetectionWrapper function, see Ground Plane and Obstacle Detection Using Lidar (Automated Driving Toolbox).

function obstacleDetectionWrapper()
%OBSTACLEDETECTIONWRAPPER process lidar data and visualize results
%   The OBSTACLEDETECTIONWRAPPER is an execution wrapper function that
%   processess the streaming lidar input data frame-by-frame, calls the PIL
%   executable, and displays the 3-D point cloud with segmenting points
%   belonging to the ground plane, the ego vehicle, and nearby obstacles.

fileName    = 'lidarData_ConstructionRoad.pcap';
deviceModel = 'HDL32E';
veloReader = velodyneFileReader(fileName, deviceModel);

% Setup Streaming Point Cloud Display
xlimits = [-25 45];   % Limits of point cloud display, meters
ylimits = [-25 45];
zlimits = [-20 20];

% Create a pcplayer
lidarViewer = pcplayer(xlimits, ylimits, zlimits);
xlabel(lidarViewer.Axes, 'X (m)')
ylabel(lidarViewer.Axes, 'Y (m)')
zlabel(lidarViewer.Axes, 'Z (m)')

% Set the colormap for labeling the ground plane, ego vehicle, and nearby
% obstacles.
colorLabels = [...
    0      0.4470 0.7410; ... % Unlabeled points, specified as [R,G,B]
    0.4660 0.6740 0.1880; ... % Ground points
    0.9290 0.6940 0.1250; ... % Ego points
    0.6350 0.0780 0.1840];    % Obstacle points

% Define indices for each label
colors.Unlabeled = 1;
colors.Ground    = 2;
colors.Ego       = 3;
colors.Obstacle  = 4;

% Set the colormap
colormap(lidarViewer.Axes, colorLabels)

% Stop processing the frame after specified time.
stopTime = veloReader.EndTime;

i = 1;
isPlayerOpen = true;
while hasFrame(veloReader) && veloReader.CurrentTime < stopTime && isPlayerOpen

    % Grab the next lidar scan
    ptCloud = readFrame(veloReader);

    % Segment points belonging to the ego vehicle
    [egoPoints,groundPoints,obstaclePoints] = segmentGroundAndObstacles_pil(ptCloud.Location);

    i = i+1;
    closePlayer = ~hasFrame(veloReader);

    % Update lidar display
    points = struct('EgoPoints',egoPoints, 'GroundPoints',groundPoints, ...
        'ObstaclePoints',obstaclePoints);
    isPlayerOpen = helperUpdateView(lidarViewer, ptCloud, points, colors, closePlayer);
end
snapnow
end

helperUpdateView updates the streaming point cloud display with the latest point cloud and associated color labels.

function isPlayerOpen = helperUpdateView(lidarViewer,ptCloud,points,colors,closePlayer)
%HELPERUPDATEVIEW update streaming point cloud display
%   isPlayerOpen =
%   HELPERUPDATEVIEW(lidarViewer,ptCloud,points,colors,closePlayers)
%   updates the pcplayer object specified in lidarViewer with a new point
%   cloud ptCloud. Points specified in the struct points are colored
%   according to the colormap of lidarViewer using the labels specified by
%   the struct colors. closePlayer is a flag indicating whether to close
%   the lidarViewer.

if closePlayer
    hide(lidarViewer);
    isPlayerOpen = false;
    return;
end

scanSize = size(ptCloud.Location);
scanSize = scanSize(1:2);

% Initialize colormap
colormapValues = ones(scanSize, 'like', ptCloud.Location) * colors.Unlabeled;

if isfield(points, 'GroundPoints')
    colormapValues(points.GroundPoints) = colors.Ground;
end

if isfield(points, 'EgoPoints')
    colormapValues(points.EgoPoints) = colors.Ego;
end

if isfield(points, 'ObstaclePoints')
    colormapValues(points.ObstaclePoints) = colors.Obstacle;
end

% Update view
view(lidarViewer, ptCloud.Location, colormapValues)

% Check if player is open
isPlayerOpen = isOpen(lidarViewer);

end

See Also

Objects

  • (MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms) | (MATLAB Coder Support Package for NVIDIA Jetson and NVIDIA DRIVE Platforms)

Related Topics