Automate Ground Truth Labeling for Lidar Point Cloud Semantic Segmentation Using Lidar Labeler

Lidar Labeler App

Good ground truth data is crucial for the development and performance evaluation of automated driving and flight algorithms. However, creating and maintaining a diverse, high-quality, and labeled data set requires significant effort. The Lidar Labeler app provides a framework to automate the labeling process using the AutomationAlgorithm class. You can create a custom algorithm and use it in the app to label your entire data set. You can also edit the results to account for challenging scenarios missed by the algorithm.

In this example, you:

Segment Point Cloud Using SqueezeSegV2 Network

Segment the point cloud using a pretrained SqueezeSegV2 network. For information on how to train a SqueezeSegV2 network yourself, see Lidar Point Cloud Semantic Segmentation Using SqueezeSegV2 Deep Learning Network. This pretrained network is for organized point clouds. For information on how to convert unorganized point clouds to organized point clouds, see Unorganized to Organized Conversion of Point Clouds Using Spherical Projection.

Download Pretrained Network

Download the pretrained SqueezeSegV2 network, which has been trained on the PandaSet data set.

outputFolder = fullfile(tempdir,"Pandaset");
preTrainedMATFile = fullfile(outputFolder,"trainedSqueezeSegV2PandasetNet.mat");
preTrainedZipFile = fullfile(outputFolder,"trainedSqueezeSegV2PandasetNet.zip");

if ~exist(preTrainedMATFile,"file")
    if ~exist(preTrainedZipFile,"file")
        disp("Downloading pretrained model (5 MB)...");
        component = "lidar";
        filename = "data/trainedSqueezeSegV2PandasetNet.zip";
        preTrainedZipFile = matlab.internal.examples.downloadSupportFile(component,filename);
    end
    unzip(preTrainedZipFile,outputFolder);
end

Download Lidar Data Set

Download the PandaSet data set from Hesai and Scale, and save the lidar data to a Pandaset folder in a temporary folder. Note that the data set is 5.2 GB in size, and the code suspends MATLAB® execution until the download process is complete. To avoid suspending MATLAB execution during the download you can download the data set to your local disk, and then extract the file.

lidarDataTarFile = fullfile(outputFolder,"Pandaset_LidarData.tar.gz");
if ~exist(lidarDataTarFile,"file")
    disp("Downloading Pandaset Lidar driving data (5.2GB)...");
    component = "lidar";
    filename = "data/Pandaset_LidarData.tar.gz";
    lidarDataTarFile = matlab.internal.examples.downloadSupportFile(component,filename);
    untar(lidarDataTarFile,outputFolder);
end

% Check if tar.gz file is downloaded, but not uncompressed.
if ~exist(fullfile(outputFolder,"Lidar"),"file")
    untar(lidarDataTarFile,outputFolder);
end

Predict Segmentation Result on Point Cloud

Use the trained network to predict results on a point cloud by following these steps:

% Load the pretrained network.
outputFolder = fullfile(tempdir,"Pandaset");
load(fullfile(outputFolder,"trainedSqueezeSegV2PandasetNet.mat"),"net");

% Read the point cloud.
ptCloud = pcread(fullfile(outputFolder,"Lidar","0001.pcd"));

% Convert the point cloud to 5-channel image.
im = helperPointCloudToImage(ptCloud);

% Predict the segmentation result.
predictedResult = semanticseg(im,net);

% Display sematic segmentation result on point cloud.
helperDisplayLabelOverlaidPointCloud(im,predictedResult);
view([39.2 90.0 60])
title("Semantic Segmentation Result on Point Cloud")

Define Lidar Semantic Segmentation Algorithm Class in Lidar Labeler

Open the Lidar Labeler app and load the PandaSet point cloud sequence.

pointCloudDir = fullfile(outputFolder,'Lidar');
lidarLabeler(pointCloudDir);

In the ROI Labels tab in the left pane, click Label. Define 12 ROI labels with the names Vegetation, Ground, Road, RoadMarkings, SideWalk, Car, Truck, OtherVehicle, Pedestrian, RoadBarriers, Signs, and Buildings, of label type Voxel. Optionally, you can select colors for the labels. Click OK.

For illustration purposes, this example runs the algorithm on a subset of the PandaSet point cloud frames. Select the time range to label. Specify the first 15 seconds of the data set by entering 0 in the Start Time box and 15 in the End Time box. A pair of red flags appear on the range slider, indicating the selected time interval. The app displays the signal frames from only this interval, and applies the automation algorithm to only this interval.

On the LABEL tab of the app toolstrip, in the Automate Labeling section, click Select Algorithm > Add Algorithm > Create New Algorithm. The app opens a lidar.labeler.AutomationAlgorithm class that enables you to define a custom automation algorithm. You can also use this class to define a user-interface within the app to run the algorithm. For more information, see Create Automation Algorithm for Labeling.

Now, define an automation class for the lidar semantic segmentation algorithm. The class inherits from the lidar.labeler.AutomationAlgorithm abstract base class. The base class defines properties and signatures for methods that the app uses to configure and run the custom algorithm. The LidarSemanticSegmentation class is based on this template, and provides you with a ready-to-use automation class for semantic segmentation in a point cloud. The comments of the class outline the basic steps required to implement each API call.

Algorithm Properties

First, define the properties that determine the name and description of the algorithm, as well as the directions for using the algorithm.

% ----------------------------------------------------------------------
% Step 1: Define the required properties describing the algorithm. This
% includes Name, Description, and UserDirections.
properties(Constant)
    
    % Name Algorithm Name
    %   Character vector specifying the name of the algorithm.
    Name = 'Lidar Semantic Segmentation';
    
    % Description Algorithm Description
    %   Character vector specifying the short description of the algorithm.
    Description = 'Segment the point cloud using SqueezeSegV2 network';
    
    % UserDirections Algorithm Usage Directions
    %   Cell array of character vectors specifying directions for
    %   algorithm users to follow to use the algorithm.
    UserDirections = {['ROI Label Definition Selection: select one of ' ...
        'the ROI definitions to be labeled'], ...
        'Run: Press RUN to run the automation algorithm. ', ...
        ['Review and Modify: Review automated labels over the interval ', ...
        'using playback controls. Modify/delete/add ROIs that were not ' ...
        'satisfactorily automated at this stage. If the results are ' ...
        'satisfactory, click Accept to accept the automated labels.'], ...
        ['Accept/Cancel: If the results of automation are satisfactory, ' ...
        'click Accept to accept all automated labels and return to ' ...
        'manual labeling. If the results of automation are not ' ...
        'satisfactory, click Cancel to return to manual labeling ' ...
        'without saving the automated labels.']};
end

Custom Properties

Next, define the custom properties required for the core algorithm.

% ---------------------------------------------------------------------
% Step 2: Define properties you want to use during the algorithm
% execution.
properties
    % AllCategories
    % AllCategories holds the default 'unlabelled', 'Vegetation',
    % 'Ground', 'Road', 'RoadMarkings', 'SideWalk', 'Car', 'Truck', 
    % 'OtherVehicle', 'Pedestrian', 'RoadBarriers', 'Signs',
    % 'Buildings' categorical types.
    AllCategories = {'unlabelled'};

    % PretrainedNetwork
    %   PretrainedNetwork saves the pretrained SqueezeSegV2 network.
    PretrainedNetwork  
end

Function Definitions

For the third step, define the function used to check for valid data and label definitions.

The checkSignalType function checks if the signal data is supported for automation. The lidar semantic segmentation algorithm supports signals of type PointCloud.

function isValid = checkSignalType(signalType)            
    % Only point cloud signal data is valid for the Lidar Vehicle
    % detector algorithm.
    isValid = (signalType == vision.labeler.loading.SignalType.PointCloud);           
end

The checkLabelDefinition function checks if the label definition is the appropriate type for automation. The lidar semantic segmentation algorithm requires the Voxel label type.

function isValid = checkLabelDefinition(algObj, labelDef)
    % Only Voxel ROI label definitions are valid for the Lidar
    %  semantic segmentation algorithm.
    isValid = labelDef.Type == lidarLabelType.Voxel;
    if isValid
        algObj.AllCategories{end+1} = labelDef.Name;
    end
end

The checkSetup function checks if an ROI label definition is selected for automation.

function isReady = checkSetup(algObj)            
    % Is there one selected ROI Label definition to automate.
    isReady = ~isempty(algObj.SelectedLabelDefinitions);
end

Execution Functions

Specify the execution functions. The initialize function populates the initial algorithm state based on the existing labels in the app. In this example, the initialize function loads the pretrained semantic segmentation network and saves it to the PretrainedNetwork property of the algorithm object.

function initialize(algObj,~)           
    % Load the pretrained SqueezeSegV2 semantic segmentation network.
    outputFolder = fullfile(tempdir,'Pandaset');
    pretrainedSqueezeSeg = load(fullfile(outputFolder,'trainedSqueezeSegV2PandasetNet.mat'));
    % Store the network in the 'PretrainedNetwork' property of this object.
    algObj.PretrainedNetwork = pretrainedSqueezeSeg.net;          
end

The run function defines the core lidar semantic segmentation algorithm of this automation class. The algorithm calls the run function for each frame of the point cloud sequence. The function expects the automation class to return a set of labels. You can extend the algorithm to any category the network is trained on. For the purposes of this example, restrict the network to segment voxels of the categories vegetation, ground, road, road markings, sidewalk, cars, trucks, other vehicles, pedestrian, road barrier, signs, and buildings.

function autoLabels = run(algObj, pointCloud)    
    % Setup categorical matrix with categories including default 
    % 'unlabelled', 'Vegetation', 'Ground', 'Road', 'RoadMarkings', 
    % 'SideWalk', 'Car', 'Truck', 'OtherVehicle', 'Pedestrian', 
    % 'RoadBarriers', and 'Signs'.
    autoLabels = categorical(zeros(size(pointCloud.Location,1),size(pointCloud.Location,2)), ...
        0:12,algObj.AllCategories);

    % Convert the input point cloud to five channel image.
    I = helperPointCloudToImage(pointCloud);

    % Predict the segmentation result.
    predictedResult = semanticseg(I,algObj.PretrainedNetwork);
    autoLabels(:) = predictedResult;
end

The terminate function handles any cleanup or tear-down required after the automation is done. This algorithm does not require any cleanup, so the function is empty.

Use Lidar Semantic Segmentation Automation Class in App

To use the properties and methods implemented in the LidarSemanticSegmentation automation algorithm class file with Lidar Labeler, you must import the algorithm into the app.

First, create the folder structure +lidar/+labeler under the current folder, and copy the automation class into it.

Note: The LidarSemanticSegmentation.m file must be in the same folder where you create the +lidar/+labeler folder structure.

 mkdir('+lidar/+labeler');
 copyfile('LidarSemanticSegmentation.m','+lidar/+labeler');

Under Select Algorithm, select Refresh list. Then, select Algorithm > Lidar Semantic Segmentation. If you do not see this option, verify that the current working folder has a folder called +lidar/+labeler, with a file named LidarSemanticSegmentation.m in it.

Click Automate. The app opens an automation session and displays directions for using the algorithm.

Click Run. The created algorithm executes on each frame of the specified sequence and segments points into the Vegetation, Ground, Road, RoadMarkings, SideWalk, Car, Truck, OtherVehicle, Pedestrian, RoadBarriers, Signs, Buildings categories. After the app completes the automation run, use the slider or arrow keys to scroll through the sequence to locate any frames where the automation algorithm labeled points incorrectly. Use the zoom, pan, and 3-D rotation options to view and rotate the point cloud. Manually adjust the results by adding or deleting voxel annotations.

Supporting Functions

The helperDisplayLabelOverlaidPointCloud function overlays the segmentation result over a 3-D organized point cloud.

function helperDisplayLabelOverlaidPointCloud(I,predictedResult)
    % helperDisplayLabelOverlaidPointCloud Overlay labels over point cloud object.
    %  helperDisplayLabelOverlaidPointCloud(I,predictedResult)
    %  displays the overlaid pointCloud object. I is the 5 channels organized
    %  input image. predictedResult contains pixel labels.
    ptCloud = pointCloud(I(:,:,1:3),Intensity = I(:,:,4));
    cmap = helperPandasetColorMap;
    B = ...
        labeloverlay(uint8(ptCloud.Intensity),predictedResult,Colormap = cmap,Transparency = 0.4);
    pc = pointCloud(ptCloud.Location,Color = B);
    ax = pcshow(pc);
    set(ax,XLim = [-70 70],YLim = [-70 70])
    zoom(ax,3.5)
end

The helperPandasetColorMap function defines the colormap used by the lidar data set.

function cmap = helperPandasetColorMap
    cmap = [[30 30 30];  % Unlabeled
        [0 255 0];       % Vegetation
        [255 150 255];   % Ground
        [237 117 32];    % Road
        [255 0 0];       % Road Markings
        [90 30 150];     % Sidewalk
        [255 255 30];    % Car
        [245 150 100];   % Truck
        [150 60 30];     % Other Vehicle
        [255 255 0];     % Pedestrian
        [0 200 255];     % Road Barriers
        [170 100 150];   % Signs
        [255 0 255]];    % Building
    cmap = cmap./255;
end

The helperPointCloudToImage function converts the point cloud to a five-channel image.

function image = helperPointCloudToImage(ptcloud)
% helperPointCloudToImage converts the point cloud to five-channel image

image = ptcloud.Location;
image(:,:,4) = ptcloud.Intensity;
rangeData = iComputeRangeData(image(:,:,1),image(:,:,2),image(:,:,3));
image(:,:,5) = rangeData;
index = isnan(image);
image(index) = 0;
end

function rangeData = iComputeRangeData(xChannel,yChannel,zChannel)
rangeData = sqrt(xChannel.*xChannel+yChannel.*yChannel+zChannel.*zChannel);
end