Transfer Learning Using Voxel R-CNN for Lidar 3-D Object Detection
Since R2024b
This example shows how to detect 3-D objects in lidar data by using transfer learning with a voxel region-based convolutional neural network (Voxel R-CNN) [1]. In this example, you:
Configure a data set for training and testing of the Voxel R-CNN object detection network. You also perform data augmentation on the training data set to improve network efficiency.
Compute anchor boxes from the training data to train the Voxel R-CNN object detection network.
Create a pretrained Voxel R-CNN object detector as a
voxelRCNNObjectDetectorobject, and perform transfer learning using thetrainVoxelRCNNObjectDetectorfunction.
Download Data Set
Download and unzip a ZIP file containing a subset of sensor data from the PandaSet data set [2].The size of data set is 1 GB. The ZIP file contains these folders:
PointCloud— Contains 400 point clouds in PCD format, arranged so that the ego vehicle moves along the positive x-axis.Cuboid— Contains thePandasetLidarGroundTruth.matfile, which stores the ground truth data corresponding to each point cloud in table format. The ground truth data specifies 3-D bounding boxes for three object classes: car, truck, and pedestrian.
Note that the data set follows lidar sensor coordinate frames. For more information, see Coordinate Systems in Lidar Toolbox. According to the sensor coordinate system, the positive x-axis must point forward from the ego vehicle, and the positive y-axis must point to the left of the ego vehicle for each point cloud. If your data set deviates from this orientation, you can adjust it by using the pctransform function for point clouds and the bboxwarp function for bounding boxes.
zipFile = matlab.internal.examples.downloadSupportFile("lidar","data/PandasetLidarData.zip"); saveFolder = fileparts(zipFile); unzip(zipFile,saveFolder) outputFolder = fullfile(saveFolder,"PandasetLidarData");
Load Data
Create a datastore by using fileDatastore object that loads the PCD files from the specified path using the pcread function.
path = fullfile(outputFolder,"PointCloud"); lds = fileDatastore(path,"ReadFcn",@(x) pcread(x));
Load ground truth data.
gtPath = fullfile(outputFolder,"Cuboid","PandaSetLidarGroundTruth.mat"); bboxLabels = load(gtPath,"lidarGtLabels"); bboxTable = bboxLabels.lidarGtLabels;
Define a point cloud range that sets the boundaries of the region of interest within the point cloud data for detecting objects. The range specifies the minimum and maximum values along the x-, y-, and z-axes of the point cloud data. The Voxel R-CNN network focuses on the specified volume and ignores data points outside the range. This can help you optimize detection, performance by reducing the computational load and focusing on areas of interest. For more information on using point cloud ranges, see voxelRCNNObjectDetector.
pointCloudRange = [0 70.4 -40 40 -3 1];
Extract the bounding boxes and their associated labels that fall within the defined pointCloudRange by using helperProcessGroundTruthData helper function, defined at the end of this example. This step is crucial for evaluating the performance of the trained detector, as it helps assess how effectively the network identifies objects within the targeted region of interest it has been trained on.
bboxTable = helperProcessGroundTruthData(bboxTable,pointCloudRange);
To load 3-D bounding box information for the car, truck, and pedestrian object classes, create a box label datastore by using a boxLabelDatastore object.
bds = boxLabelDatastore(bboxTable);
Combine the point cloud data and its ground truth information into a single datastore by using the combine function.
cds = combine(lds,bds);
Create training and test sets from the data set. This example uses 80% of the data for training, and 20% data for testing.
rng(8) totalElements = numpartitions(cds); % Create a random permutation of indices. shuffledIndices = randperm(totalElements); % Specify split ratio, and calculate the number of training elements. trainRatio = 0.8; numTrainElements = round(totalElements*trainRatio); % Determine indices for training and testing sets. trainIndices = shuffledIndices(1:numTrainElements); testIndices = shuffledIndices(numTrainElements+1:end); % Create subsets of the datastore based on the calculated indices. trainDs = subset(cds,trainIndices); testDs = subset(cds,testIndices);
Read and display a sample point cloud from the training data set, before augmentation, by using the helperShowPointCloudWith3DBoxes helper function, defined at the end of this example.
trainingSample = preview(trainDs);
[ptCld,bboxes,labels] = deal(trainingSample{1},trainingSample{2},trainingSample{3});
% Define the classes for object detection.
classNames = {'Car','Truck','Pedestrian'};
% Define colors for each class to plot bounding boxes.
colors = {'green','magenta','yellow'};
helperShowPointCloudWith3DBoxes(ptCld,bboxes,labels,classNames,colors)
Perform Data Augmentation
Data augmentation improves network accuracy by adding more variety to the training data without actually increasing the number of labeled training samples. To perform data augmentation, you randomly transform a subset of your original training samples to generate new training samples. For more information on data augmentation, see the Data Augmentations for Lidar Object Detection Using Deep Learning.
Use the sampleLidarData function to sample 3-D bounding boxes and their corresponding points from the training data. By default, this example writes sampled data to the folder specified by the sampleLocation variable. If your sampled data is already present in the specified folder, set the writeFiles variable to false.
sampleLocation = fullfile(outputFolder,"GTsamples"); writeFiles = true; if writeFiles [ldsSampled,bdsSampled] = sampleLidarData(trainDs,classNames,MinPoints=[20 20 10], ... Verbose=false,WriteLocation=sampleLocation); cdsSampled = combine(ldsSampled,bdsSampled); save(fullfile(sampleLocation,"augmentedSample"),"cdsSampled") else load(fullfile(sampleLocation,"augmentedSample")) end
Use the pcBboxOversample function to randomly add a fixed number of car, truck, and pedestrian class objects to every point cloud. Use the transform function to apply the ground truth and custom data augmentations to the training data.
numObjects = 10; cdsAugmented = transform(trainDs,@(x)pcBboxOversample(x,cdsSampled,classNames,numObjects));
This example applies these additional data augmentation techniques to each sampled point cloud by using the helperAugmentData helper function, defined at the end of this example:
Random scaling by 5%.
Random rotation along the z-axis in the range [—45, 45] degrees
Random translation by [0.2, 0.2, 0.1] meters along the x-, y-, and z-axes, respectively
cdsAugmented = transform(cdsAugmented,@(x)helperAugmentData(x));
Display an augmented point cloud and its bounding boxes using the helperShowPointCloudWith3DBoxes helper function.
augData = preview(cdsAugmented);
[ptCld,bboxes,labels] = deal(augData{1},augData{2},augData{3});
helperShowPointCloudWith3DBoxes(ptCld,bboxes,labels,classNames,colors)
Create Voxel R-CNN Object Detector
Estimate the anchor boxes from training data by using the helperEstimate3DAnchorBoxesForVoxelRCNN helper function, attached to this example as a supporting file.
anchorBoxes = helperEstimate3DAnchorBoxesForVoxelRCNN(trainDs,classNames);
Create a pretrained voxelRCNNObjectDetector object configured to perform transfer learning. For more information on the Voxel R-CNN network, see Get Started with Voxel R-CNN.
detector = voxelRCNNObjectDetector("kitti",classNames,anchorBoxes,PointCloudRange=pointCloudRange); Specify Training Options
Use the Adam optimization algorithm to train the network. Use the trainingOptions (Deep Learning Toolbox) function to specify the hyperparameters.
options = trainingOptions("adam", ... InitialLearnRate=0.01, ... MiniBatchSize=4, ... MaxEpochs=10, ... PreprocessingEnvironment="parallel", ... VerboseFrequency=100, ... CheckpointFrequency=10, ... CheckpointPath=userpath, ... Plots="training-progress", ... Shuffle="every-epoch", ... L2Regularization=0.01);
Note: Reduce or increase the miniBatchSize value to control memory usage when training.
Train Voxel R-CNN Object Detector
Use the trainVoxelRCNNObjectDetector function to train the Voxel R-CNN object detector.
To train the network, you must have Parallel Computing Toolbox™ and a CUDA® enabled NVIDIA® GPU. For more information, see GPU Computing Requirements (Parallel Computing Toolbox).
[detector,info] = trainVoxelRCNNObjectDetector(cdsAugmented,detector,options);
Detect Objects in Point Cloud
Read a point cloud from the test sample, and run the trained detector to get detections, scores, and labels.
testData = preview(testDs);
ptCloud = testData{1};
[bboxes,score,labels] = detect(detector,ptCloud);Overlay the object detections on the point cloud, and display the overlaid detections.
helperShowPointCloudWith3DBoxes(ptCloud,bboxes,labels,classNames,colors)
Evaluate Network
Run the detector on test data.
testPtClouds = testDs.UnderlyingDatastores{1,1};
detectionResults = detect(detector,testPtClouds,PredictedBoxType="rotated-rectangle");Running VoxelRCNN network. -------------------------- * Processed 80 point clouds.
Generate rotated rectangles of format Mx5 from the ground truth cuboid labels of format M-by-9 by using the helperGetRotatedRectangle helper function, defined at the end of this example.
testGrdTruth = testDs.UnderlyingDatastores{1,2};
groundTruthData = transform(testGrdTruth,@(x)helperGetRotatedRectangle(x));Evaluate the performance of the object detector by using the evaluateObjectDetection function.
metrics = evaluateObjectDetection(detectionResults,groundTruthData,"AdditionalMetrics","AOS"); metrics.ClassMetrics
ans=3×8 table
NumObjects APOverlapAvg AP Precision Recall AOSOverlapAvg AOS OrientationSimilarity
__________ ____________ __________ ________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ _____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________ _____________ __________ _________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
The precision-recall (PR) curve highlights how precise a detector is at varying levels of recall. The ideal precision is 1 at all recall levels.
Choose a class, extract the precision and recall metrics for that class, and then plot the precision and recall curves.
class =classNames(1); precision = metrics.ClassMetrics{class,"Precision"}; recall = metrics.ClassMetrics{class,"Recall"}; figure plot(recall{:}',precision{:}') xlabel("Recall") ylabel("Precision") grid on title(class + " Precision/Recall")
Supporting Functions
The helperShowPointCloudWith3DBoxes function displays the point cloud alongside its associated bounding boxes, employing distinct colors for different classes to facilitate easy differentiation.
function helperShowPointCloudWith3DBoxes(ptCld,bboxes,labels,classNames,colors) % Validate the length of classNames and colors are the same. assert(numel(classNames)==numel(colors),"ClassNames and Colors must have the same number of elements.") % Get unique categories from labels. uniqueCategories = categories(labels); % Create a mapping from category to color. colorMap = containers.Map(uniqueCategories,colors); labelColor = cell(size(labels)); % Populate labelColor based on the mapping. for i = 1:length(labels) labelColor{i} = colorMap(char(labels(i))); end figure ax = pcshow(ptCld); showShape("cuboid",bboxes,Parent=ax,Opacity=0.1, ... Color=labelColor,LineWidth=0.5) zoom(ax,5) end
The helperAugmentData function applies these augmentations to the data:
Random scaling by 5%.
Random rotation along the z-axis from [—45, 45] degrees
Random translation by [0.2, 0.2, 0.1] meters along the x-, y-, and z-axes respectively.
function data = helperAugmentData(data) pc = data{1}; % Define outputView based on the grid-size and XYZ limits. outView = imref3d([32,32,32],[-100,100], ... [-100,100],[-100,100]); rotationAngle = [-45, 45]; rotationNoise = (rotationAngle(2) - rotationAngle(1)).*rand() + rotationAngle(1); tform = randomAffine3d(Rotation=@() deal([0,0,1],rotationNoise), ... Scale=[0.95 1.05], ... XTranslation=[0 0.2], ... YTranslation=[0 0.2], ... ZTranslation=[0 0.1]); ptCloud = pctransform(pc,tform); % Apply the same transformation to the boxes. bbox = data{2}; [bbox,indices] = bboxwarp(bbox,tform,outView); if ~isempty(indices) data{1} = ptCloud; data{2} = bbox; data{3} = data{1,3}(indices,:); end end
The helperProcessGroundTruthData function extracts bounding boxes for each label within the specified point cloud range.
function boxTable = helperProcessGroundTruthData(boxTable,pcRange) for colIdx = 1:size(boxTable,2) % Apply the operation to each cell in the current column. boxTable{:,colIdx} = cellfun(@(x) helperGetBbox(x,pcRange),boxTable{:,colIdx},UniformOutput=false); end end
The helperGetBbox function extracts bounding boxes within the specified point cloud range.
function data = helperGetBbox(data,pcRange) bbox = data; if ~isempty(bbox) % Get index of bounding boxes that lie within pcRange. idx = (bbox(:,1) > pcRange(1)) & ... (bbox(:,1) < pcRange(2)) & ... (bbox(:,2) > pcRange(3)) & ... (bbox(:,2) < pcRange(4)); data = bbox(idx,:); end end
The helperGetRotatedRectangle function converts the bounding box format from M-by-9 (cuboid) to M-by-5 (rotated rectangle).
function data = helperGetRotatedRectangle(data) bbox = data{1}; if ~isempty(bbox) % Convert bounding box format to rotated rectangle. data{1} = bbox(:,[1 2 4 5 9]); end end
References
[1] Deng, Jiajun, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. “Voxel R-CNN: Towards High Performance Voxel-Based 3D Object Detection.” Proceedings of the AAAI Conference on Artificial Intelligence 35, no. 2 (May 18, 2021): 1201–9. https://doi.org/10.1609/aaai.v35i2.16207.
[2] Hesai and Scale. PandaSet. https://scale.com/open-datasets/pandaset.
See Also
voxelRCNNObjectDetector | trainVoxelRCNNObjectDetector | detect
Related Topics
You can also select a web site from the following list:
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)