bboxLidarToCamera

R2026b

Estimate 2-D bounding boxes in image from 3-D bounding boxes in point cloud

collapse all in page

Syntax

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform)
bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform,L)
[bboxesCamera,boxesUsed] = bboxLidarToCamera(___)
[___] = bboxLidarToCamera(___,'ProjectedCuboid',true)

Description

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform) estimates 2-D bounding boxes in an undistorted image from 3-D bounding boxes in the lidar frame bboxesLidar. The function uses the camera intrinsic parameters intrinsics and a lidar to camera transformation matrix tform.

bboxesCamera = bboxLidarToCamera(bboxesLidar,intrinsics,tform,L) further refines the 2-D bounding boxes to the edges of the object inside it using L. L is the corresponding labeled 2-D image of the 2-D bounding boxes, where the objects are labeled distinctively.

[bboxesCamera,boxesUsed] = bboxLidarToCamera(___) indicates for which of the specified 3-D bounding boxes the function detects a corresponding 2-D bounding box in the undistorted image.

[___] = bboxLidarToCamera(___,'ProjectedCuboid',true) returns 3-D projected cuboids instead of 2-D bounding boxes.

example

Examples

collapse all

Open Live Script

Load data from a MAT file into the workspace. 

ld = load("bboxData.mat");

Extract the image, point cloud, and camera intrinsics.

I = ld.I;
ptCloud = ld.ptCloud;
intrinsics = ld.intrinsics;

Undistort the image using the camera intrinsics.

[I,newIntrinsics] = undistortImage(I,intrinsics);

Extract one of the 3-D bounding boxes to project from the point cloud to the image.

bboxLidar = ld.bboxPtCloud(1,:);

Display the 3-D bounding box overlaid on the point cloud.

figure
pcshow(ptCloud)
showShape("cuboid",bboxLidar,LineWidth=1,Color="white")

Figure contains an axes object. The axes object contains an object of type scatter.

Estimate the projection of the 3-D bounding box in the image using the transformation from the lidar sensor to the camera.

lidarToCameraTform = ld.lidarToCameraTform;
bboxesImage = bboxLidarToCamera(bboxLidar,newIntrinsics,lidarToCameraTform);

Display the 2-D bounding box annotated in the image.

Iannotated = insertObjectAnnotation(I,"rectangle",bboxesImage,"Vehicle","LineWidth",2);
figure
imshow(Iannotated)

Figure contains an axes object. The hidden axes object contains an object of type image.

Open Live Script

Load data from a MAT file into the workspace. 

ld = load("bboxData.mat");

Extract the image, point cloud, and camera intrinsics.

I = ld.I;
ptCloud = ld.ptCloud;
intrinsics = ld.intrinsics;

Undistort the image using the camera intrinsics.

[I,newIntrinsics] = undistortImage(I,intrinsics);

Extract one of the 3-D bounding boxes to project from the point cloud to the image.

bboxPtCloud = ld.bboxPtCloud(2,:);

Display the 3-D bounding box overlaid on the point cloud.

figure
pcshow(ptCloud)
showShape("cuboid",bboxPtCloud,Color="white")

Figure contains an axes object. The axes object contains an object of type scatter.

Use the fuseCameraToLidar function with the transformation from the camera to the lidar sensor to get a point cloud of the points within the field of view of the camera.

cameraToLidarTform = ld.cameraToLidarTform;
cameraFOVPtCloud = fuseCameraToLidar(I,ptCloud,newIntrinsics,cameraToLidarTform);

Use the findPointsInModel function of the cuboidModel object to verify that the object enclosed by the point cloud bounding box is in the image.

model = cuboidModel(bboxPtCloud);
indices = findPointsInModel(model,cameraFOVPtCloud);
isObjectInImage = ~isempty(indices)
isObjectInImage = logical
   1

Visualize the point cloud of the field of view of the camera with the color from the image and the 3-D bounding box.

figure
pcshow(cameraFOVPtCloud,MarkerSize=50)
showShape("cuboid",bboxPtCloud,Color="white")

Figure contains an axes object. The axes object contains an object of type scatter.

Estimate the projection of the 3-D bounding box in the image using the transformation from the lidar sensor to the camera.

lidarToCameraTform = ld.lidarToCameraTform;
bboxesImage = bboxLidarToCamera(bboxPtCloud,newIntrinsics,lidarToCameraTform,ProjectedCuboid=true);

Annotate the image to display the projected cuboid overlaid on the image.

Iannotated = insertObjectAnnotation(I,"projected-cuboid",bboxesImage,"Vehicle","LineWidth",2);
figure
imshow(Iannotated)

Figure contains an axes object. The hidden axes object contains an object of type image.

Input Arguments

collapse all

3-D bounding boxes in the lidar frame, specified as a cuboidModel object or an N-by-9 matrix of real values. N is the number of 3-D bounding boxes. Each row of the matrix has the form [xctr yctr zctr xlen ylen zlen xrot yrot zrot].

  • xctr, yctr, and zctr — These values specify the x-, y-, and z-axis coordinates, respectively, of the center of the cuboid bounding box.

  • xlen, ylen, and zlen — These values specify the length of the cuboid along the x-, y-, and z-axis, respectively, before it is rotated.

  • xrot, yrot, and zrot — These values specify the rotation angles of the cuboid around the x-, y-, and z-axis, respectively. These angles are clockwise-positive when you look in the forward direction of their corresponding axes.

This figure shows how these values determine the position of a cuboid.

Plot of bounding box showing rotations and lengths.

Note

The function assumes that the point cloud data that corresponds to the 3-D bounding boxes and the image data are time synchronized.

Data Types: single | double

Camera intrinsic parameters corresponding to the undistorted image, specified as a cameraIntrinsics (Computer Vision Toolbox) object. You can obtain the camera intrinsics using the undistortImage (Computer Vision Toolbox) function to undistort the image.

Camera to lidar rigid transformation, specified as a rigidtform3d (Image Processing Toolbox) object.

Labeled 2-D image, specified as a matrix of real values. The matrix size is the same as the ImageSize property of intrinsics.

Note

Labeled images are assumed to be undistorted.

Data Types: single | double | int8 | int16 | uint8 | uint16

Output Arguments

collapse all

2-D bounding boxes in the undistorted image, returned as an M-by-4 matrix of real values. M is the number of detected bounding boxes. Each row of the matrix contains the location and size of a rectangular bounding box in the form [x y width height]. The x and y elements specify the x and y coordinates, respectively, for the upper-left corner of the rectangle. The width and height elements specify the size of the rectangle.

If 'ProjectedCuboid' is set to true, the 2-D bounding boxes are returned as an M-by-8 matrix of real values. The bounding boxes have a cuboid shape and enclose the object. Each row of the matrix contains the size and location of the cuboid bounding box in the form [frontFace backFace]. Both the faces are represented as 2-D bounding boxes.

Data Types: single | double

Bounding box detection flag, returned as an N-element row vector of logicals. 2 is the number of input 3-D bounding boxes. If the function detects a corresponding 2-D bounding box in the undistorted image, then it returns a value of true for that input 3-D bounding box. If the function does not detect a corresponding 2-D bounding box, then it returns a value of false.

Data Types: logical

Version History

Introduced in R2021a

expand all

See Also

Functions