本页采用了机器翻译。点击此处可查看英文原文。

基于立体视频的深度估计

打开实时脚本

本示例演示了如何通过使用已标定的立体相机拍摄的视频，检测视频中的人及其与相机的距离。

加载立体相机的参数

加载 stereoParameters 对象，该对象是通过使用 stereoCameraCalibrator 应用程序或 estimateCameraParameters 函数对相机进行标定后生成的结果。

% Load the stereoParameters object.
load("handshakeStereoParams.mat");

% Visualize camera extrinsics.
showExtrinsics(stereoParams);

Figure contains an axes object. The axes object with title Extrinsic Parameters Visualization, xlabel X (mm), ylabel Z (mm) contains 32 objects of type patch, text, line.

创建视频文件读取器和视频播放器

创建用于读取和显示视频的 System object。

videoFileLeft = "handshake_left.avi";
videoFileRight = "handshake_right.avi";

readerLeft = VideoReader(videoFileLeft);
readerRight = VideoReader(videoFileRight);
player = vision.VideoPlayer(Position=[20,200,740 560]);

读取并校正视频帧

为了计算视差并重建三维场景，必须对左右两台相机的图像进行校正。校正后的图像具有水平极线，且行对齐。这通过将匹配点的搜索空间简化为一维，从而简化了视差的计算。校正后的图像还可以组合成一幅红蓝立体图，佩戴红蓝立体眼镜观看时，即可看到三维效果。

frameLeft = readFrame(readerLeft);
frameRight = readFrame(readerRight);

[frameLeftRect, frameRightRect, reprojectionMatrix] = ...
    rectifyStereoImages(frameLeft, frameRight, stereoParams);

figure;
imshow(stereoAnaglyph(frameLeftRect, frameRightRect));
title("Rectified Video Frames");

Figure contains an axes object. The hidden axes object with title Rectified Video Frames contains an object of type image.

计算视差

在校正后的立体图像中，任意一对对应点都位于同一行像素上。对于左图中的每个像素，计算其到右图中对应像素的距离。该距离称为视差，它与对应世界点到相机的距离成正比。

frameLeftGray  = im2gray(frameLeftRect);
frameRightGray = im2gray(frameRightRect);
    
disparityMap = disparitySGM(frameLeftGray, frameRightGray);
figure;
imshow(disparityMap, [0, 64]);
title("Disparity Map");
colormap jet
colorbar

Figure contains an axes object. The hidden axes object with title Disparity Map contains an object of type image.

重建三维场景

根据视差图，重建与每个像素对应的点的三维世界坐标。

points3D = reconstructScene(disparityMap, reprojectionMatrix);

% Convert to meters and create a pointCloud object
points3D = points3D ./ 1000;
ptCloud = pointCloud(points3D, Color=frameLeftRect);

% Create a streaming point cloud viewer
player3D = pcplayer([-3, 3], [-3, 3], [0, 8], VerticalAxis="y", ...
    VerticalAxisDir="down");

% Visualize the point cloud
view(player3D, ptCloud);

Figure Point Cloud Player contains an axes object. The axes object with xlabel X, ylabel Y contains an object of type scatter.

检测左侧图像中的人

加载一个预训练的人体检测器，并检测左侧相机画面中的人。

peopleDetector = peopleDetectorACF();
bboxes = detect(peopleDetector,frameLeftGray);

确定每个人到相机的距离

求出每个被检测到的人的质心在三维空间中的坐标，并计算质心到相机的距离（单位：米）。

% Find the centroids of detected people.
centroids = [round(bboxes(:, 1) + bboxes(:, 3) / 2), ...
    round(bboxes(:, 2) + bboxes(:, 4) / 2)];

% Find the 3-D world coordinates of the centroids.
centroidsIdx = sub2ind(size(disparityMap), centroids(:, 2), centroids(:, 1));
X = points3D(:, :, 1);
Y = points3D(:, :, 2);
Z = points3D(:, :, 3);
centroids3D = [X(centroidsIdx)'; Y(centroidsIdx)'; Z(centroidsIdx)'];

% Find the distances from the camera in meters.
dists = sqrt(sum(centroids3D .^ 2));
    
% Display the detected people and their distances.
labels = dists+" meters";
figure
imshow(insertObjectAnnotation(frameLeftRect, "rectangle", bboxes, labels));
title("Detected People");

Figure contains an axes object. The hidden axes object with title Detected People contains an object of type image.

处理视频的其余部分

应用上述步骤，在视频的每一帧中检测人员并测量其与相机的距离。

while hasFrame(readerLeft) && hasFrame(readerRight)
    % Read the frames.
    frameLeft = readFrame(readerLeft);
    frameRight = readFrame(readerRight);
    
    % Rectify the frames.
    [frameLeftRect, frameRightRect] = ...
        rectifyStereoImages(frameLeft, frameRight, stereoParams);
    
    % Convert to grayscale.
    frameLeftGray  = im2gray(frameLeftRect);
    frameRightGray = im2gray(frameRightRect);
    
    % Compute disparity. 
    disparityMap = disparitySGM(frameLeftGray, frameRightGray);
    
    % Reconstruct 3-D scene.
    points3D = reconstructScene(disparityMap, reprojectionMatrix);
    points3D = points3D ./ 1000;
    ptCloud = pointCloud(points3D, Color=frameLeftRect);
    view(player3D, ptCloud);
    
    % Detect people.
    bboxes = detect(peopleDetector,frameLeftGray);
    
    if ~isempty(bboxes)
        % Find the centroids of detected people.
        centroids = [round(bboxes(:, 1) + bboxes(:, 3) / 2), ...
            round(bboxes(:, 2) + bboxes(:, 4) / 2)];
        
        % Find the 3-D world coordinates of the centroids.
        centroidsIdx = sub2ind(size(disparityMap), centroids(:, 2), centroids(:, 1));
        X = points3D(:, :, 1);
        Y = points3D(:, :, 2);
        Z = points3D(:, :, 3);
        centroids3D = [X(centroidsIdx), Y(centroidsIdx), Z(centroidsIdx)];
        
        % Find the distances from the camera in meters.
        dists = sqrt(sum(centroids3D .^ 2, 2));
        
        % Display the detect people and their distances.
        labels = dists+" meters";
        dispFrame = insertObjectAnnotation(frameLeftRect, "rectangle", bboxes,...
            labels);
    else
        dispFrame = frameLeftRect;
    end
    
    % Display the frame.
    step(player, dispFrame);
end

Figure Point Cloud Player contains an axes object. The axes object with xlabel X, ylabel Y contains an object of type scatter.

% Clean up
release(player);

Figure Movie Player contains an axes object and other objects of type uiflowcontainer, uimenu, uitoolbar. The hidden axes object contains an object of type image.

总结

本示例演示了如何使用经过标定的立体相机对行人进行三维定位。

参考资料

[1] G. Bradski and A. Kaehler, "Learning OpenCV :Computer Vision with the OpenCV Library," O'Reilly, Sebastopol, CA, 2008.

[2] Dalal, N. and Triggs, B., Histograms of Oriented Gradients for Human Detection.CVPR 2005.