Mapping RGB and depth (Kinect v2)

Question

alheka 2016-2-15

2
链接

此问题的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/268152-mapping-rgb-and-depth-kinect-v2

编辑： Walter Roberson 2019-6-11

Hi everyone, for my project, I'm working with the Kinect v2 and I have to map the depth information onto the RGB images to process them: in particular, I need to know which pixels in the RGB images are in a certain range of distance (depth) along the Z axis; I'm acquiring all the data with a C# program and saving them as images.

I had a look on the internet and I found some equations to map depth and RGB info here. I am actually able to get the intrinsic parameters for the single camera calibration using a checkerboard recorded from both the RGB camera (1920x1080) and the IR sensor (512x424) and processing them with MatLab Camera Calibrator app, but I'm stuck doing the stereo calibration (to find the rotation and translation matrices), because MatLab Stereo Camera Calibrator doesn't accept pair of images with different resolution.

I've also tried some other method with no results; It would be great if anyone could help me figuring out what I have to do in order to solve my problem. Thanks very much!

9 个评论
显示 7更早的评论隐藏 7更早的评论

Muhammad Hammad Malik 2018-11-6

编辑：Walter Roberson 2018-11-6

在 MATLAB Online 中打开

i tried to use that link but i am unable. kindly check it what things i need to do.see attach code. thanks

function [aligned] = ...
                    depth_rgb_registration('F:\kinectdata\cloud\tomato.ply', 'F:\kinectdata\color\tomato1.jpg',...
                    1.1752e+03, 1.2589e+03, 849.6059, 726.8106,...
                    1.0585e+03, 1.1341e+03, 965.1123, 583.2686,...
                    [1.0000   -0.0055    0.0003  -78.2239;
    0.0055    1.0000    0.0002   31.3206;
   -0.0003   -0.0002    1.0000   -8.0845])
fx_rgb_len =   3.28135;
fy_rgb_len =   3.51571;
fx_d_len =   3.64132;
fy_d_len =   3.90295;
 depthHeight=424;
  depthWidth=512;
    depthHeight = size('F:\kinectdata\cloud\tomato.ply', 1);
    depthWidth = size('F:\kinectdata\cloud\tomato.ply', 2);
    % Aligned will contain X, Y, Z, R, G, B values in its planes
    aligned = zeros(depthHeight, depthWidth, 6);
      for v = 1 : (depthHeight)
          for u = 1 : (depthWidth)
              % Apply depth intrinsics
              z = single('F:\kinectdata\cloud\tomato.ply'(v,u)) / depthScale;
              x = single((u - 849.6059) * z) / 1.1752e+03;
              y = single((v - 726.8106) * z) / 1.2589e+03;
              % Apply the extrinsics
              transformed = ([1.0000   -0.0055    0.0003  -78.2239;
      0.0055    1.0000    0.0002   31.3206;
     -0.0003   -0.0002    1.0000   -8.0845] * [x;y;z;1])';
              aligned(v,u,1) = transformed(1);
              aligned(v,u,2) = transformed(2);
              aligned(v,u,3) = transformed(3);
          end
      end
      for v = 1 : (depthHeight)
          for u = 1 : (depthWidth)
              % Apply RGB intrinsics
              x = (aligned(v,u,1) * 1.0585e+03 / aligned(v,u,3)) + 965.1123;
              y = (aligned(v,u,2) * 1.1341e+03/ aligned(v,u,3)) + cy_rgb;
              % "x" and "y" are indices into the RGB frame, but they may contain
              % invalid values (which correspond to the parts of the scene not visible
              % to the RGB camera.
              % Do we have a valid index?
              if (x > rgbWidth || y > rgbHeight ||...
                  x < 1 || y < 1 ||...
                  isnan(x) || isnan(y))
                  continue;
              end
              % Need some kind of interpolation. I just did it the lazy way
              x = round(x);
              y = round(y);
              aligned(v,u,4) = single('F:\kinectdata\color\tomato1.jpg'(y, x, 1);
              aligned(v,u,5) = single('F:\kinectdata\color\tomato1.jpg'(y, x, 2);
              aligned(v,u,6) = single('F:\kinectdata\color\tomato1.jpg'(y, x, 3);
          end
      end    
  end

Walter Roberson 2018-11-6

Following some links I see that someone has created a small toolbox at https://sourceforge.net/projects/kinectcalib/

Muhammad Hammad Malik 2018-11-15

thanks for this, i used this but it is always saying that device is not detected but when i run in some other code then device is working

请先登录，再进行评论。

请先登录，再回答此问题。

Answer 1

Shida 2016-2-16

4
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/268152-mapping-rgb-and-depth-kinect-v2#answer_209992

在 MATLAB Online 中打开

Hi alheka, I used to have the same problem. It's a pity that Matlab still hasn't solved this issue. But there's a solution to that. It took me quite a while to figure it out. Here you have the steps you need:

1. Stereo Calibration:

a) Find checkerboard points:

    [pointsRGB, boardSizeRGB, imagesRGBused] =detectCheckerboardPoints(imagesRGB);
    [pointsIR, boardSizeIR, imagesIRused] = detectCheckerboardPoints(imagesIR);

b) Remove the pairs where in one of the sets the points were not detected. Matlab does that automatically for the stereo calibration. But you need to write your code. I did make such code, but it's too unreadable to share here !

c) Now you need to manually combine the two sets.

imagePoints = cat(4, poitnsRGB, pointsIR);

d) Now the word points: where squaresize is the width of each of the squares in your calibration target (unit is mm).

worldPoints = generateCheckerboardPoints(boardSizeRGB, squaresize);

e) Calibration:

    [stereoParams,pairsUsed,estimationErrors] = estimateCameraParameters(imagePoints,...
    worldPoints,'EstimateSkew', false, 'EstimateTangentialDistortion', false, ...
    'NumRadialDistortionCoefficients', 2, 'WorldUnits', 'mm');
figure; showReprojectionErrors(stereoParams);
figure; showExtrinsics(stereoParams);

f) If needed, remove pairs of images with high error and re-calibrate until you're satisfied with the error.

2) Map the images: Actually the link you have is correct for this part. Just follow those instructions.

3) Remember that some artifacts (repeated pixels) can be caused in the mapped image due to parallax and the fact that some pixels are visible in one camera and not in the other. There are ways to fix that.

I hope it works for you.

Good luck.

8 个评论
显示 6更早的评论隐藏 6更早的评论

alheka 2016-2-16

在 MATLAB Online 中打开

Thank you very much for the help, Shida. I still have some problems:

1) in the figure that comes out from the stereo calibration, camera 1 (RGB camera) is represented on the left respect to camera 2 (depth sensor), but in the Kinect is the opposite

2) I still don't manage to map depth information onto RGB images.

Through the calibration I have calculated the rotation (R) and translation (T) matrix between the depth sensor and the RGB camera, as well as their intrinsic parameters.

I have created P3D_d (depth pixels in world coordinates related to depth sensor) and P3D_rgb (depth pixels in world coordinates related to rgb camera).

row_num = 424;
col_num = 512;
P3D_d = zeros(row_num,col_num,3);
P3D_rgb = zeros(row_num,col_num,3);
cont = 1;
for row=1:row_num
    for col=1:col_num
        P3D_d(row,col,1) = (row - cx_d) * depth(row,col) / fx_d;
        P3D_d(row,col,2) = (col - cy_d) * depth(row,col) / fy_d;
        P3D_d(row,col,3) = depth(row,col);
          temp=[P3D_d(row,col,1);P3D_d(row,col,2);P3D_d(row,col,3)];
          P3D_rgb(row,col,:) = R*temp+T;
      end
  end

I have created P2D_rgb_x and P2D_rgb_y.

P2D_rgb_x(:,:,1) = (P3D_rgb(:,:,1)./P3D_rgb(:,:,3))*fx_rgb+cx_rgb;
P2D_rgb_y(:,:,2) = (P3D_rgb(:,:,2)./P3D_rgb(:,:,3))*fy_rgb+cy_rgb;

but now I don't understand how to continue.

Assuming that the calibration parameters are correct, I've tried to click on a defined point in both the depth (coordinates: row_d, col_d) and rgb (coordinates: row_rgb, col_rgb) images, but the return of P2D_rgb_x(row_d, col_d) is totally different from row_rgb, as well as the return of P2D_rgb_y(row_d, col_d) is totally different from col_rgb.

So, what do exactly mean P2D_rgb_x and P2D_rgb_y? How can I use them to map depth value onto rgb images or just to get the depth of a certain RGB pixel?

I'll apreciate any suggest or help!

Shida 2016-2-22

编辑：Shida 2016-2-22

map_RGBD_test.m

Hi, About the first point, could you first confirm that you have already mirrored the Kinect images before doing the calibration. If so, then it could be the coordinate system of Matlab which causes this. If it's not either of them, then I need to check my own setup and see what I've done. That was a while ago!

About the mapping, I didn't have time to read all of your steps. But I attached my own (experimental) code here. I know that's not a nice code to look at as I wrote it back then in a hurry, but it might help you get an idea. I also don't guarantee if the code is correct. I won't be surprised if there are some stupid mistakes which I have overlooked. One should definitely be able to write it much simpler and more optimized.

Keep in mind that since depth image has a lower resolution, your results will be sparse (as some pixels won't have a value). One last thing, don't forget the parallax problem and don't worry if some pixels are repeated. One can write another code to fix that.

Good luck!

Dominik Melcher 2016-12-6

Hi Shida,

I am trying to map the Color Pixel to Cameracoordinates. I followed your step you gave above, but I stuck at step 2, where I have to use the functions of the link. For some reason the depth Pixel doesnt match to the calculated colorpixel and I dont know why. Could you send me you matlab code of this matching pls?

Thanks, Dominik

Franz-Josef Siegemund 2018-10-26

编辑：Franz-Josef Siegemund 2018-10-29

Hi there,

i know this is a really old entry, but i would like to know if anybody did map from rgb->depth? I did get the transformation from depth->rgb working. But i am having trouble understanding the whole extrinsic/instrinsic camera parameters and how to transform with each. (A book or some reference would be enough).

In particular, i want to transform the position of 4 points from rgb image space -> depth.

请先登录，再进行评论。

Answer 2

Muhammad Hammad Malik 2018-10-29

0
链接

此回答的直接链接

https://ww2.mathworks.cn/matlabcentral/answers/268152-mapping-rgb-and-depth-kinect-v2#answer_343915

hello all, i want to do depth-color image registration but do not know how to do it, kindly guide me for this. if anyone have working code or wahtever, kindly help me. thanks