Note that the geometric transformation convention used in the Computer Vision Toolbox (CVT) is different from the one used in the Robotics System Toolbox. If you transform a 3-D point [X, Y, Z] from one coordinate system to another, its new location is computed as
[X2, Y2, Z2] = [X, Y, Z] * R_cvt + t_cvt
where R_cvt is the rotation matrix and t_cvt is the translation vector. This is diffrent from the common convention you may have seen:
[X2, Y2, Z2]^T = R * [X, Y, Z]^T + t
where T represents transpose.
In other words, CVT uses the post-multiply convention and the rotation matrix and the translation vector are the transposed version of the common format.
Back to the question you were asking, following the common convention, the relative pose is computed as:
R_rel = R1^T * R2
t_rel = R1^T * (t2 - t1)
Now with the new convention,
R_rel_cvt = R_rel^T = R2^T * R1 = R2_cvt * R1_cvt^T
t_rel_cvt = (t2-t1)^T * R1 = (t2_cvt - t1_cvt) * R1_cvt^T
this explains the code in the helperAddNewKeyFrame function.
I know this is a little confusing, but we're actively working on adopting the common geometric transformation convention in the Computer Vision Toolbox.
