Stereo Calibration

The MultiSense stereo camera adheres to the OpenCV stereo camera calibration format The MultiSense stereo calibration is broken out into two primary files: intrinsics.yml and extrinsics.yml. Camera calibration matrices have associated indices corresponding to the specific camera they were generated from. The left camera has an index of 1, the right camera has an index of 2, and the aux camera has an index of 3.

Note

Calibrations are stored for full-resolution images. When operating in 1/4 resolution, the MultiSense internally scales the calibration’s fx, fy, cx, and cy values when rectifying images. Calibration values will need to be scaled in user applications based on the operating resolution of the camera.

The MultiSense’s calibration can be queried with the LibMultiSense ImageCalUtility.

Intrinsics

The intrinsics.yml file contains calibration metadata for the raw camera images. These include M, and D matrices.

M matrix

A 3x3 pinhole projection matrix for the raw camera image. This will project points in the raw camera’s coordinate frame into the raw camera image. In OpenCV this is typically named K. This matches the form of cameraMatrix from the output of OpenCV’s calibrateCamera function.

\[\begin{split}M = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}\end{split}\]

D matrix

A 5 element or 8 element distortion parameters associated with the raw camera image. The MultiSense support the OpenCV ordering of the 5 parameter pinhole distortion model (k1, k2, p1, p2, k3), or the 8 parameter rational polynomial distortion model (k1, k2, p1, p2, k3, k4, k5, k6). This matches the distCoeffs from the output of OpenCV’s calibrateCamera function.

Extrinsics

The extrinsics.yml file contains stereo rectification matrices associated with the rectified camera images. These include the P and R matrices.

P matrix

A 3x4 projection matrix in the rectified coordinate systems for the first camera. Each P matrix projects points given in the left camera’s rectified coordinate system into the corresponding rectified image. P1 and P2 match the output from OpenCV’s stereoRectify function run on the calibrations from the left and right cameras.

\[\begin{split}P = \begin{bmatrix} f_x & 0 & c_x & f_xTx \\ 0 & f_y & c_y & f_yTy \\ 0 & 0 & 1 & Tz \end{bmatrix}\end{split}\]

Note

\(T_y\) and \(T_z\) will only be non-zero for the Aux camera. This is due to a limitation with the OpenCV rectification model with three cameras.

R matrix

A 3x3 rectification rotation matrix for each of the rectified cameras. This matrix brings points given in the unrectified camera’s coordinate system to points in the rectified camera’s coordinate system. R1 and R2 match the output from OpenCV’s stereoRectify function run on the calibrations from the left and right cameras.