Image Sources

The MultiSense publishes various image sources which are captured or derived from a single exposure. A raw left, right and potentially aux image is captured with each exposure. All captured and derived sources from a single exposure will contain the same frame_id and timestamp transmitted with each header.

Left Luma

A 8 bit grayscale image from the MultiSense’s left camera. This image has no distortion compensation applied to it, and is the raw input to the main MultiSense stereo pipeline

Right Luma

A 8 bit grayscale image from the MultiSense’s right camera. This image has no distortion compensation applied to it, and is the raw input to the main MultiSense stereo pipeline

Left Luma Rectified

A 8 bit grayscale image from the MultiSense’s left camera rectified using the MultiSense calibration. This is the output from the MultiSense rectification process, and is a direct input for the MultiSense disparity matching algorithm.

Right Luma Rectified

A 8 bit grayscale image from the MultiSense’s right camera rectified using the MultiSense calibration. This is the output from the MultiSense rectification process, and is a direct input for the MultiSense disparity matching algorithm.

Disparity

In a MultiSense stereo camera setup, the disparity source refers to the disparity values computed from the left rectified camera’s perspective. For each pixel in the left rectified image, the disparity source provides the difference in the horizontal position between that pixel and the pixels along the corresponding epipolar line in the right rectified image.

Values of 0 in the disparity image indicate no valid matches were found between the left rectified pixel and the pixels along the corresponding epipolar line in the right disparity image.

The left disparity image is quantized to 16 bits where each pixel in the disparity image represents 1/16th of a disparity. When converting disparity images to point clouds or depth images, each quantized disparity pixel must be converted to a floating point value and divided by 16 to get the pixel disparity measurement.

When disparity images are transmitted over-the-wire from the MultiSense to the host machine, they are packed into 12 bits to minimize network bandwidth.

Cost

The corresponding cost for each pixel in the disparity image. Higher cost values represent less confidence in the disparity value generated for that pixel location.

Aux Luma

A 8 bit grayscale image from the MultiSense’s aux camera. This image has no distortion compensation applied to it, and is the luminance component (Y) of the planar YCbCr420 encoding for aux color images.

See the create color images overview for how to convert aux luma and aux chroma images into RGB color images.

Note

Only MultiSense S27 and MultiSense S30 cameras include an aux camera

Aux Chroma

A 16 bit image from the MultiSense’s aux camera encoding the Cb and Cr components of the aux cameras planar YCbCr420 encoding. The 8 bit Cb and 8 bit Cr components are packed into each 16 bit pixel. The chroma image is 1/4 resolution of the corresponding luma image.

The CbCr packing follows the following scheme, and uses little-endian byte ordering for each component.

Cb	Cr
8 bits	8 bits

This image has no distortion compensation applied to it.

See the create color images overview for how to convert aux luma and aux chroma images from the same MultiSense frame into RGB color images.

Note

Only MultiSense S27 and MultiSense S30 cameras include an aux camera

Aux Luma Rectified

A 8 bit grayscale image from the MultiSense’s aux camera rectified using the MultiSense calibration. This is the luminance component (Y) of the planar YCbCr420 encoding of aux color images.

See the create color images overview for how to convert aux luma and aux chroma images into RGB color images. Aux luma rectified and aux chroma rectified images can be used as inputs to the colorspace conversion scheme to create RGB aux rectified images.

Note

Only MultiSense S27 and MultiSense S30 cameras include an aux camera

Aux Chroma Rectified

A 16 bit image from the MultiSense’s aux camera encoding the Cb and Cr components of the aux cameras planar YCbCr420 encoding. This image is rectified using the MultiSense calibration. The 8 bit Cb and 8 bit Cr components are packed into each 16 bit pixel. The rectified chroma image is 1/4 resolution of the corresponding rectified luma image.

The CbCr packing follows the following scheme, and uses little-endian byte ordering for each component.

Cb	Cr
8 bits	8 bits

See the create color images overview for how to convert rectified aux luma and rectified aux chroma images from the same MultiSense frame into RGB color images.

Note

Only MultiSense S27 and MultiSense S30 cameras include an aux camera

Left Compressed

A h.264 compressed variant of the Left Luma data source. Images are encoded as h.264 I-Frames, making decoding independent allowing for lower frame rates. The onboard h.264 compression hardware uses a zeroed chroma color plane to transmit the luma image. When decoding these image, only the output luma channel should be used.

See the Decoding Compressed Image Streams section for details on how to decode this stream to an image.

Right Compressed

A h.264 compressed variant of the Right Luma data source. Images are encoded as h.264 I-Frames, making decoding independent allowing for lower frame rates. The onboard h.264 compression hardware uses a zeroed chroma color plane to transmit the luma image. When decoding these image, only the output luma channel should be used.

See the Decoding Compressed Image Streams section for details on how to decode this stream to an image.

Left Rectified Compressed

A h.264 compressed variant of the Left Luma Rectified data source. Images are encoded as h.264 I-Frames, making decoding independent allowing for lower frame rates. The onboard h.264 compression hardware uses a zeroed chroma color plane to transmit the luma image. When decoding these image, only the output luma channel should be used.

See the Decoding Compressed Image Streams section for details on how to decode this stream to an image.

Right Rectified Compressed

A h.264 compressed variant of the Right Luma Rectified data source. Images are encoded as h.264 I-Frames, making decoding independent allowing for lower frame rates. The onboard h.264 compression hardware uses a zeroed chroma color plane to transmit the luma image. When decoding these image, only the output luma channel should be used.

See the Decoding Compressed Image Streams section for details on how to decode this stream to an image.

Aux Compressed

A h.264 compressed variant of the combination of the Aux Luma and Aux Chroma data sources. Images are encoded as h.264 I-Frames, making decoding independent allowing for lower frame rates. The onboard h.264 compression hardware uses both the luma and chroma frames as inputs. When decoding these image, both the luma and chroma output channels should be used.

See the Decoding Compressed Image Streams section for details on how to decode this stream to an image.

Aux Rectified Compressed

A h.264 compressed variant of the combination of the Aux Luma Rectified and Aux Chroma Rectified data sources. Images are encoded as h.264 I-Frames, making decoding independent allowing for lower frame rates. The onboard h.264 compression hardware uses both the luma and chroma frames as inputs. When decoding these image, both the luma and chroma output channels should be used.

See the Decoding Compressed Image Streams section for details on how to decode this stream to an image.

IMU Sources

Accelerometer

Gyroscope

Miscellaneous Sources

PPS

Note

This is largely a legacy data source which is used in conjunction with the opto-isolated PPS output on legacy cameras

Contains the time of the last PPS in the sensor’s clock frame. Transmitted once per second on the MultiSense clock’s integer second boundary.

ORB Features

Note

Feature streams are only supported in firmware newer than v7.0. Please see the firmware update documentation for details on downloading and updating MultiSense firmware

Note

Currently only LibMultiSense and the MultiSense Viewer support streaming ORB features.

The MultiSense computes ORB keypoints and descriptors using it’s onboard compute. The onboard ORB detector can configured with the following specifications:

Number of Octaves : 8
Scale factor: 1.2
Number of feature points to detect: ~5000 (full res @ 5fps) ~1500 (quarter res @ 15 fps)
(optional) Motion detector: Averaged 2D vector of inter-frame matched feature points distance on octave #3 (Left camera only)
(optional) Feature point grouping: Function to group together feature points that indicate the same corner between different octaves
Descriptor size : 256bits (32 bytes)

The onboard ORB detector can compute keypoint and descriptor detections on a pair of image streams simultaneously. The two valid image stream pairs are the left and right camera or left and aux camera. The detector must be configured to process two streams simultaneously by setting the Feature Detectors grouping configuration to true.

Left ORB Features

ORB features computed on images from the left camera. These images have no distortion compensation applied.

Right ORB Features

ORB features computed on images from the right camera. These images have no distortion compensation applied.

Aux ORB Features

ORB features computed on images from the aux camera. These images have no distortion compensation applied.

Note

Only MultiSense S27 and MultiSense S30 cameras include an aux camera

Left Rectified ORB Features

ORB features computed on left camera images rectified using the MultiSense calibration.

Right Rectified ORB Features

ORB features computed on right camera images rectified using the MultiSense calibration.

Aux Rectified ORB Features

ORB features computed on aux camera images rectified using the MultiSense calibration.

Note

Only MultiSense S27 and MultiSense S30 cameras include an aux camera