Live depth sensor

Visualize the live-streaming frames from an Intel RealSense depth sensor.

This example requires a connected realsense depth sensor.

Used Rerun types

Pinhole, Transform3D, Image, DepthImage

Background

The Intel RealSense depth sensor can stream live depth and color data. To visualize this data output, we utilized Rerun.

Logging and visualizing with Rerun

The RealSense sensor captures data in both RGB and depth formats, which are logged using the Image and DepthImage archetypes, respectively. Additionally, to provide a 3D view, the visualization includes a pinhole camera using the Pinhole and Transform3D archetypes.

The visualization in this example were created with the following Rerun code.

rr.log("realsense", rr.ViewCoordinates.RDF, static=True)  # Visualize the data as RDF

Image

First, the pinhole camera is set using the Pinhole and Transform3D archetypes. Then, the images captured by the RealSense sensor are logged as an Image object, and they're associated with the time they were taken.

rgb_from_depth = depth_profile.get_extrinsics_to(rgb_profile)
rr.log(
    "realsense/rgb",
    rr.Transform3D(
        translation=rgb_from_depth.translation,
        mat3x3=np.reshape(rgb_from_depth.rotation, (3, 3)),
        relation=rr.TransformRelation.ChildFromParent,
    ),
    static=True,
)

rr.log(
    "realsense/rgb/image",
    rr.Pinhole(
        resolution=[rgb_intr.width, rgb_intr.height],
        focal_length=[rgb_intr.fx, rgb_intr.fy],
        principal_point=[rgb_intr.ppx, rgb_intr.ppy],
    ),
    static=True,
)

rr.set_time("frame_nr", sequence=frame_nr)
rr.log("realsense/rgb/image", rr.Image(color_image))

Depth image

Just like the RGB images, the RealSense sensor also captures depth data. The depth images are logged as DepthImage objects and are linked with the time they were captured.

rr.log(
    "realsense/depth/image",
    rr.Pinhole(
        resolution=[depth_intr.width, depth_intr.height],
        focal_length=[depth_intr.fx, depth_intr.fy],
        principal_point=[depth_intr.ppx, depth_intr.ppy],
    ),
    static=True,
)

rr.set_time("frame_nr", sequence=frame_nr)
rr.log("realsense/depth/image", rr.DepthImage(depth_image, meter=1.0 / depth_units))

Run the code

To run this example, make sure you have the Rerun repository checked out and the latest SDK installed:

pip install --upgrade rerun-sdk  # install the latest Rerun SDK
git clone git@github.com:rerun-io/rerun.git  # Clone the repository
cd rerun
git checkout latest  # Check out the commit matching the latest SDK release

Install the necessary libraries specified in the requirements file:

pip install -e examples/python/live_depth_sensor

To experiment with the provided example, simply execute the main Python script:

python -m live_depth_sensor # run the example

If you wish to customize it, explore additional features, or save it use the CLI with the --help option for guidance:

python -m live_depth_sensor --help