Visualize depth images, heightmaps, and voxels

Depth image to 3D point cloud

Log a DepthImage under an entity with a Pinhole camera. The 3D view then projects every pixel into 3D, so you get a point cloud without computing it yourself. Set meter to the number of depth units per meter (for example 1000 for depth in millimeters).

"""Create and log a depth image and pinhole camera."""

import numpy as np

import rerun as rr

depth_image = 65535 * np.ones((200, 300), dtype=np.uint16)
depth_image[50:150, 50:150] = 20000
depth_image[130:180, 100:280] = 45000

rr.init("rerun_example_depth_image_3d", spawn=True)

# If we log a pinhole camera model, the depth gets automatically
# back-projected to 3D
rr.log(
    "world/camera",
    rr.Pinhole(
        width=depth_image.shape[1],
        height=depth_image.shape[0],
        focal_length=200,
    ),
)

# Log the tensor.
rr.log(
    "world/camera/depth",
    rr.DepthImage(depth_image, meter=10_000.0, colormap="viridis"),
)

To place the camera in the world, log a Transform3D to the camera entity. The same depth image is also shown as an image in a 2D view. point_fill_ratio sets the size of the points, and colormap and depth_range set their colors.

Heightmaps and elevation grids

There is no built-in heightmap or surface archetype. Build a Mesh3D from the grid: one vertex per cell, and two triangles per grid square.

import numpy as np
import rerun as rr

rr.init("rerun_example_heightmap", spawn=True)

rows, cols = 100, 120
cell_size = 0.1
y, x = np.mgrid[0:rows, 0:cols] * cell_size
height = 0.5 * np.sin(x) * np.cos(y)

positions = np.column_stack([x.ravel(), y.ravel(), height.ravel()])

index = np.arange(rows * cols).reshape(rows, cols)
top_left = index[:-1, :-1].ravel()
top_right = index[:-1, 1:].ravel()
bottom_left = index[1:, :-1].ravel()
bottom_right = index[1:, 1:].ravel()
triangles = np.concatenate([
    np.column_stack([top_left, bottom_left, top_right]),
    np.column_stack([top_right, bottom_left, bottom_right]),
])

rr.log("terrain", rr.Mesh3D(vertex_positions=positions, triangle_indices=triangles))

The mesh has no normals, so it is drawn without shading. Pass vertex_normals to get lighting, and vertex_colors to color it by height. See Control mesh appearance.

For a quick look, you can instead log the grid as a Points3D with the same positions, or as a 2D Image or DepthImage with a colormap.

Voxels

For sparse voxel grids, such as occupancy maps, use VoxelGridMap. It takes integer voxel indices, a voxel size, and optional values with a colormap. VoxelGridMap is unstable: its data format may change in future releases.

"""Log a simple sparse voxel grid map."""

import numpy as np

import rerun as rr

voxel_indices = np.array(
    [
        [-1, 0, 0],
        [1, 0, 0],
        [1, 1, 0],
        [3, 0, 0],
        [3, 0, 1],
        [4, 0, 1],
    ],
    dtype=np.int32,
)
values = np.array([0.0, 0.2, 0.4, 0.6, 0.8, 1.0], dtype=np.float32)

rr.init("rerun_example_voxel_grid_map_simple", spawn=True)

rr.log(
    "world/voxels",
    rr.VoxelGridMap(
        voxel_indices,
        voxel_size=[0.25, 0.25, 0.25],
        values=values,
        value_range=[0.0, 1.0],
        colormap=rr.components.Colormap.Turbo,
        translation=[-0.5, -0.5, 0.0],
    ),
)

You can also draw voxels as Boxes3D with fill_mode="solid", one box per voxel. This works for a moderate number of voxels.

Dense volumes

For dense 3D scalar fields, such as CT or MRI scans, signed distance fields, or occupancy probabilities, use Volume3D. It is rendered with ray marching. Volume3D is unstable: its data format may change in future releases.

"""Log a small volume: a soft sphere sampled on a 32³ grid."""

import numpy as np

import rerun as rr

SIZE = 32
RADIUS = np.float32(14.0)

rr.init("rerun_example_volume3d_simple", spawn=True)

# Squared distance from the center of the grid, per voxel, in voxel units.
axis = np.arange(SIZE, dtype=np.float32) - np.float32(0.5 * (SIZE - 1))
z, y, x = np.meshgrid(axis, axis, axis, indexing="ij")
distance_sq = x * x + y * y + z * z

# A soft sphere: 1 at the center, falling off to 0 at `RADIUS`.
values = np.maximum(
    np.float32(0.0), np.float32(1.0) - distance_sq / (RADIUS * RADIUS)
)

# Dimensions are ordered `[z, y, x]`. Only `f16` values are supported for now.
rr.log(
    "volume",
    rr.Volume3D(values.astype(np.float16), voxel_size=[0.1, 0.1, 0.1]),
)

For 2D occupancy grids, for example from a robot's map, see GridMap.