Robotics

LeRobot dataset from RRD

Convert robot recordings into training-ready datasets by using the OSS Rerun server to query and transform RRD files into LeRobot v3 format.

Background

This example demonstrates how to use the Rerun OSS server API to process robot recordings and prepare them for imitation learning. The workflow uses the server to load RRD files, query robot data (actions, observations, videos), align time series to a target framerate, and write the result as a LeRobot v3 dataset compatible with robotics model training pipelines.

LeRobot is a project by Hugging Face that provides models, datasets, and tools for real-world robotics in PyTorch. This example shows how Rerun recordings can be converted into LeRobot's standardized dataset format.

Conversion workflow

The converter loads RRD files into the OSS server, infers data types from the recordings, resamples all data to a target framerate, and writes the result as a LeRobot v3 dataset. Video streams are efficiently remuxed without re-encoding.

Run the code

Installation

This example has its own uv project, separate from the workspace .venv, because LeRobot pins an incompatible rerun-sdk.

Standalone (sparse-checkout of just this directory, no local Rerun build):

uv sync --no-sources --no-dev

Monorepo dev (full repo checkout, editable local rerun-sdk):

cd examples/python/rerun_export
RERUN_ALLOW_MISSING_BIN=1 uv sync
uv pip install ../../../rerun_py/rerun_dev_fixup

Then either source .venv/bin/activate or prefix subsequent commands with uv run.

Basic usage

The example provides a CLI that converts a directory of RRD recordings into a LeRobot v3 dataset:

rerun_export \
  --rrd-dir /path/to/recordings \
  --output /path/to/output/dataset \
  --dataset-name my_robot_dataset \
  --fps 10 \
  --index real_time \
  --action /action:Scalars:scalars \
  --state /observation/joint_positions:Scalars:scalars \
  --task /language_instruction:TextDocument:text \
  --video front:/camera/front:VideoStream:sample

Video specification format

Videos are specified as key:path:

key: Camera identifier (e.g., front, wrist)
path: Entity path to the video stream (e.g., /camera/front)

The converter expects VideoStream, components at the specified paths.

Example workflow

Here's a complete example converting simulated robot teleop data:

# Convert RRD recordings to LeRobot dataset
rerun_export \
  --rrd-dir ./robot_recordings \
  --output ./lerobot_dataset \
  --dataset-name robot_demos \
  --fps 15 \
  --action /robot/action:Scalars:scalars \
  --state /robot/state:Scalars:scalars \
  --task /task:TextDocument:text \
  --video front:/camera/front:VideoStream:sample \
  --video wrist:/camera/wrist:VideoStream:sample \
  --action-names "joint_0,joint_1,joint_2,gripper" \
  --state-names "joint_0,joint_1,joint_2,gripper"

# The resulting dataset can be used with LeRobot training scripts

The output directory will contain:

data/: Parquet files with aligned time series data
videos/: Encoded video files (if using --use-videos)
meta/: Dataset metadata and episode information