Working with Le­Robot datasets

Rerun reads LeRobot datasets, the directory-based format used for robot-learning datasets. Both the v2.1 and v3.0 dataset layouts are supported.

Viewing a LeRobot dataset

A LeRobot dataset is a directory (metadata, parquet, and video files), so point Rerun at the dataset directory:

rerun path/to/lerobot_dataset

You can also drag and drop the dataset directory into the Rerun Viewer, or load it using the SDK:

"""Load a LeRobot dataset using the Python SDK."""

import sys

import rerun as rr

path_to_lerobot_dataset = sys.argv[1]

# Initialize the SDK and give our recording a unique name
rr.init("rerun_example_load_lerobot", spawn=True)

# Load the LeRobot dataset (a directory of metadata, parquet, and video files)
rr.log_file_from_path(path_to_lerobot_dataset)
recording = rr.get_data_recording()
assert recording is not None
recording.flush()

Converting a LeRobot dataset to RRD

To query a LeRobot dataset on a catalog server or train on it with the Rerun dataloader, convert each episode to its own .rrd file with LeRobotReader:

from pathlib import Path

import rerun as rr

Path("rrd").mkdir(exist_ok=True)
profile = rr.chunk.OptimizationProfile.OBJECT_STORE

reader = rr.experimental.LeRobotReader("path/to/lerobot_dataset")
for episode in reader.episodes():
    reader.stream(episode).collect(optimize=profile).write_rrd(
        f"rrd/episode_{episode}.rrd",
        application_id="my_dataset",
        recording_id=f"episode_{episode}",
    )

• The recording_id becomes the segment ID once the file is registered, so pick one that names the episode.
• collect(optimize=profile) merges chunks into larger ones for catalog queries, at the cost of holding the episode in memory. Without it, write_rrd streams the episode straight to disk.
• Video without B-frames, such as AV1 (LeRobot's default), is copied as-is. H.264 or H.265 with B-frames is re-encoded, and a v3.0 episode that starts mid-GOP has its first GOP re-encoded. Both need ffmpeg on the PATH.

Then load the folder into a local catalog with rerun server -d my_dataset=rrd, or register the files to an existing dataset.

Data model

Each episode becomes its own recording, and each feature becomes an entity named after the feature key:

Dots in a feature key are part of the entity name, not path separators: observation.images.up becomes the single entity /observation.images.up. A content filter such as /observation.images.** therefore matches nothing, so use the full entity name.

The dataset's frame rate is not stored in the recording. To convert frame_index to seconds, divide by fps from the dataset's meta/info.json.

Related

To go the other way — querying recordings and exporting them as a LeRobot dataset — see Export recordings to LeRobot datasets.