Rerun

Export recordings to LeRobot datasets

Convert Rerun data into the training dataset version of your choice. This guide demonstrates how to use the OSS Rerun server to query recordings, align multi-rate sensor data to a common timeline, and export the result as a LeRobot dataset.

Prerequisites prerequisites

This example requires the rerun_export package from the Rerun repository:

pip install -e examples/python/rerun_export

This will install the necessary dependencies including LeRobot, DataFusion, and PyArrow.

Time alignment and resampling time-alignment-and-resampling

By default, the export uses the frame rate specified in the config to create evenly spaced samples (a LeRobot requirement).

For more details on time alignment, see Time-align data.

Setup setup

Start a local server and load your recordings. Each recording becomes a segment in the dataset, and each unique segment id becomes one LeRobot episode.


from pathlib import Path

import rerun as rr
from lerobot.datasets.lerobot_dataset import LeRobotDataset  # type: ignore[import-untyped]
from rerun_export.lerobot.converter import convert_dataframe_to_episode
from rerun_export.lerobot.feature_inference import infer_features
from rerun_export.lerobot.types import LeRobotConversionConfig, VideoSpec

# Start a server with RRD recordings
# In practice, you would point this to your directory of RRD files
sample_5_path = Path(__file__).parents[4] / "tests" / "assets" / "rrd" / "sample_5"

server = rr.server.Server(datasets={"robot_dataset": sample_5_path})
client = server.client()
dataset = client.get_dataset(name="robot_dataset")

See Catalog object model for how recordings are represented on the Data Platform.

Filter data for training filter-data-for-training

Robot recordings often contain more data than needed for training. Filter the dataset to include only the relevant entity paths and components that will map to LeRobot’s standardized format.

For example, you might include joint position commands as actions, joint states and end-effector pose as observations, RGB camera streams as video inputs, and a language instruction as the task description. Other signals such as debug visualizations, intermediate computations, or unused sensors can be excluded.

# Select a single recording (episode) to export
single_recording = dataset.segment_ids()[0]

# Filter the dataset to include only the data we need for training:
# - Action commands sent to the robot
# - Observed joint positions (robot state)
# - Camera feeds from multiple viewpoints
# - Task descriptions (e.g., language instructions)
training_data = (
    dataset.filter_segments(single_recording)
    .filter_contents([
        "/action/joint_positions",
        "/observation/joint_positions",
        "/camera/**",
        "/language_instruction",
    ])
    .reader(index="real_time")
)

Configure the export configure-the-export

Define how to map the data to LeRobot's standardized format. This requires specifying:

  • Which components contain actions and observations
  • Video streams to include
  • Target frame rate for the dataset
  • Timeline to use for alignment
# Define how to extract task instructions from the recording
# This could be from a TextDocument, static metadata, etc.
# For this example, we assume a static instruction
instructions = "/language_instruction:TextDocument:text"

# Specify video streams to include in the dataset
# Each stream needs a key (camera identifier) and entity path where the VideoStream is logged
videos = [
    VideoSpec(key="ext1", path="/camera/ext1", video_format="h264"),
    VideoSpec(key="ext2", path="/camera/ext2", video_format="h264"),
    VideoSpec(key="wrist", path="/camera/wrist", video_format="h264"),
]

# Configure the conversion parameters
# This maps Rerun's flexible data model to LeRobot's standardized format
config = LeRobotConversionConfig(
    fps=15,  # Target framerate for the dataset
    index_column="real_time",  # Timeline to use for alignment
    action="/action/joint_positions:Scalars:scalars",  # Fully qualified action column
    state="/observation/joint_positions:Scalars:scalars",  # Fully qualified state column
    task=instructions,  # Task description column
    videos=videos,  # Video streams to include
)

Infer feature schema infer-feature-schema

LeRobot uses a schema called "features" to describe dataset structure. The infer_features function automatically creates this schema by inspecting your data.

# Infer the LeRobot feature schema from the data
# This automatically detects data types, shapes, and creates the appropriate
# LeRobot feature definitions
features = infer_features(
    table=training_data.to_arrow_table(),
    config=config,
)

Feature inference examines the underlying data to determine:

  • Data types (float, int, image, video, text)
  • Array shapes (scalar, vector, matrix)
  • Video dimensions (by decoding the first frame)

Create the LeRobot dataset create-the-lerobot-dataset

Create the LeRobot dataset instance, using the LeRobot dataset API:

# Create the LeRobot dataset structure on disk
lerobot_dataset = LeRobotDataset.create(
    repo_id="rerun/droid_lerobot",  # Dataset identifier
    fps=config.fps,
    features=features,
    root=TMP_DIR / "lerobot_dataset",
    use_videos=config.use_videos,
)

Note, root is where the dataset files will be written, and LeRobot requires this to be an empty or non-existing directory.

Export the episode export-the-episode

Convert the filtered data into a LeRobot episode. This is the core transformation step.

# Convert the recording to a LeRobot episode
# This aligns all time series to the target framerate, extracts video frames,
# and writes the episode data in LeRobot's Parquet format
print("Creating episode")

convert_dataframe_to_episode(
    df=training_data,
    config=config,
    lerobot_dataset=lerobot_dataset,
    segment_id=single_recording,
    features=features,
)

# Finalize the dataset (write metadata, close files, etc.)
lerobot_dataset.finalize()

The convert_dataframe_to_episode function performs time alignment and resamples the dataframe to the target frame rate. It generates a sequence of evenly spaced timestamps at the target frame rate and treats these as the canonical timesteps for the episode. For each timestep, it queries the most recent available value of every selected component using Rerun’s latest-at semantics. If a stream has no sample exactly at that time, its last observed value is forward-filled.

The finalize() call completes the dataset by writing metadata and closing all files.

Multi-episode export multiepisode-export

To export multiple recordings as separate episodes, iterate over segment IDs:

# Create a new LeRobot dataset for multiple episodes
lerobot_dataset = LeRobotDataset.create(
    repo_id="rerun/droid_lerobot_full",
    fps=config.fps,
    features=features,
    root=TMP_DIR / "lerobot_dataset_full",
    use_videos=config.use_videos,
)

# To export multiple recordings, repeat the filtering and conversion steps
for segment_id in dataset.segment_ids():
    print(f"Exporting segment: {segment_id}")

    segment_data = (
        dataset.filter_segments(segment_id)
        .filter_contents([
            "/action/joint_positions",
            "/observation/joint_positions",
            "/camera/**",
            "/language_instruction",
        ])
        .reader(index="real_time")
    )

    convert_dataframe_to_episode(
        df=segment_data,
        config=config,
        lerobot_dataset=lerobot_dataset,
        segment_id=segment_id,
        features=features,
    )

lerobot_dataset.finalize()

Using the exported dataset using-the-exported-dataset

The exported LeRobot dataset can be used directly with LeRobot's training scripts:


# Access episode data
print(f"Total episodes: {lerobot_dataset.num_episodes}")
print(f"Total frames: {lerobot_dataset.num_frames}")

Or push it to the Hugging Face Hub for sharing:

dataset.push_to_hub(repo_id="your-username/your-dataset-name")

Command-line interface commandline-interface

The rerun_export package includes a CLI that implements this workflow for batch processing:

rerun_export \
  --rrd-dir ./tests/assets/rrd/sample_5 \
  --output ./lerobot_dataset \
  --dataset-name rerun-example-droid \
  --fps 15 \
  --action /action/joint_positions:Scalars:scalars \
  --state /observation/joint_positions:Scalars:scalars \
  --task /language_instruction:TextDocument:text \
  --video ext1:/camera/ext1 \
  --video ext2:/camera/ext2 \
  --video wrist:/camera/wrist

See the rerun_export example for the complete implementation.

Privacy policy