Collect: from MCAP to recordings

The experiment loop starts where data is born, and on most robots it's already being born as MCAP, the format much of the ROS 2 world writes. Collection in practice is rarely "log from scratch." It's convert what the robot already produced into something you can work with.

The raw material

That's a raw episode before careful conversion, opened naively by the viewer. It's unsurprising robot data: H.264 video streams, joint states in a custom protobuf schema, a task description, and camera calibrations. Real logs also have bugs (here, two cameras have their calibration width and height swapped).

Conversion is a pipeline

For a quick look, conversion is zero work: the viewer reads MCAP directly, as seen above. But the converted episode you explored in Foundations had a full 3D scene derived from the log. That's the point of a conversion step: it's where you fix and enrich. The demo's preprocessing.py module turns one episode_*.mcap into Rerun recordings and, in the same pass:

• fixes the recorded bug: corrects the swapped camera resolution;
• derives the 3D scene: feeds the logged joint states through each robot's URDF to compute the /tf transforms for every link, added as a layer alongside the original data.

At its core the conversion is small. The raw MCAP becomes a lazy chunk stream, transforms enrich it in place, and the result is written back out as a recording:

# src/trossen_oss/preprocessing.py
stream = McapReader(episode_mcap).stream()      # read the raw MCAP
# ...lenses fix the calibration bug and derive /tf from the URDF...
stream.collect().write_rrd(data_rrd, recording_id=recording_id)

The full version, lenses and all, is preprocessing.py. To speed up your Rerun adoption there are two skills you installed earlier: rerun-mcap for decoding topics (including custom protobuf) and rerun-urdf for the forward-kinematics layer, both built on Rerun's chunk-processing API (which simplifies this process for humans and agents). Ask your agent to convert the file and the skills supply the patterns.

Run it yourself

The recording above isn't a video — it's a few minutes of work on your own laptop. From the demo repo:

pixi install        # set up the environment
pixi run download   # fetch 10 sample episodes
pixi run preprocess # MCAP → Rerun recordings in outputs/rrds/

Open one of the resulting .rrd files in the viewer and you're looking at the converted episode from Foundations, built by the code above. Full setup is in the repo's README.

Explore and extend

• In the raw MCAP embed above, select /external/cam_high and compare Pinhole:resolution against the video's actual aspect ratio. That's the bug the pipeline fixes.
• In the converted episode (Foundations), select a robot link. Everything on it was derived at conversion time, not recorded.

Next

One file is a demo; a folder is a dataset. Refine: register the recordings, enrich them after the fact, and query across all of them at once.