What is Rerun?
Here is a situation every robotics team knows. A robot did something wrong yesterday. The evidence arrives as a 4 GB folder: video files from three cameras, joint states in a custom protobuf schema, a JSON of calibration offsets, none of them time aligned. Someone spends the afternoon writing a script to line them up. By the time the plots render, it's tomorrow.
Rerun exists so that afternoon stops happening. It's the data layer for Physical AI: one way to log, store, query, and visualize the multimodal, multi-rate data that robots and perception systems produce. The same data model carries you from quick local debugging all the way to building training datasets. For the argument behind that, why an immature data layer quietly taxes every robotics team at every stage of the loop, see The data layer tax for robot learning: the companion read to this course.
Try it here
That's a real Rerun recording, running in your browser. Two Trossen arms, the blue one and the orange one, work over a tabletop. The 3D scene, the camera calibrations, and the joint plots are all one recording: press play and everything moves together, because everything shares a clock. If you want a tour of the panels, timeline, and controls, see Navigating the viewer in the docs.
This episode is the course's running example. It started life as an ordinary MCAP file, the log format many robots already write, full of real-world texture: joint states in a custom protobuf schema, four compressed video streams, camera calibrations with a bug in them, and no explicit 3D scene. Everything you see above was derived: the robot models come from URDF files, their motion is computed from the logged joint states, and the camera fix is part of the pipeline. By the end of this course you'll know exactly how that works, and you'll have taken a whole dataset of these episodes through the full lifecycle.
Under the hood, everything in a recording is entities (the paths in the tree on the left), components (the typed data on them), and archetypes (the bundles the viewer knows how to draw). You don't need that vocabulary to follow this course. When it matters, we link the data model and entities & components pages, which teach it properly. For now: paths, data, and "what kind of thing it is."
Let your coding agent read the manual
Rerun ships Agent Skills, focused playbooks for the exact tasks in this course: converting MCAP, deriving robot transforms from a URDF, querying a dataset, exporting to LeRobot. Install them into any agent-capable editor once:
npx skills add rerun-io/rerunNow when you ask your favorite coding agent to "convert this MCAP to Rerun," it works from Rerun's own patterns instead of guessing. The demo repo this course walks through uses these same skills, and we'll point at the relevant one at each stage.
The plan
The course follows the experiment loop, the cycle every Physical AI team lives in, using Rerun's Trossen demo as the hands-on companion. Its dataset: synthetic episodes of this bimanual Trossen setup (recorded in MuJoCo), small enough to run everything locally.
- Foundations (this article): meet a recording and see what Rerun is.
- Collect: convert raw MCAP into recordings, fixing recorded bugs and deriving the 3D scene, then register them as a queryable dataset.
- Refine: enrich recordings after collection and query across the whole dataset at once.
- Train: turn a filtered slice of the dataset into LeRobot training data, because the training set is a query, not a pipeline.
- Deploy: treat evaluation as recordings with provenance, then feed what you learn back into collection.
Everything in this course runs on your laptop against open-source Rerun. Open source and Rerun Hub give you the same capabilities and the same APIs — the difference is scale: open source is built for laptop-scale data, and Rerun Hub runs that exact same code on company-scale data across a fleet.
Next
The loop starts with your data. Collect: convert raw MCAP into recordings, fix the bugs hiding in the log, and derive the 3D scene you just watched.