Rerun

Introducing Experimental MCAP Support

photo of Jochen Görtler
Written by Jochen Görtler 1 day ago

Introducing experimental MCAP support in Rerun introducing-experimental-mcap-support-in-rerun

In the most recent release, we're adding experimental MCAP support to Rerun to make it easier to visualize your existing robot data without changing your current workflows or data formats. This is still an early feature, but we're eager to share this development with the community. We believe it should be easy to visualize robotics data, no matter the format.

Let's get into what we've built and why we think it's important. Firstly, MCAP is an open-source format for storing (generally robotics) log data.

From mcap.dev:

MCAP is an open-source container file format for multimodal log data. It supports multiple channels of timestamped pre-serialized data, and is ideal for use in pub/sub or robotics applications.

Motivations motivations

MCAP has been the default bag format in ROS2 since May 2023, and over the past year, we've seen more interest from the community in the ability to visualize MCAP data with Rerun.

At Rerun, we believe strongly that open-source visualization for robotics data is crucial for the future of robotics research and development. We're excited to start this journey with MCAP support, and we're committed to making it as robust and user-friendly as possible. We welcome feedback from the community to help us improve our support.

Our Approach our-approach

We want to discuss a bit about our approach to MCAP support in Rerun before we share how to get started.

The data model the-data-model

As you may know, Rerun's data model is based on an entity component system that is distinct from the topic/message model of MCAP.

We had to make some decisions in order to bridge the gap. Those decisions are primarily:

  1. MCAP topics correspond to Rerun entities.
  2. MCAP topics are stored in Rerun chunks.
  3. MCAP message contents will be extracted into Rerun components and grouped under the corresponding Rerun archetype.
  4. log_time and publish_time from MCAP messages will carry over to Rerun as two different timelines.

Lots of design choices are embedded in these assumptions. Let's get into some more detail.

Extracting in layers extracting-in-layers

Rerun uses a layered architecture to process MCAP files at different levels of abstraction. This design allows the same MCAP file to be ingested in multiple ways simultaneously, from raw bytes to semantically meaningful visualizations. This builds on some recent Rerun internals that we'll be diving deeper into in the coming months.

Each layer extracts different types of information from the MCAP source and maps to distinct Rerun archetypes.

For the purposes of this post, Rerun extracts data from the MCAP files into layers. Those layers are subsequently queryable (and visualizable) by our users. By default, Rerun analyzes an MCAP file to determine which layers are active to provide the most comprehensive view of your data, while avoiding duplication.

Message interpretation message-interpretation

At the core of MCAP support is how we interpret the contents of MCAP messages. This is critical to touch on but also under development and subject to change.

ROS2 semantic interpretation

The ros2msg layer provides semantic interpretation and visualization of standard ROS2 message types. This creates meaningful Rerun visualization components from data.

This layer understands the semantics of ROS2 messages and creates appropriate visualizations: images become Image, point clouds become Points3D, IMU messages become SeriesLines with the data plotted over time, and so on. This makes it easy to get up and running quickly.

This layer supports standard ROS2 packages including sensor_msgs, geometry_msgs, std_msgs, and builtin_interfaces. This layer provides visualization of sensor data like cameras and LiDAR with minimal setup required as well.

Protobuf decoding

Protobuf decoding/interpretation works a little differently than the ROS2 implementation.

The protobuf layer automatically decodes protobuf-encoded messages using reflection. This creates structured component data based on the protobuf schema.

To do this, we parse the protobuf definitions in the schema of the topic and use that to parse the (nested) protobuf types into Arrow types. Message fields then become Rerun components that you can query and analyze.

However, this layer provides structured access without semantic visualization meaning. While the data becomes queryable, it won't automatically appear as meaningful visualizations like images or point clouds, it gives you the data structure, not the visual interpretation.

This is an important nuance and one that we'll dive deeper into as we talk more about querying this kind of data in follow-up posts. Suffice it to say that we make the data available, but not necessarily usable for immediate visualization, like we do with ROS2.

Accessing layer data accessing-layer-data

Each layer creates different archetypes on the same entity paths (derived from MCAP channel topics) that can be accessed through Rerun's SDK:

  • Data from the protobuf layer appears as structured components that can be queried by field name
  • Data from the ros2msg layer appears as native Rerun visualization components (Image, Points3D, etc.)
  • Data from the raw layer appears as blob components containing the original message bytes
  • Metadata from schema, stats, and recording_info layers appears as dedicated metadata entities

For more information on querying data and working with archetypes, see how to query data out of Rerun.

Getting started getting-started

Getting started is easy!

You can drag-and-drop MCAP files directly into the Rerun viewer or use File > Open to load them.

mcap-demo

Of course, you can also load MCAP files directly into Rerun using the CLI:

pip install rerun-sdk
rerun recording.mcap

Conversion conversion

Alternatively, you can convert an MCAP file into a .rrd file:

rerun mcap convert input.mcap -o output.rrd

You can do that on a per-layer basis as well:

# Use only specific layers
rerun mcap convert input.mcap -l stats -o output.rrd

# Use multiple layers for different perspectives
rerun mcap convert input.mcap -l ros2msg -l raw -l recording_info -o output.rrd

Doing this conversion will enable more capabilities that we'll talk about in upcoming blog posts.

What's supported today whats-supported-today

Our initial implementation focuses on common ROS2 message types that are heavily used for visualization:

  • sensor_msgs
  • std_msgs
  • geometry_msgs
  • builtin_interfaces

We've prioritized message types that map well to Rerun's existing visualization capabilities, but we will cover more and more over time—input from the community on what messages to add will help us prioritize.

Important limitations important-limitations

This is a significant first step towards making state-of-the-art open source visualization accessible to wider parts of the robotics ecosystem. But our MCAP support is still nascent.

As it stands, many message types, especially custom messages, are not yet supported and will not map cleanly to Rerun archetypes. tf transforms are missing (in progress), ROS1 messages are not supported, and there are a number of other limitations:

  1. MCAP files larger than available RAM cannot currently be processed.
  2. ROS1 messages are not supported in this initial release.
  3. Custom messages are not yet supported or at least need to be converted to a Rerun archetype.

This is all subject to change as we continue to develop our MCAP support.

Conclusion and where to learn more conclusion-and-where-to-learn-more

We're excited to continue work here and invite you to provide your feedback on what we're doing! Join the Discord to discuss or submit issues on GitHub.

Read more in-depth about MCAP layers and MCAP message formats.

DocsBlogExamplesTeamCareers
Privacy policy