Backwards compatibility of .rrd files
Since we open sourced Rerun in 2023, one of the top requested features has been backwards data compatibility: the ability to open old data files with the newest Rerun Viewer. So why has this taken so long?
It’s all about balancing agility against stability.
Early on in a startup, everything is flux. There is a lot of experimentation going on in a closed loop with users. We build a feature, ship it, and then see how users use it. We take that as feedback and improve things, and the cycle repeats. At this stage, we don’t want to be slowed down by a focus on stability.
But now we’ve reached a point where companies are building their technology stack on Rerun, and we can’t keep shifting the ground under their feet. Luckily, we’ve also reached a point where we are happy with our data model. So, it’s time to introduce some data stability!
Over the last couple of months we’ve made some final breaking changes to our .rrd files (sorry!) in order to make 0.23 the last release where we break them.
This does not mean we will stop evolving, but we will evolve the data format in such a way that the old data can be migrated to newer Rerun versions. For instance: we’re planning on changing the Arrow encoding of our TensorBuffer type, but we will migrate old tensors on‑the‑fly when loading old files (and do so in a zero-copy way).
There are some limitations. As a first step, we’re only promising that Rerun can load the data from the previous Rerun version. This means you will be able to open an 0.23 .rrd file in Rerun 0.24, but maybe not in Rerun 0.31.
We’re also planning on doing some potentially breaking changes to some of our more exotic types, like AnnotationContext and ViewCoordinates (and that is now called out in the documentation for these types), but all the core data primitives will continue to load, including all text, images, videos, point clouds, scalars, and tensors.
We’re also not promising complete backwards compatibility of our APIs yet - that will have to wait for Rerun 1.0. Meanwhile we will evolve it so that old functions and types still work with clear deprecation notices and migration guides.