Geospatial data and full H.264 support
Rerun is building the multimodal data stack for robotics and spatial intelligence. Many of our users' products move about the physical world so we're particularly excited that the 0.20 release adds the first piece of supporting geospatial data in Rerun.
This video shows the new map view in an example visualization of the nuScenes dataset.
The geospatial support is a combination of two things: a set of geospatial primitives you can send to Rerun, and a new map view that draws these primitives on top of map tiles from either OpenStreetMap or Mapbox.
The new geospatial primitives were designed to be compatible with GeoArrow, and in future releases we'll make sure interop is both low friction and high performance. This release includes the GeoPoints and GeoLineStrings archetypes and a future release will include a GeoPolygons archetype as well for full compatibility.
Geospatial support in Rerun has been heavily requested for a long time and this effort was even kicked off by a community contribution by Tamás Földi who opened the first pull request for the maps view, which we very much appreciate.
You can read more about supported coordinate systems and how to use these new features here.
Using FFMpeg to decode H.264 video in the native viewer using-ffmpeg-to-decode-h264-video-in-the-native-viewer
Rerun 0.20 also brings the second part of video support in Rerun: support for H.264 video files in the native viewer. The next piece after this will be support for sending individual video encoded frames at a time to Rerun without the need for a container format like mp4. This then will pave the way to live ingest video streams as they are encoded.
Video decoding in Rerun is handled by interfacing with an FFmpeg executable installed on the user’s system. We do this using the excellent open source crate ffmpeg-sidecar. We chose this approach to make Rerun easy to embed in all environments, whether commercial or non-commercial. Since critical parts of FFmpeg are GPL-licensed, bundling or even linking against it would have restricted this flexibility. By accessing FFmpeg externally, we provide optional native H.264 support that can be disabled on a case-by-case basis.
This release also comes with many general stability and performance improvements. That includes updates to our standalone open source crate for parsing mp4 files that significantly reduce time and memory overhead for opening mp4 files. Among other things, we addressed issues around accurate timing and sample re-ordering (something very common in H.264 & H.265 videos). To that end, we added advanced inspection options for the raw video data and the frames produced by the video decoder. These are geared towards helping ourselves to better understand video streams and to debug issues, but they may also be useful if you want to gain a deeper understanding yourself:
Video is a complex area and we're only starting our journey here, stay tuned for more!
Try it out and let us know what you think try-it-out-and-let-us-know-what-you-think
We're really excited to heat how the new geospatial features work for you. It's a first step into a very deep field but we're pretty happy with this first step and hope it will be useful for many teams. The best places to start are either this guide or this simple OpenStreetMap example.
Join us on Github or Discord and let us know what you think and would hope to see in the future.