Exploiting column chunks for faster ingestion and lower memory use
To enable working with more kinds of datasets in Rerun, the 0.18 release introduces a new column oriented API and performance improvements to help handle larger time-series.
Rerun is an SDK, time-series database, and visualizer for temporal and multimodal data. It’s used in fields like robotics, spatial computing, 2D/3D simulation, and finance to verify, debug, and explain.
Rerun is open source 5,000pip install rerun-sdk
rerun
Log data like tensors, point clouds, and text to create streams. Easily correlate input, intermediate state, and output from multiple sources.
import rerun as rr
rr.init("my_data_generating_application")
rr.connect() # Connect to a remote viewer
…
rr.log("tensor", rr.Tensor(array))
rr.log("points", rr.Points3D(positions))
rr.log("text", rr.TextDocument(string))
Get powerful interactive visualizations of live and recorded streams. Time travel through your data, dynamically aligned to any timeline.
Express the semantics of your data with Rerun's composable data model. Build layouts and customize visualizations directly through code or interactively in the UI.
Use Rerun's dynamic schemas to log data in your own formats. Extend the Rerun Viewer with custom views or by embedding it in your own app.
# Protocols for custom data with Rerun
# Similar protocols exist in all languages
class AsComponents(Protocol):
"""Describes interface for interpreting an object as a bundle of Components."""
def as_component_batches(self) -> Iterable[rr.ComponentBatchLike]:
…
def num_instances(self) -> int | None:
…
class ComponentBatchLike(Protocol):
"""Describes interface for objects that can be converted to batch of rerun Components."""
def component_name(self) -> str:
…
def as_arrow_array(self) -> pa.Array:
…
Rerun's data model, a time aware Entity Component System, is designed to be expressive, composable, and extensible.
Everything from serialization, transport, out-of-order and multi-process ingestion, and multi-timeline indexing is handled for you.
Rerun understands spatial relationships by default. Use projections and transform hierarchies to connect spatial data.
The high performance and cross platform 2D/3D engine is built with Rust and wgpu to run native and in the browser with Wasm.
To enable working with more kinds of datasets in Rerun, the 0.18 release introduces a new column oriented API and performance improvements to help handle larger time-series.
Rerun 0.17 introduces blueprint defaults and overrides, bringing a massive step up in user control. For any visualizer input, you can now use the blueprint to set default values or override the data you logged.
You can now specify some blueprint properties through the Python log API. Among other things this lets you control the visible time range query per view from code.
Rerun 0.15 introduces Blueprint Python APIs for controlling the layout and contents of views in the Rerun Viewer. It also introduces the ability to save and load blueprints from file.
This is a follow up post that dives into how we managed to achieve such huge performance gains for time series (among other things), and why it was so hard in the first place.