esc
Start typing to search the docs
Navigate Open
Examples

Robotics

DROID semantic frame search

Find moments in robot demonstrations by describing them in plain language, and jump straight to the matching frame in the Rerun viewer.

DROID semantic frame search screenshot

This pulls frame embeddings from a Rerun dataset, indexes them in an external vector store, and resolves text queries back into deep-links that open the relevant frames in the viewer. It uses LanceDB as the concrete store, but the pattern — embeddings out of Rerun, search in your vector database of choice, deep-links back into the viewer — applies to any vector store.

What it shows off

How it works

Three scripts:

  1. prepare_dataset.py — registers a few DROID episodes to your local catalog as a dataset (using the episodes bundled in the repo, or downloading them from the Hugging Face Hub when those aren't present). This is the data the other two scripts read.

  2. ingest.py — builds the index. For each camera it auto-detects the embedding source:

    • Read path: if the dataset already has a /camera/{role}/embedding column (DROID registered with --create-embeddings), it reads those vectors directly via a DataFusion query — no model, no video decoding.
    • Compute path: otherwise it streams the VideoStream through the dataloader, decodes frames, and embeds them with SigLIP-2.

    Either way it writes (segment_id, camera, timestamp_ms, vector) rows into a searchable LanceDB table.

  3. search.py — embeds your example image or text prompt with the SigLIP-2 encoder, runs a vector search over the LanceDB table, prints the ranked matches, and opens the best one in the viewer.

Run the code

1. Install dependencies

This example has its own uv project, separate from the workspace .venv, because it needs the experimental rerun-sdk[dataloader] extras plus heavy ML deps (transformers, lancedb).

Standalone (sparse-checkout of just this directory, no local Rerun build):

uv sync --no-sources --no-dev

Monorepo dev (full repo checkout, editable local rerun-sdk):

cd examples/python/droid_semantic_search
RERUN_ALLOW_MISSING_BIN=1 uv sync
uv pip install ../../../rerun_py/rerun_dev_fixup

The second command installs the .pth shim that points import rerun (and the rerun CLI) at the in-repo editable source tree. It's a separate uv pip install rather than a dev-group dependency because uv resolves all dependency groups unconditionally, so a path-only package in pyproject.toml would break the standalone --no-sources resolution above.

Then either source .venv/bin/activate or prefix subsequent commands with uv run.

2. Start a local Rerun server

This example reads data from a local open-source Rerun catalog server. In a separate terminal, start one:

rerun server

This serves a catalog at rerun+http://127.0.0.1:51234 — the default the scripts use. Leave it running; the steps below connect to it.

3. Register a dataset

Registers a few DROID episodes to the catalog as droid:sample:

uv run python prepare_dataset.py

By default it auto-selects the source:

Useful flags:

4. Build the search index

Index a handful of segments (the exterior camera works well for scene-level search):

uv run python ingest.py --num-segments 15 --cameras ext1

These sample episodes ship video only (no pre-computed embeddings), so ingest.py takes the compute path: the first run downloads the SigLIP-2 model (a few hundred MB) and then decodes and embeds frames — the slow step. Start with a small --num-segments to keep it quick; raise it once you've seen it work.

5. Search

Search by text and open the best hit in the viewer:

uv run python search.py "an open drawer full of tools" --top-k 5

Or search by example image instead of text (same vector space):

uv run python search.py --image ./query.jpg --top-k 5

Queries that discriminate well on DROID describe concrete, visible objects/scenes, e.g. "a pink flower", "a cardboard box", "a white plastic bag", "a robot arm over an empty table".

Run ingest.py --help and search.py --help for the full flag list — index multiple cameras, change the sampling rate, widen the time selection around a hit, and more.

Scope and extension points

Notes and gotchas

Files