Spotlight: 3D reconstruction is a data layer problem

Written by Pablo Vela 1 day ago

One of the recent developments in robotics is the use of human demonstration to scale up robot training data. Many robotics labs, Physical Intelligence, Tesla, and Sunday Robotics among them, collect human demonstrations with body-worn cameras.

But raw first-person video is not training data. To become training data, it needs 3D structure attached: where the camera was at every frame, how far every surface is, what the room looks like. Hand tracking, trajectory extraction, all of it sits on top of that structure.

A system that produces all of this already lives in everyone's pocket. Point an iPhone at a room and ARKit returns poses, depth, and a mesh, in real time. Unfortunately it's also closed source, Apple-only, and built for on-device speed rather than dataset quality.

This series is about replacing that black box with the explicit purpose of generating better robot training data: an open source pipeline that does what ARKit does, on whatever hardware you have. It's also a great excuse for me to finally build it end-to-end. I have been obsessed with 3D reconstruction for a long time, and this project gives me the chance to scratch that itch.

Open source means the trade-offs become ours to make. Run it on Android? Sure. Swap in a slower but higher-quality model? Great. Re-process every old capture when the models improve? Totally in our control with an open stack.

To build this, I need some ground truth to measure against. This led me to ARKitScenes, a dataset of roughly 5,000 indoor scans of real-world apartments, captured entirely on an iPhone (and an iPad). Each scan carries ARKit's predictions and each room also has a laser-scanned reference, so every piece I replace can be scored against both.

A dataset on its own is not a baseline, though. It only becomes one once a data pipeline can reliably serve it to everything downstream, and building that pipeline on Rerun's open source data layer is what this first post walks through.

What is 3D reconstruction?

3D reconstruction is the problem of recovering the geometry of a space from ordinary 2D observations. Given a set of images, you work out where the camera was when it captured each one, and where the surfaces in the room actually sit. Classically, that means matching features across frames, solving for camera poses, then estimating depth per frame and fusing it into a single surface.

Two stages get you from a walk through a room to a mesh.

The first is working out where the camera was, which is what SLAM does: simultaneous localization and mapping. You track visual features from frame to frame and solve for the camera motion that explains how they moved. Feature tracking on its own drifts, and it struggles with blank walls and fast motion, so it is fused with the IMU, the inertial measurement unit that reports acceleration and rotation rate around a hundred times a second. The output is a camera pose for every frame.

The second is turning depth into a surface. Once every frame has a pose, each pixel of a depth map becomes a point in one shared world frame. The usual method is TSDF fusion, short for truncated signed distance field: a voxel grid where each voxel stores how far it is from the nearest surface, updated by every frame that sees it. Averaging many noisy views cancels most of the error, and a mesh is extracted from the result.

An iPhone hands you both stages. ARKit runs the tracking and returns a pose per frame, and the LiDAR returns the depth maps.

The hardware we'll be using

There are two versions of the iPhone that I want to target, differing by a single sensor. The base model and the pro model.

Resolutions are as captured in ARKitScenes, and the fields of view are measured from its calibration. Both cameras point the same direction, so the ultrawide captures most of a room at once while the wide camera captures the detail.

What we are building

Replacing ARKit for the purposes of 3D reconstruction takes three pieces.

1. Visual-inertial SLAM. Images and IMU in, a camera pose per frame out.
2. Depth from images. Given those poses, predict a low-resolution depth map from the images alone, the way SimpleRecon does. This is what stands in for the LiDAR on a phone that does not have one.
3. Depth upsampling. Poses, images, and low-resolution depth in, high-resolution depth out. Prompt Depth Anything (PromptDA) treats the sparse depth as a prompt and predicts a dense one.

We start at the third piece and work backwards, because a Pro iPhone hands us the first two for free. ARKit runs the SLAM and the LiDAR supplies the low resolution depth, which leaves upsampling as the only part that needs building. The other two come in part 2.

Working backwards has a second advantage. Each piece we replace can be checked against the one Apple already ships, so we always have something to be wrong against.

What ARKitScenes provides

ARKitScenes is 5,047 indoor scans, each one a walk through a room with a LiDAR-equipped iPhone or iPad. Every scan carries what the device recorded:

• wide camera video at 1920×1440;
• ultrawide camera video;
• LiDAR depth at 256×192, with a per-pixel confidence map;
• IMU;
• a camera pose per frame, from ARKit.

Each venue was also scanned separately with a stationary laser scanner, which supplies the ground truth:

• a high resolution mesh of the room;
• oriented 3D bounding boxes with labels;
• high resolution depth rendered from that scan.

That combination makes ARKitScenes close to ideal for establishing a baseline, which is the first thing the data stage in the lifecycle above asks for. Whatever piece I replace, ARKit's own answer and a laser-scanned one are both sitting there to measure against, on the same rooms.

Why the data layer is the hard part

I want to walk through some of the challenges that come with building a good baseline in the default format of ARKitScenes. Each scan ships as a pile of formats: a .mov container for the video, folders of PNG frames for color, depth, and confidence, per-frame intrinsics as text files, a .traj file for the poses, a JSON of 3D boxes, and a .ply mesh. Every one of them needs its own parser before any work can start.

It is worth being clear about why this matters. In robot learning, gains come disproportionately from iterating on the data: HuggingFace's robot folding project found that curating 1,200 episodes from a pool of 5,688 moved success rates by 50 percentage points, while algorithmic improvements moved them by 5 to 20. Having a unified data format at every step accelerates the cycle from data to curation to training.

1. Images are simple but expensive

Extracted frames are easy to load and address by index. They are also enormous. A single 1920×1440 frame from this dataset averages 2.3 MB as a PNG, which puts one second of the wide camera at 60 fps at about 140 MB, and one minute at roughly 8 GB.

The wide camera alone comes to roughly 29 million frames across all the scans. Written out as PNG files that is about 66 TB. Stored as encoded video the same frames take 957 GB, a factor of roughly 70, and that is one camera of the two before any depth or IMU.

2. Video is compact but awkward

Video solves the storage problem and creates an access problem. Frames are no longer independent, so reading one in the middle means decoding from the nearest keyframe forward. That cost shows up every time visualization, inference, or analysis wants a frame on demand.

3. The sensors run at different rates

Nothing shares a clock. The wide camera runs at about 60 fps, the ultrawide at about 10, the IMU at about 100 Hz, depth at its own rate, and the mesh is static.

These streams have to share a coherent timeline even though most timestamps in one stream have no exact counterpart in another.

4. Every downstream task wants a different shape

The formats above are only the input side. The consumers disagree too: visualization wants one thing, SQL-style analysis another, and a PyTorch dataloader a third. Without a shared representation, every combination of format and consumer is its own piece of glue code.

5. Dataset quality is hard to judge from files alone

Before training or running inference, the questions worth asking are things like:

• Which trajectories move fast enough to suggest tracking failure?
• Which captures rotate aggressively?
• Which switch between portrait and landscape?
• Which are missing a modality, or have gaps in their timestamps?

Those answers need to lead back to synchronized visual evidence. A row in a table tells you a sequence is suspect; it does not let you watch the moment it went wrong.

Ingesting into rrd

Every one of those five is a data layout problem. A unified representation solves all of these issues, and that's exactly what the RRD format does. Convert each scan once into a Rerun recording, and let everything downstream read that.

One file, four consumers

A recording is a single .rrd holding every stream on one shared timeline. Four different things read it, and none of them needs an export step first:

• Visualization. The viewer opens the recording directly.
• Querying. The dataframe and SQL APIs run over the same file.
• Post-processing. Running a model over the corpus is inference, so it pulls batches through the PyTorch dataloader and writes its predictions back.
• Training. The same dataloader over the same recordings, with no export step in between.

That is the fourth problem gone. There is no conversion per task because there is no format per task.

Video stays video

The first two problems pull against each other. Extracted frames are addressable but huge, and video is compact but hard to seek into. Storing the encoded video as columns gets both at once: sample payloads and keyframe flags are written as columns on the timeline, and the viewer and the dataloader decode on demand and handle the keyframe seeking themselves. The 70× saving stays, and random access comes back.

video = rr.VideoStream(codec=rr.VideoCodec.AV1)
recording.log(path, video, static=True)

pts = samples.timestamps - epoch
rr.send_columns(
    path,
    indexes=[rr.TimeColumn("video_time", duration=pts)],
    columns=rr.VideoStream.columns(
        sample=samples.payloads,
        is_keyframe=samples.is_keyframes,
    ),
)

Every stream keeps its own clock

Nothing is resampled onto a common grid. Each stream is written against its own native timestamps, so the IMU stays at 100 Hz and the ultrawide stays at 10 fps. Alignment becomes something a query decides, rather than something an export decided once and for everyone.

Read the recording as it was written and you get exactly the rows that exist, which means most columns are null at any given timestamp. Ask for a grid instead and the reader resamples onto it, carrying each stream forward from its last real sample:

# the rows that actually exist, at their own timestamps
rows = dataset.reader(index="video_time")

# one row per requested timestamp, each column filled forward
aligned = dataset.reader(
    index="video_time",
    using_index_values=grid_ns,
    fill_latest_at=True,
)

Recordings become a dataset

Converting gives you files. Registering them gives you something you can ask questions of. In the catalog object model, a dataset is a collection of segments, one per recording, and each segment is built from layers that stack under the same segment id. The seven files a scan converts into are seven layers of one segment, so calibration and video and depth arrive as one thing rather than as files you have to pair up yourself.

Registering all 5,015 scans is what turns a directory into a corpus. That is 5,015 of the 5,047 published: 24 scans ship without a mesh, annotations, or a trajectory, and 8 more failed the pipeline's transcode-integrity checks and were excluded rather than ingested broken. It is also the entry point for everything that follows: the viewer opens a segment from the catalog, the reader queries across all of them at once, and the dataloader is handed a dataset rather than a list of paths.

Every scan becomes a row you can filter

The judgments made during conversion get written into the recording as properties, which show up as columns in the catalog's segment table, one row per scan. The questions from the fifth problem turn into filters over that table:

• fast translation, from the pose columns;
• aggressive rotation, from the same;
• orientation changes, from the orientation measured at ingest;
• missing modalities, from which layers a segment actually has.

Every matching row opens in the viewer at the moment it describes, which is the part a table on its own cannot do.

The same reader answers questions the segment table does not already have a column for. Pulling every camera position out of the corpus is one query, and total distance travelled falls out of it:

TRANSLATION = "/world/rig_00:Transform3D:translation"

client = rr.catalog.CatalogClient("rerun+http://127.0.0.1:51235")
dataset = client.get_dataset("arkitscenes")

poses = (
    dataset.filter_contents(["world/rig_00"])
    .reader(index="video_time")
    .filter(col(TRANSLATION).is_not_null())
    .select("rerun_segment_id", "video_time", TRANSLATION)
    .to_arrow_table()
)

Predictions land beside the originals

Because a recording is a stack of layers, a model's output can be added as one more layer next to the streams it was computed from. Running PromptDA over the corpus writes high resolution depth alongside the LiDAR depth it was prompted with, rather than producing a fourth copy of the dataset.

The model needs decoded frames and the LiDAR depth that prompts it, which is what the dataloader is asked for. Video packets are decoded on the way out, so nothing had to be materialized as images first:

source = DataSource(dataset=dataset, segments=[segment_id])

samples = RerunMapDataset(
    source,
    index="video_time",
    fields={
        "video": Field(VIDEO_WIDE, decode=VideoFrameDecoder("av1")),
        "depth": Field(DEPTH_LOWRES, decode=ImageDecoder()),
    },
    timeline_sampling=FixedRateSampling(rate_hz=60.0),
)

for batch in DataLoader(samples, batch_sampler=batches):
    predictions = predictor(batch["video"], batch["depth"])

The predictions go back as a promptda layer under the same segment id, next to the depth that prompted them. Nothing else about the scan is copied or rewritten.

A baseline to build on

All 5,015 scans now live in one format: about 1.9 TB of recordings holding both cameras, the LiDAR and laser depth, the IMU, and the ground truth. I can open any of them in the viewer, query across the whole corpus, and feed batches straight to a model, all from the same recordings.

That is also why depth upsampling was the right place to start. The model needs poses, images, and low-resolution depth showing up together at the right timestamps, and with the data layer handling that, inference came down to a dataloader and a batch loop.

In part 2, I'll take on the pieces the iPhone has been providing for free: visual-inertial SLAM for the poses, then depth prediction to stand in for the LiDAR. Each one becomes another layer on the same scans, sitting right next to ARKit's answer and the laser ground truth, so I always have something to be wrong against.

The room reconstruction is just a proxy for what I actually want. The pipeline that turns these scans into one queryable corpus is the same pipeline that turns a head-mounted capture of someone doing the dishes into robot training data: poses, depth, and geometry attached to every frame of video. ARKitScenes is a great proxy because it provides the same type of data that applies to robot training, realtime slam and depth estimation.

Until then, the full corpus is public at rerun/arkitscenes-rrd if you want to explore it yourself.

References

• apple/ARKitScenes: the dataset.
• rerun/arkitscenes-rrd: the converted corpus.
• arkitscenes-download: the conversion pipeline.
• prompt-da: the depth upsampling run.