Image and video understanding

EgoExo forge

A collection of datasets and tools for egocentric and exocentric human activity understanding, featuring hand-object interactions, manipulation tasks, and multi-view recordings.

Background

EgoExo Forge provides a consistent labeling scheme and data layout across multiple egocentric and exocentric human datasets with varying sensor configurations and annotations.

The following datasets are supported:

• Assembly101: A procedural activity dataset with 4321 multi-view videos of people assembling and disassembling 101 take-apart toy vehicles, featuring rich variations in action ordering, mistakes, and corrections.
• HO-Cap: A dataset for 3D reconstruction and pose tracking of hands and objects in videos, featuring humans interacting with objects for various tasks including pick-and-place actions and handovers.
• EgoDex: The largest and most diverse dataset of dexterous human manipulation with 829 hours of egocentric video and paired 3D hand tracking, covering 194 different tabletop tasks with everyday household objects.

Run the code

This is an external example. Check the repository for more information.

You can try the example on a HuggingFace space here.

Or locally, make sure you have the Pixi package manager installed and run

git clone https://github.com/rerun-io/egoexo-forge.git
cd egoexo-forge
pixi run app