Summary (Overview)

• ACE-Data-0 is a large-scale, long-horizon human-object interaction (HOI) dataset comprising 150 hours, 17M video frames, 75,000 interaction episodes, 200 task categories, 50 participants, and 2 environments, captured with synchronized multi-modal sensing.
• The paper introduces the Ambient Capture Engine (ACE), a capture system that transforms real home environments into spatially calibrated, temporally synchronized recording studios operating at two complementary scales: table-scale (fine-grained hand-object manipulation) and room-scale (whole-body motion and scene-level interactions).
• ACE records egocentric video, multi-view exocentric video, full-body and articulated hand motion, per-object 6-DoF trajectories, multi-channel audio, and tactile signals — all registered into a common spatio-temporal frame with metric ground truth.
• A three-level hierarchical benchmark is established, progressing from low-level signals (tactile-from-vision), to scene components (human motion estimation), to embodied interactions (hand-object interaction from ego/exo views), with evaluations of 30+ state-of-the-art methods exposing substantial gaps under contact, occlusion, egomotion, and long temporal horizons.
• The dataset provides rich annotations including camera calibrations, full-body and hand poses, per-object mesh models with 6-DoF poses and bounding boxes, motion trails, tactile pressure maps, and language descriptions — with most annotations measured rather than estimated from tracked physical states.

---

Introduction and Theoretical Foundation

Motivation

The paper addresses a fundamental data bottleneck in embodied intelligence. While language and vision models benefited from centuries of accumulated human archives, physical skills have never been written down — how a hand closes around a cup, with what force a fragile glass is held, or by what coordination of vision and balance an object is carried across a room. The data for embodied intelligence must therefore be built rather than found by instrumenting everyday life and recording human-object interaction (HOI) as it naturally unfolds.

Limitations of Existing Datasets

The authors identify three critical shortcomings in existing datasets:

1. Fragmented modalities: Large-scale egocentric datasets (Ego-Exo4D, EPIC-Kitchens, Xperience-10M) offer naturalistic behavior but lack ground-truth body/object motion and synchronized third-person observations. Motion-captured HOI datasets (BEHAVE, GRAB, ARCTIC, OakInk2, HOT3D) provide accurate poses but omit egocentric perspectives. Audio and tactile sensing are absent from nearly all.

2. Unnatural environments: Physically annotated datasets are captured almost exclusively in laboratories, whose sparse layouts eliminate the occlusions, spatial constraints, and object diversity of real homes.

3. Short horizons: Most HOI clips span seconds and depict one simple movement. Genuine household activities are goal-directed, unfolding over minutes or hours, chaining sub-tasks and multiple objects, requiring movement across the scene.

Theoretical Foundation

The paper argues that a complete perception-action loop requires synchronized multisensory grounding — all signals (visual, kinematic, acoustic, tactile) must describe the same physical event on a common timeline and in a common spatial frame. This unified grounding supports learning from imitation, world models, vision-language-action systems, and embodied AI.

---

Methodology

The Ambient Capture Engine (ACE)

ACE is designed with two complementary configurations addressing conflicting sensor placement requirements:

Table-Scale Configuration

• Environment: 30 m² workspace with a work desk, over 25 interactable object instances from 8+ categories
• Sensors: 8 close-range GoPro RGB cameras (0.3–0.5 m distance), 16 OptiTrack cameras for optical motion capture
• Focus: Fine-grained dexterous hand-object manipulation at sub-millimeter effective resolution

Room-Scale Configuration

• Environment: Fully furnished apartment (~200 m²) with kitchen, dining area, living room, and bedroom
• Sensors: 8 wide-baseline ZED One RGB cameras (via GMSL2 to a single Jetson Orin host with shared frame trigger), 12 OptiTrack cameras mounted on a ceiling truss
• Focus: Whole-body motion, locomotion, and interactions distributed across the scene

Sensor Suite

Synchronization

The OptiTrack clock serves as the reference timeline. Key techniques include:

• Exocentric cameras (room-scale): ZED One cameras share a common frame trigger via GMSL2. Alignment to OptiTrack uses a QR-code clock displayed on the motion-capture host monitor, read off recorded frames to correct a ~0.29 s timestamp lag with slow drift. Final residuals are at millisecond level.
• Egocentric cameras: Each take begins with a deliberate "clock glance" where the wearer points a camera at the monitor for ~10 seconds, establishing the offset between the egocentric camera clock and OptiTrack clock.
• Tactile gloves: Synchronized via onboard IMU signals — the operator performs a short motion pattern at the start of each take, correlating glove and headset IMU streams.

Calibration

• Exocentric cameras: Use an ArUco board with retroreflective markers at each corner, visible to both RGB and infrared cameras simultaneously. This bridges cameras that share no common field of view. Final median reprojection error: below 3 px (~1 cm 3D error).
• Egocentric cameras: Five markers on the headset chassis form a rigid body tracked at 60 Hz. The fixed transformation from each fisheye camera to this body is solved via hand-eye calibration with joint bundle adjustment. Final median reprojection error: ~2 px. Poses are measured, not estimated, and do not drift.

Task Design

Three types of tasks are captured:

1. Atomic HOI tasks (36.5% of data): 1–3 household tasks per take (~3 minutes each), including pouring water, drinking, making tea, chopping vegetables, cooking, tidying up
2. Chains of HOI tasks (42.9%): Full range of short tasks combined into continuous activities of ~20–30 minutes, with freely interleaving sub-tasks
3. HSI tasks (20.6%): Human-scene interactions with minimal objects — walking, exercising, sitting, lying, leaning (~5 minutes per take)

Participants receive goal-level instructions rather than step-by-step scripts, allowing natural behavioral variation in object choice, task ordering, movement paths, hesitation, and recovery.

Annotation Pipeline

Five annotation types are provided:

1. Object annotations: Category labels, bounding boxes, per-frame 6-DoF poses, motion trails, scanned/2DGS-reconstructed meshes for 50+ instances
2. Human poses: Full body pose (41 joints) with SMPL-X parameters, reprojected onto every frame of every camera
3. Hand annotations: Articulated finger configurations during dexterous manipulation
4. Tactile annotations: Full-hand grasp pressure values resolving interaction events ambiguous under visual occlusion
5. Audio and language descriptions: Natural-language descriptions generated by Gemini-3.1-pro-preview from ego-view videos, paired with synchronized multi-channel audio

Critically, all annotations except textual descriptions are measured rather than estimated — derived from tracked physical states via projection and tactile sensing, with no estimation model in the loop.

---

Empirical Validation / Results

Benchmark Track 1: Tactile from Vision

Task: Predict full-hand grasp pressure from egocentric video of HOI.

Metrics: Temporal accuracy, Contact IoU (C-IoU), Volumetric IoU (V-IoU), Center-of-Pressure (CoP) error.

Results:

Key findings: TouchAnything achieves the strongest performance, but absolute C-IoU and V-IoU remain modest. Detecting when contact occurs is easier than recovering where pressure is distributed. Accurate pressure localization under hand-object occlusion remains a significant open challenge.

Benchmark Track 2: Human Motion Estimation

Task: Estimate articulated body pose from visual observations across three input settings (multi-view exocentric, single-view exocentric, egocentric) and three method families (per-frame, temporal, scene-aware).

Metrics: PA-MPJPE, PA-PVE, MPJPE, PVE, WA-MPJPE (all in mm).

Selected results (single-view exocentric, temporal):

Selected results (egocentric):

Key findings: Egocentric methods perform significantly worse than exocentric methods, with errors roughly 2–3× larger. Global trajectory errors (WA-MPJPE) are particularly large (180–306 mm), revealing that long-horizon drift remains a major unsolved problem.

Benchmark Track 3: Hand-Object Interaction

The benchmark evaluates hand motion reconstruction during HOI from both egocentric and exocentric perspectives, with cross-view analysis enabled by the paired viewpoint design. (Detailed results are presented in the full paper.)

---

Theoretical and Practical Implications

For Embodied AI Research

1. Unified multisensory grounding: ACE-Data-0 provides the first dataset where egocentric video, multi-view exocentric video, motion trajectories, and contact-level supervision are synchronized rather than assembled from disparate sources — every training signal refers to the same physical moment, essential for imitation learning, policy learning, and world modeling.

2. Natural behavioral diversity: Goal-level instructions preserve planning, hesitation, and improvisation, capturing the full perception-action loop of everyday interaction rather than scripted atomic actions.

3. Measured rather than estimated ground truth: The annotation pipeline derives labels from tracked physical states, avoiding the circularity of pseudo-labeling where datasets are annotated by the very methods under evaluation.

For Benchmarking

The three-level benchmark hierarchy (signals → components → interactions) mirrors the perceptual capabilities an embodied agent must chain together: sensing contact, estimating scene state, and mastering hand-object coordination. Results reveal substantial generalization gaps:

• Tactile estimation from vision remains far from usable accuracy (best C-IoU: 0.165)
• Human motion estimation degrades severely under egocentric viewpoints and long temporal horizons (WA-MPJPE errors of 180–306 mm)
• Hand-object interaction under occlusion and egomotion remains challenging

For Robot Learning

ACE-Data-0 bridges geometric HOI benchmarks, long-form egocentric video, and robot trajectory corpora. The synchronized human demonstrations with aligned perceptual, kinematic, and contact supervision offer a scalable foundation for:

• Imitation learning with natural human strategies
• Vision-language-action (VLA) systems
• World models with physical grounding
• Human-to-robot transfer methods (EgoVLA, UniVLA, EgoBridge)

---

Conclusion

ACE-Data-0 addresses the fundamental data bottleneck in embodied intelligence by providing a large-scale, long-horizon, home-scene HOI dataset with synchronized multi-modal sensing and metric ground truth. The key contributions are:

1. ACE capture system: A "human-centric ambient capture as embodied data engine paradigm" with table-scale and room-scale configurations recording temporally and spatially aligned egocentric video, multi-view exocentric video, human body and hand motion, object motion, audio, and tactile signals.

2. ACE-Data-0 dataset: 150 hours, 17M frames, 75,000 interaction episodes across 200 task categories, with rich measured annotations.

3. Three-level benchmark: Evaluations of 30+ state-of-the-art methods exposing open challenges for embodied perception and robot learning, particularly under contact, occlusion, egomotion, and long temporal horizons.

Future Directions

• Object pose estimation is left to future benchmarking work (too few applicable methods currently exist)
• The released ground truth directly supports this evaluation
• The dataset provides a foundation for scaling toward even larger and more diverse embodied AI training corpora
• The ACE paradigm can be extended to additional environments and interaction types beyond the two current sites

The paper concludes that robust embodied perception requires not only recognizing contact events and estimating scene states, but also resolving fine-grained interaction dynamics from visually ambiguous and frequently occluded observations — challenges that ACE-Data-0 is specifically designed to expose and address.