Summary (Overview)

• EgoXperience is a full-stack paradigm for learning world-grounded egocentric representations, comprising data collection, automatic annotation, unified modeling, and benchmarking.
• The authors introduce EgoX-Engine, an automatic annotation pipeline that couples geometrically verified object tracking with SLAM-based world alignment to produce consistent supervision for cameras, scenes, hands, and objects.
• They release EgoX-Data-6M, a 50-hour dataset with 5.4M frames, 270 task scripts, and 101 objects, spanning workspace manipulation, cross-room interactions, and outdoor loco-manipulation, plus a held-out EgoX-Bench for joint evaluation.
• The proposed model EgoX jointly reconstructs camera, scene depth, hands, and objects at metric scale in a shared world coordinate frame from monocular egocentric video in a single forward pass.
• EgoX achieves state-of-the-art performance, reducing camera, hand, and depth errors by 68.9%, 65.6%, and 42.9% respectively, and improving object pose scores by 28.0% against the strongest baselines.

Introduction and Theoretical Foundation

Egocentric video provides a natural perspective on human-environment interaction, requiring only a wearable camera. However, existing methods typically recover only a subset of tightly coupled components (camera pose, scene depth, hands, objects), and combining outputs from different methods requires reconciling inconsistent scales and coordinate frames. This motivates a unified feed-forward framework that jointly recovers all components within a single model.

Training such a unified model requires large-scale egocentric data with joint annotations. Existing datasets often rely on calibrated multi-camera or motion-capture systems, making scaling difficult across diverse real-world environments. The key challenge is that the observer and interacting entities move simultaneously—a change in image position may arise from camera motion, entity motion, or both—creating geometric ambiguity that must be resolved.

The theoretical foundation builds on recent advances in:

• Scene and entity reconstruction: DUSt3R, VGGT, CUT3R, and dynamic-scene methods that recover camera and geometry
• Hand-object reconstruction: HaMeR, WiLoR, HaWoR, Hand3R, MINT, and ReViV for hand pose and interaction modeling
• Egocentric datasets: HOI4D, HOT3D, SHOW3D, and ACE-Data-0 providing various annotation coverage

Methodology

EgoX-Engine (Annotation Pipeline)

The annotation engine addresses the ambiguity of simultaneous observer-entity motion by:

1. Capture and geometric reference: Records with the HOMIE Gen2 head-mounted rig, estimating metric depth from synchronized stereo pairs and camera trajectories via SLAM from four calibrated fisheye cameras
2. Object geometry: Constructs metric object meshes from 31 synchronized dome cameras, using SAM 3 for mask extraction, MV-SAM3D for mesh reconstruction, and AprilTag boards for scale reference
3. Object motion: Uses FoundationPose for tracking with geometric verification against observed depth, transforming poses through SLAM-estimated camera poses into the session frame
4. Scene and hand annotations: Back-projects stereo depth for scene geometry, recovers 3D hand keypoints via cross-view triangulation, and constrains MANO fitting with reprojection, depth, and temporal terms

EgoX-Data-6M

• 50 hours of task-driven recordings with 101 rigid objects and 270 task scripts
• Covers workspace manipulation, cross-room interactions, and outdoor locomotion
• Provides geometrically aligned camera, scene, hand, and object annotations in a session-fixed metric frame
• Includes second-level task captions
• EgoX-Bench: quality-controlled one-hour held-out subset for joint evaluation

EgoX Model

Given monocular observations $\boldsymbol{\tau} = \{I_t\}_{t=1}^{T}$ and optional first-frame object masks, EgoX jointly predicts camera intrinsics/poses, dense depth, two-hand MANO states, object origins/orientations, and a positive metric scale in a single forward pass.

Key architectural components:

• Joint scene and entity representation: Shared Transformer with DINOv3 patch features, camera/hand/object tokens with distinct left/right embeddings
• Reconstruction in shared metric frame: The mapping from image position $\tilde{\mathbf{u}}$ with normalized depth $z$ to world coordinates $\mathcal{W}$:

$$\boldsymbol{\mathcal{U}}_t(\tilde{u}, z) = \hat{R}_t^{\top}\left(z \hat{K}_t^{-1} \tilde{u} - \hat{\tau}_t\right).\tag{1}$$

• Metric scale: Shared ScaleHead predicts one positive factor $\hat{s}$ for the sequence, scaling scene points and camera, wrist, and object positions consistently

Training objective:

$$\mathcal{L} = \mathcal{L}_{\text{local}} + \mathcal{L}_{\text{dense}} + \mathcal{L}_{\text{coupled}} + \mathcal{L}_{\text{scale}},\tag{2}$$

where $\mathcal{L}_{\text{local}}$ supervises camera and view-local entity predictions, $\mathcal{L}_{\text{dense}}$ supervises depth and scene points, $\mathcal{L}_{\text{coupled}}$ supervises the composed hand-object geometry (backpropagating through camera transformations), and $\mathcal{L}_{\text{scale}}$ supervises metric scale. Losses are applied only where labels are available, accommodating partially annotated public datasets.

Empirical Validation / Results

Hand Reconstruction (Table 3)

EgoX achieves the lowest PA-, WA-, and W-MPJPE on all four benchmarks:

Object 6-DoF Pose (Table 4)

EgoX achieves the best ADD, ADD-S, and AUC-S on all benchmarks. On EgoX-Bench, AUC-S is 76.59 vs. 59.80 for GigaPose + MegaPose (which additionally uses metric CAD assets and per-frame masks).

Camera and Depth Reconstruction (Table 5)

Ablation Studies

ScaleHead ablation (Table 6): Removing ScaleHead with oracle scale increases W-MPJPE from 38.89 to 40.67 mm and decreases AUC-S from 73.71 to 60.87, showing the head benefits beyond final rescaling.

DepthHead ablation (Table 7): Removing DepthHead increases W-MPJPE to 67.53 mm and decreases AUC-S to 41.09, showing explicit depth prediction improves hand/object reconstruction through the shared representation.

Theoretical and Practical Implications

Theoretical contributions:

• Demonstrates that geometrically coupled supervision—transforming hand/object predictions through estimated cameras and supervising the composed geometry—ties component estimation together, encouraging consistency
• Shows that explicit depth prediction improves hand and object reconstruction through shared representations, validating the joint modeling approach
• Establishes that scale prediction integrated into training (rather than post-hoc rescaling) improves reconstruction quality

Practical implications:

• EgoX-Data-6M provides a scalable alternative to mocap-based annotation, reducing infrastructure requirements for egocentric data collection
• The full-stack paradigm (capture → annotation → modeling → benchmark) can be adapted to other embodied AI domains
• EgoX's single-pass joint reconstruction enables downstream applications in robotics, AR/VR, and embodied AI without reconciling inconsistent outputs from separate methods
• The model requires only RGB video and first-frame object masks, reducing input requirements compared to baselines needing CAD models, per-frame masks, or estimated depth

Conclusion

EgoXperience presents a unified approach to learning world-grounded egocentric representations through task-driven capture, geometrically aligned annotation, and joint reconstruction. EgoX-Engine provides aligned multimodal annotations for the 50-hour EgoX-Data-6M, from which a held-out one-hour EgoX-Bench is curated. Trained on these annotations and harmonized public datasets, EgoX jointly reconstructs cameras, dense depth, hands, and rigid-object 6-DoF poses in a common metric world frame, achieving strong performance across all benchmarks.

Limitations and future directions:

• Current formulation assumes rigid objects and does not predict object meshes
• Future work will explore object mesh prediction and multiview inputs
• The paradigm could extend to articulated objects and more complex interaction scenarios