Summary (Overview)

• HOLI-WORLD is an integrated ecosystem that automatically converts human egocentric recordings into grounded, multi-modal learning resources for building world intelligence, connecting experience capture, world representation, automated data construction, model learning, and independent evaluation.
• HOLIWORLD-Foundry combines complementary static and dynamic perception pipelines: static objects are detected via dual-path (image + point-cloud) 3D detection with VLM validation, while dynamic objects are tracked via identity-guided mask propagation and 3D trajectory recovery in shared world coordinates.
• HOLIWORLD-Data organizes records from ~100 self-collected scenes and 444 egocentric videos into ~80K evidence-driven QA pairs, covering spatial relations, temporal changes, and interactions, with each answer traceable to supporting observations.
• HOLIWORLD-VL-4B, trained on this data, improves Token F1 from 16.35% to 78.02% on HOLIWORLD-Bench and from 18.29% to 66.39% on EgoDynamic4D, surpassing even a zero-shot 27B-parameter model.
• HOLIWORLD-Foundry achieves state-of-the-art results in static perception (AP₂₅ = 0.68 on ScanNet vs. 0.40 for strongest baseline) and dynamic tracking (HOTA₀.₁ = 54.31% on Aria Digital Twin vs. 36.00% for IT3DEgo).

---

Introduction and Theoretical Foundation

The paper addresses a fundamental challenge in AI: building a coherent understanding of the physical world across viewpoints and time. While vision-language models and predictive video models can recognize objects and describe scenes, they lack an integrated understanding of how environments, objects, and interactions form a shared, evolving world.

Key motivation: Human egocentric experience naturally brings together spatial structure (moving through an environment) and temporal dynamics (interacting with objects). However, scaling this into learning supervision traditionally requires costly manual annotation.

Theoretical basis: The authors argue that recent advances in complementary capabilities can be combined:

• Static scene reconstruction (e.g., Holi-Spatial, Boxer) for spatial supervision
• Egocentric object tracking (e.g., IT3DEgo, LMK) for temporal identity maintenance

The core insight is that connecting these capabilities through shared world coordinates and persistent identities enables the automatic construction of coherent learning resources that remain traceable to underlying observations.

Research gap: Existing egocentric datasets (EPIC Fields, Aria Digital Twin, HD-EPIC) provide rich annotations but require extensive manual effort, limiting scalability. HOLIWORLD addresses this by automating the annotation pipeline.

---

Methodology

3.1 HoliWorld Overview

The ecosystem consists of five components:

1. Eagle Capture Device: Records synchronized stereo RGB and IMU data in-the-wild
2. HoliWorld-Foundry: Reconstructs scenes and annotates static/dynamic objects
3. HoliWorld-Data: Organizes outputs into grounding, captioning, and spatial QA supervision
4. HoliWorld-VL: Vision-language model trained on the generated supervision
5. HoliWorld-Bench: Independent human annotations for evaluation

3.2 HoliWorld-Foundry

Geometric Preparation: Camera poses and sparse points are recovered via visual-inertial SLAM; depth is estimated with Depth Anything 3 (DA3); a 3D Gaussian scene representation $\mathcal{G}$ is optimized. Observations are organized as:

$$\mathcal {O} = \left\{\left(I _ {t}, D _ {t}, \mathbf {K} _ {t}, \mathbf {T} _ {t}\right) \right\} _ {t = 1} ^ {T},\tag{1}$$

where $I_t$ is an RGB frame, $D_t$ is a depth map, $\mathbf{K}_t$ is the camera intrinsic matrix, and $\mathbf{T}_t$ maps camera to world coordinates.

Static Object Perception:

• Dual-path 3D detection: Boxer on stereo RGB streams (image-based) + TR3D on the scene point cloud (point-cloud-based)
• Geometric merging/filtering: Low-cost geometric checks merge spatially compatible boxes and discard unsupported ones
• VLM validation: A VLM checks each box against source observations and rendered views, with one fallback retry per box

Dynamic Object Perception:

• Interaction grounding: VLM identifies manipulated objects across interaction segments, establishing consistent identities
• Identity-guided mask tracking: SAM selects and propagates target masks within segments
• 3D recovery: Back-projection of tracked mask pixels:

$$\mathcal {X} _ {t} = \left\{\mathbf {T} _ {t} \left(D _ {t} (\mathbf {u}) \mathbf {K} _ {t} ^ {- 1} \bar {\mathbf {u}}\right) \Big | \mathbf {u} \in m _ {t}, D _ {t} (\mathbf {u}) \text {is valid} \right\},\tag{2}$$

Unified World Representation:

$$\mathcal {S} (t) = \big (\mathcal {O} _ {t}, \mathcal {U} (t), \mathcal {G}, \mathcal {P}, \mathcal {B} ^ {\mathrm{sta}}, \mathcal {B} ^ {\mathrm{dyn}} (t), \mathcal {Z} (t) \big),\tag{3}$$

3.3 HoliWorld-Data

• QA pairs: Following EgoDynamic4D patterns, covering spatial (direction, distance, containment), temporal (motion, event ordering), and interaction (manipulation, state changes) reasoning
• Dataset composition: ~100 scenes, 444 videos, 1,322K RGB frames, 90K 2D boxes, 2,466 static 3D boxes, 2,095 dynamic tracks, ~80K QA pairs (13,647 static + 64,409 dynamic)

---

Empirical Validation / Results

Static 3D Object Perception

Table 1 shows HOLIWORLD-FOUNDRY achieves the highest AP₂₅ and AP₅₀ across all three datasets:

Dynamic 3D Object Perception and Tracking

Table 2 shows HOLIWORLD-FOUNDRY attains HOTA scores of 54.31% on ADT and 43.14% on HoliWorld-Bench, versus 36.00% and 15.13% for the strongest baselines.

Downstream Spatial and Temporal Understanding

Table 3 shows fine-tuning Qwen3.5-4B on HoliWorld-Data yields dramatic improvements:

Ablation Studies

• Static components: Image-based lifting captures more small objects; point-cloud detection recovers larger extents; stereo fusion combines both views; caption guidance recovers missed entities; VLM verification corrects labels and rejects unsuitable candidates
• Dynamic components: Cross-interaction label continuity reduces identity fragmentation; SAM association provides temporally consistent masks, reducing trajectory deviations

---

Theoretical and Practical Implications

Theoretical contributions:

• Demonstrates that human egocentric experience can serve as a scalable foundation for learning world intelligence without manual annotation
• Establishes that shared world coordinates and persistent identities are sufficient to unify static scene structure with dynamic object states
• Shows that automatically constructed supervision can outperform much larger models (27B) trained with standard methods, suggesting data quality and grounding matter more than model scale for spatial reasoning

Practical implications:

• Provides a complete, open-source blueprint (code, models, datasets) for building world intelligence from egocentric recordings
• Reduces the annotation bottleneck that limits scalability in egocentric learning
• Enables continuous expansion of learning resources: new recordings automatically generate new supervision
• The HOLIWORLD-Bench provides an independent evaluation standard for both annotation quality and learned spatial understanding

---

Conclusion

HOLIWORLD presents an integrated ecosystem that places human egocentric experience at the center of learning world intelligence. Key takeaways:

1. Automated annotation at scale is feasible: Complementary static and dynamic perception pipelines can automatically construct world records linking scene geometry, object semantics, and interaction histories.

2. Grounded supervision improves reasoning: Training on HOLIWORLD-Data substantially improves spatial and temporal reasoning, with the 4B model surpassing a 27B zero-shot model.

3. Practical foundation for future work: The open-source ecosystem (code, models, datasets) provides a blueprint for expanding learning resources with new recordings.

Future directions implied by the work include:

• Relaxing the current assumption of at most one moving object per time step
• Extending to more diverse environments and interaction types
• Scaling to larger collections of egocentric recordings from broader populations
• Further integration of static and dynamic perception for even richer world representations