# Holi-World: Building Holistic World Intelligence from Human-Ego Experience

> HOLIWORLD automatically converts egocentric video into grounded spatial-temporal QA data, enabling a 4B model to surpass a 27B model on world understanding benchmarks.

- **Source:** [arXiv](https://arxiv.org/abs/2610.26549)
- **Published:** 2026-10-21
- **Permalink:** https://picx.dev/p/lcvQQp
- **Whiteboard:** https://picx.dev/p/lcvQQp/image

## Summary

## Summary (Overview)

- **HOLI-WORLD** is an integrated ecosystem that automatically converts human egocentric recordings into grounded, multi-modal learning resources for building world intelligence, connecting experience capture, world representation, automated data construction, model learning, and independent evaluation.
- **HOLIWORLD-Foundry** combines complementary static and dynamic perception pipelines: static objects are detected via dual-path (image + point-cloud) 3D detection with VLM validation, while dynamic objects are tracked via identity-guided mask propagation and 3D trajectory recovery in shared world coordinates.
- **HOLIWORLD-Data** organizes records from ~100 self-collected scenes and 444 egocentric videos into ~80K evidence-driven QA pairs, covering spatial relations, temporal changes, and interactions, with each answer traceable to supporting observations.
- **HOLIWORLD-VL-4B**, trained on this data, improves Token F1 from 16.35% to 78.02% on HOLIWORLD-Bench and from 18.29% to 66.39% on EgoDynamic4D, surpassing even a zero-shot 27B-parameter model.
- **HOLIWORLD-Foundry** achieves state-of-the-art results in static perception (AP₂₅ = 0.68 on ScanNet vs. 0.40 for strongest baseline) and dynamic tracking (HOTA₀.₁ = 54.31% on Aria Digital Twin vs. 36.00% for IT3DEgo).

---

## Introduction and Theoretical Foundation

The paper addresses a fundamental challenge in AI: building a coherent understanding of the physical world across viewpoints and time. While vision-language models and predictive video models can recognize objects and describe scenes, they lack an integrated understanding of how environments, objects, and interactions form a shared, evolving world.

**Key motivation:** Human egocentric experience naturally brings together spatial structure (moving through an environment) and temporal dynamics (interacting with objects). However, scaling this into learning supervision traditionally requires costly manual annotation.

**Theoretical basis:** The authors argue that recent advances in complementary capabilities can be combined:
- **Static scene reconstruction** (e.g., Holi-Spatial, Boxer) for spatial supervision
- **Egocentric object tracking** (e.g., IT3DEgo, LMK) for temporal identity maintenance

The core insight is that connecting these capabilities through **shared world coordinates** and **persistent identities** enables the automatic construction of coherent learning resources that remain traceable to underlying observations.

**Research gap:** Existing egocentric datasets (EPIC Fields, Aria Digital Twin, HD-EPIC) provide rich annotations but require extensive manual effort, limiting scalability. HOLIWORLD addresses this by automating the annotation pipeline.

---

## Methodology

### 3.1 HoliWorld Overview

The ecosystem consists of five components:
1. **Eagle Capture Device**: Records synchronized stereo RGB and IMU data in-the-wild
2. **HoliWorld-Foundry**: Reconstructs scenes and annotates static/dynamic objects
3. **HoliWorld-Data**: Organizes outputs into grounding, captioning, and spatial QA supervision
4. **HoliWorld-VL**: Vision-language model trained on the generated supervision
5. **HoliWorld-Bench**: Independent human annotations for evaluation

### 3.2 HoliWorld-Foundry

**Geometric Preparation:** Camera poses and sparse points are recovered via visual-inertial SLAM; depth is estimated with Depth Anything 3 (DA3); a 3D Gaussian scene representation $\mathcal{G}$ is optimized. Observations are organized as:

$$
\mathcal {O} = \left\{\left(I _ {t}, D _ {t}, \mathbf {K} _ {t}, \mathbf {T} _ {t}\right) \right\} _ {t = 1} ^ {T},\tag{1}
$$

where $I_t$ is an RGB frame, $D_t$ is a depth map, $\mathbf{K}_t$ is the camera intrinsic matrix, and $\mathbf{T}_t$ maps camera to world coordinates.

**Static Object Perception:**
- **Dual-path 3D detection**: Boxer on stereo RGB streams (image-based) + TR3D on the scene point cloud (point-cloud-based)
- **Geometric merging/filtering**: Low-cost geometric checks merge spatially compatible boxes and discard unsupported ones
- **VLM validation**: A VLM checks each box against source observations and rendered views, with one fallback retry per box

**Dynamic Object Perception:**
- **Interaction grounding**: VLM identifies manipulated objects across interaction segments, establishing consistent identities
- **Identity-guided mask tracking**: SAM selects and propagates target masks within segments
- **3D recovery**: Back-projection of tracked mask pixels:

$$
\mathcal {X} _ {t} = \left\{\mathbf {T} _ {t} \left(D _ {t} (\mathbf {u}) \mathbf {K} _ {t} ^ {- 1} \bar {\mathbf {u}}\right) \Big | \mathbf {u} \in m _ {t}, D _ {t} (\mathbf {u}) \text {is valid} \right\},\tag{2}
$$

**Unified World Representation:**

$$
\mathcal {S} (t) = \big (\mathcal {O} _ {t}, \mathcal {U} (t), \mathcal {G}, \mathcal {P}, \mathcal {B} ^ {\mathrm{sta}}, \mathcal {B} ^ {\mathrm{dyn}} (t), \mathcal {Z} (t) \big),\tag{3}
$$

### 3.3 HoliWorld-Data

- **QA pairs**: Following EgoDynamic4D patterns, covering spatial (direction, distance, containment), temporal (motion, event ordering), and interaction (manipulation, state changes) reasoning
- **Dataset composition**: ~100 scenes, 444 videos, 1,322K RGB frames, 90K 2D boxes, 2,466 static 3D boxes, 2,095 dynamic tracks, ~80K QA pairs (13,647 static + 64,409 dynamic)

---

## Empirical Validation / Results

### Static 3D Object Perception

**Table 1** shows HOLIWORLD-FOUNDRY achieves the highest AP₂₅ and AP₅₀ across all three datasets:

| Method | ScanNet AP₂₅↑ | ScanNet AP₅₀↑ | ScanNet mIoU↑ | ScanNet++ AP₂₅↑ | ScanNet++ AP₅₀↑ | ScanNet++ mIoU↑ | Bench AP₂₅↑ | Bench AP₅₀↑ | Bench mIoU↑ |
|---|---|---|---|---|---|---|---|---|---|
| Map-Det3D | 0.40 | 0.12 | 0.48 | 0.18 | 0.05 | 0.48 | 0.12 | 0.01 | 0.37 |
| Boxer | 0.32 | 0.09 | 0.47 | 0.23 | 0.05 | 0.46 | 0.16 | 0.07 | 0.52 |
| Holi-Spatial | 0.33 | 0.10 | 0.47 | 0.17 | 0.05 | 0.45 | 0.07 | 0.01 | 0.37 |
| **Ours** | **0.68** | **0.55** | **0.72** | **0.30** | **0.13** | **0.56** | **0.24** | **0.12** | **0.53** |

### Dynamic 3D Object Perception and Tracking

**Table 2** shows HOLIWORLD-FOUNDRY attains HOTA scores of 54.31% on ADT and 43.14% on HoliWorld-Bench, versus 36.00% and 15.13% for the strongest baselines.

### Downstream Spatial and Temporal Understanding

**Table 3** shows fine-tuning Qwen3.5-4B on HoliWorld-Data yields dramatic improvements:

| Model | Bench F1↑ | Bench Y/N↑ | Bench MCQ↑ | Bench Num.↑ | EgoDyn F1↑ | EgoDyn Y/N↑ | EgoDyn MCQ↑ | EgoDyn Num.↑ |
|---|---|---|---|---|---|---|---|---|
| Qwen3.5-4B | 16.35 | 34.86 | 62.00 | 10.80 | 18.29 | 59.47 | 79.16 | 0.02 |
| Qwen3.8-27B | 43.67 | 82.85 | 75.00 | 66.13 | 42.93 | 61.07 | 57.54 | 0.16 |
| **HoliWorld-VL-4B** | **78.02** | **93.52** | **94.00** | **81.84** | **66.39** | **99.67** | **94.92** | **16.33** |

### Ablation Studies

- **Static components**: Image-based lifting captures more small objects; point-cloud detection recovers larger extents; stereo fusion combines both views; caption guidance recovers missed entities; VLM verification corrects labels and rejects unsuitable candidates
- **Dynamic components**: Cross-interaction label continuity reduces identity fragmentation; SAM association provides temporally consistent masks, reducing trajectory deviations

---

## Theoretical and Practical Implications

**Theoretical contributions:**
- Demonstrates that **human egocentric experience can serve as a scalable foundation** for learning world intelligence without manual annotation
- Establishes that **shared world coordinates and persistent identities** are sufficient to unify static scene structure with dynamic object states
- Shows that **automatically constructed supervision** can outperform much larger models (27B) trained with standard methods, suggesting data quality and grounding matter more than model scale for spatial reasoning

**Practical implications:**
- Provides a **complete, open-source blueprint** (code, models, datasets) for building world intelligence from egocentric recordings
- Reduces the **annotation bottleneck** that limits scalability in egocentric learning
- Enables **continuous expansion** of learning resources: new recordings automatically generate new supervision
- The HOLIWORLD-Bench provides an **independent evaluation standard** for both annotation quality and learned spatial understanding

---

## Conclusion

HOLIWORLD presents an integrated ecosystem that places human egocentric experience at the center of learning world intelligence. Key takeaways:

1. **Automated annotation at scale is feasible**: Complementary static and dynamic perception pipelines can automatically construct world records linking scene geometry, object semantics, and interaction histories.

2. **Grounded supervision improves reasoning**: Training on HOLIWORLD-Data substantially improves spatial and temporal reasoning, with the 4B model surpassing a 27B zero-shot model.

3. **Practical foundation for future work**: The open-source ecosystem (code, models, datasets) provides a blueprint for expanding learning resources with new recordings.

**Future directions** implied by the work include:
- Relaxing the current assumption of at most one moving object per time step
- Extending to more diverse environments and interaction types
- Scaling to larger collections of egocentric recordings from broader populations
- Further integration of static and dynamic perception for even richer world representations

---

_Markdown view of https://picx.dev/p/lcvQQp, served by PicX — AI-generated visual whiteboard summaries of research papers._