Summary (Overview)

• Holi-Spatial is the first fully automated, large-scale, spatially-aware multimodal dataset construction pipeline that converts raw video streams into holistic 3D spatial annotations without human intervention.
• The pipeline operates in three stages: (1) geometric optimization using 3D Gaussian Splatting (3DGS) with monocular depth priors, (2) image-level perception using VLMs and SAM3 for open-vocabulary segmentation, and (3) scene-level refinement with multi-view merging, confidence filtering, and VLM-based verification.
• The resulting Holi-Spatial-4M dataset contains 12K optimized 3DGS scenes, 1.3M 2D masks, 320K 3D bounding boxes, 320K instance captions, 1.2M 3D grounding instances, and 1.2M spatial QA pairs.
• Holi-Spatial significantly outperforms existing methods: improving multi-view depth estimation by 0.5 F1 and boosting 3D detection AP50 by 64% on ScanNet, achieving an AP of 81.06 on ScanNet++ 3D detection (vs. 12.2 for LLaVA-3D).
• Fine-tuning Qwen3-VL on Holi-Spatial-4M yields state-of-the-art results: 15% AP50 gain on ScanNet++ 3D grounding and 7.9% accuracy improvement on MMSI-Bench.

Introduction and Theoretical Foundation

Spatial intelligence is a fundamental capability for large multimodal models (LMMs) to understand the real 3D world, enabling applications in robotic manipulation, navigation, scene editing, and augmented reality. However, the field faces a critical bottleneck: scarcity and imbalance of raw spatial data.

Key limitations of prior approaches:

• Existing methods generate QA pairs from a small set of manually annotated 3D datasets (e.g., ScanNet, ScanNet++) or naively apply feed-forward perception models to single images.
• These approaches rely on specialized scanning hardware and human-in-the-loop annotation, making them difficult to scale.
• Semantic coverage is limited (e.g., only 50 labeled classes in ScanNet).

Theoretical insight: Recent advances in AI tools (Depth-Anything-V3, SAM3, Gemini, Qwen3-VL) have exceeded expectations; by systematically composing them, one can build an automated spatial annotation engine that may even outperform human annotations, enabling a positive data flywheel.

Comparison of paradigms:

• 2D VLM methods (SAM3, SA2VA): operate on images only, lack 3D capabilities.
• 3D VLM methods (SpatialLM, LLaVA-3D, SceneScript): require explicit 3D point clouds as input.
• 3DGS-based methods (M3-Spatial, LangSplat): leverage 3DGS but require per-scene optimization and lack comprehensive outputs.

Holi-Spatial uniquely unifies all these capabilities: it accepts raw images as input and outputs depth, 2D segmentation, 3D detection, grounding, and spatial QA.

Methodology

The Holi-Spatial pipeline consists of three progressive stages:

Stage 1: Geometric Optimization

• Uses Structure-from-Motion (SfM) to resolve camera intrinsics and extrinsics.
• Initializes a dense point cloud using Depth-Anything-V3 (a spatial foundation model).
• Optimizes a 3D Gaussian Splatting (3DGS) scene with geometric regularization to enforce multi-view depth consistency, eliminating floaters and producing clean, surface-aligned geometry.

Stage 2: Image-level Perception

• Uniformly samples keyframes $J = \{I_1, \ldots, I_T\}$ from the raw video stream.
• Uses Gemini3-Pro to generate captions sequentially, maintaining a dynamic class-label memory $\mathcal{M}_t$ for semantic consistency:

$$\mathcal{M}_t = \mathcal{M}_{t-1} \cup \text{Extract}(I_t)$$

• SAM3 performs open-vocabulary instance segmentation, producing predictions $O_t = \{(M_k, s_k)\}_{k=1}^N$.
• 2D-to-3D lifting: unprojects each pixel $\mathbf{u} = (u, v)$ in mask $M_k$ into 3D:

$$\mathbf{P} = D_t(\mathbf{u}) \cdot \mathbf{K}^{-1} \tilde{\mathbf{u}}$$

where $\mathbf{K}$ is the camera intrinsic matrix and $\tilde{\mathbf{u}} = [u, v, 1]^\top$ is the homogeneous coordinate.

• A geometry-aware filtering strategy handles depth edge floaters via: (i) mask erosion to mitigate 2D boundary errors, and (ii) mesh-guided depth filtering for 3D outliers.
• Floor-aligned OBB post-processing: detects floor plane for global up-axis, re-aligns instance OBBs under yaw-lock constraint.

Stage 3: Scene-level Refinement

1. Multi-View Merge & Post-Process: Merges redundant detections when:

$$c_i = c_j \wedge \text{IoU}_{3D}(B_i, B_j) > \tau_{\text{merge}}$$

with $\tau_{\text{merge}} = 0.2$. Confidence updated as $s_k = \max(s_i, s_j)$.

2. Confidence-Based Filtering & Refinement: A tri-level decision rule:

$$\text{Action}(p_k) = \begin{cases} \text{keep}, & s_k \geq \tau_{\text{high}}, \\ \text{discard}, & s_k < \tau_{\text{low}}, \\ \text{verify}, & \tau_{\text{low}} \leq s_k < \tau_{\text{high}}, \end{cases}$$

with $\tau_{\text{high}} = 0.9$ and $\tau_{\text{low}} = 0.8$. Borderline cases are re-assessed by a VLM-based agent with image zoom-in and SAM3 re-segmentation tools.

3. Annotation Generation: For validated instances, retrieves the optimal source image and uses Qwen3-VL-30B to generate fine-grained captions and procedurally synthesize spatial QA pairs covering 3D grounding, spatial reasoning, and attribute identification.

Empirical Validation / Results

Framework Evaluation (Table 2)

Key findings:

• Depth F1 of 0.89 on ScanNet++ vs. 0.39 for M3-Spatial (over 2× improvement).
• 3D detection AP of 81.06 on ScanNet++ exceeds LLaVA-3D (12.2 AP) by an order of magnitude.
• 2D segmentation IoU of 0.64 vs. 0.25 for SA2VA on ScanNet++.

VLM Fine-tuning Results

Spatial Reasoning (Table 3):

3D Grounding on ScanNet++ (Table 4):

The AP₅₀ of 27.98 exceeds the strongest baseline by 14.48 AP points.

Ablation Study (Table 5)

Key ablations:

• 3DGS training improves precision from 0.13 to 0.81 (critical for geometric quality).
• Confidence filtering improves precision from 0.35 to 0.67 but reduces recall (0.74 → 0.69).
• Agent recall recovers true positives with low confidence, yielding the best precision-recall balance.

Theoretical and Practical Implications

Theoretical contributions:

1. Scalability paradigm shift: Demonstrates that raw web videos can serve as an abundant source for 3D spatial data, breaking the dependency on specialized scanning hardware and human annotation.
2. Compositional AI: Shows that systematically composing foundation models (depth estimation, segmentation, VLMs) can outperform human-level annotation quality.
3. Multi-level supervision: Provides a unified framework supporting diverse spatial tasks (depth, 2D segmentation, 3D detection, grounding, spatial QA) from a single pipeline.

Practical implications:

1. Data flywheel: The fully automated pipeline can be scaled up as computational resources permit, enabling continuous dataset growth.
2. Model improvement: Fine-tuning VLMs on Holi-Spatial-4M significantly improves 3D grounding and spatial reasoning, with applications in robotics, AR/VR, and embodied AI.
3. Open-vocabulary richness: Unlike closed-set datasets, Holi-Spatial-4M captures fine-grained indoor object categories through open-world VLM knowledge.

Limitations:

• Computationally expensive due to per-scene optimization.
• May degrade under challenging videos (limited viewpoints, motion blur, heavy occlusion, dynamic objects).
• Open-vocabulary labeling may inherit biases from foundation models.

Conclusion

Holi-Spatial presents a fully automated pipeline converting raw videos into high-fidelity 3D geometry and holistic spatial annotations, addressing the fundamental scalability bottleneck in spatial intelligence. The released Holi-Spatial-4M dataset—with 12K optimized 3DGS scenes, 1.3M masks, 320K 3D boxes/captions, 1.2M 3D grounding instances, and 1.2M QA pairs—demonstrates state-of-the-art performance across depth estimation, 2D segmentation, and 3D detection, while fine-tuning VLMs on this data consistently improves 3D grounding and spatial reasoning.

Future directions:

• Improve efficiency (adaptive early stopping, better confidence-based validation).
• Expand to broader domains and longer video contexts.
• Build stronger benchmarks for holistic 3D spatial understanding.
• Address privacy concerns through responsible deployment with consent and data governance safeguards.