Summary (Overview)

• SpatialScout is a new benchmark evaluating vision-language model (VLM) agents through active object-level 3D mapping, requiring agents to autonomously explore scenes, associate objects across views, and estimate object centers in a shared coordinate frame.
• The benchmark comprises 80 diverse indoor scenes with 3,140 target objects, spanning multiple embodied AI simulators (iTHOR, RoboTHOR, ArchitecTHOR, Replica-CAD, ProcTHOR).
• Evaluation jointly measures localization accuracy (via 3D IoU) and target coverage, using object-scale overlap scores that penalize missing predictions.
• Experiments with frontier VLMs (GPT-6 Astra, Claude Opus 5.5, DeepSeek-V4.1-Flash, etc.) reveal substantial limitations: the best model (Claude Opus 5.5) achieves only 41.71% mean IoU, with just 3.25% of targets exceeding IoU > 0.9.
• The paper provides trajectory analysis, tool-use pattern analysis, and failure case studies, identifying gaps between coarse object layout recovery and precise metric map construction.

---

Introduction and Theoretical Foundation

Motivation

• Spatial intelligence requires agents to actively acquire visual evidence, resolve occlusion, and integrate observations into coherent 3D representations.
• Existing spatial benchmarks rely on image/video QA, which has key limitations:
  • Finite answer choices allow guessing or elimination.
  • Language priors may enable correct answers without spatial understanding.
  • QA does not capture the multi-step process of autonomous evidence gathering and representation building.

Key Gap

• Recent benchmarks (ESI-Bench, S³-Bench) introduce active interaction but still evaluate answer correctness.
• Theory of Space examines discrete cognitive maps, but not continuous metric mapping.
• SpatialScout advances the field by making continuous 3D object map construction through active interaction the direct evaluation objective.

Theoretical Foundation

The task requires sustained coordination of:

1. Viewpoint selection – choosing where to look next.
2. Cross-view association – recognizing the same object across views.
3. Geometric estimation – computing 3D positions from 2D observations.
4. Map revision – updating estimates as evidence accumulates.

This forms a closed-loop feedback system: current spatial estimates determine what to observe next, while new observations refine the estimates.

---

Methodology

Task Definition

• Each scene contains N targets with inventory $\mathcal{E} = \{(i, c_i, g_i)\}_{i=1}^N$: instance identifier $i$, category $c_i$, and equivalence group $g_i$.
• Geometric target: Each target's center is the midpoint of its tight world-axis-aligned bounding box (AABB).

The reference center in meters is:

$$\mathbf{p}_i = R_0^\top \left(\frac{\mathbf{b}_i^- + \mathbf{b}_i^+}{2} - \mathbf{o}_0\right)\tag{1}$$

where $\mathbf{b}_i^-, \mathbf{b}_i^+$ are bounding box corners, $\mathbf{o}_0, R_0$ are initial camera position and rotation.

Agent Interface (Tools)

Geometric Support

The triangulation tool minimizes squared distances to viewing lines:

$$\widehat{\mathbf{p}}_{\mathrm{tri}} = \arg \min_{\mathbf{p}} \sum_k \|A_k(\mathbf{p} - \mathbf{o}_k)\|_2^2\tag{3}$$

where $A_k = I - \mathbf{d}_k \mathbf{d}_k^\top$ and $\mathbf{d}_k$ are unit viewing directions.

Image Coordinate Conversion

Delivered images are resized; pixel conversion from delivered $(u_d, v_d)$ to original grid is:

$$u = \left(u_d + \frac{1}{2}\right) \frac{W}{W_d} - \frac{1}{2}, \qquad v = \left(v_d + \frac{1}{2}\right) \frac{H}{H_d} - \frac{1}{2}\tag{2}$$

Budgets

• $B = 100$ model calls
• $V = 100$ acquired images (including initial image)

Scene Composition

• 80 scenes: 32 iTHOR, 12 RoboTHOR, 10 ArchitecTHOR, 6 Replica-CAD, 30 ProcTHOR
• 30–40 targets per scene (12 scenes with 35, 68 with 40 = 3,410 instances)
• 1,489 targets (43.3%) belong to equivalence groups with multiple instances per scene

---

Empirical Validation / Results

Quantitative Results (Object-Micro Localization)

Tool-Use Patterns

• Claude Opus 5.5: most tool calls, balanced use of observation, movement, and mapping.
• GPT-6 Astra: competitive performance with fewer calls; balanced strategy.
• DeepSeek-V4.1-Flash/Qwen3.8-Max: more camera movement but less observation; difficulty with viewpoint selection.

Success/Failure Analysis (40 trajectories)

Geometric Grounding Comparison

Cost Analysis

• Claude Opus 5.5 achieves highest IoU but highest cost ($28.85/task).
• Open-source models offer limited accuracy-cost advantages; Kimi-K3 and Qwen3.8-Max are both expensive and low-scoring.

---

Theoretical and Practical Implications

Theoretical Implications

1. Spatial QA ≠ Spatial Construction: The gap between recognition and metric mapping suggests that language-based spatial reasoning does not reflect true 3D understanding.
2. Geometric consistency alone is insufficient: Low triangulation residuals do not guarantee correct object-center estimates, since correspondence errors and center-selection errors remain undetected by geometric checks alone.
3. Active perception is critical: Models that maintain balanced observation, movement, and mapping behavior (Claude Opus 5.5, GPT-6 Astra) substantially outperform models that explore too much or observe too little, suggesting exploration strategy matters as much as raw perception ability.

Practical Implications

1. Benchmark as a testbed: SpatialScout provides quantitative evaluation for developing agents with reliable 3D spatial representations.
2. Diagnostic capability: The scoring of intermediate map snapshots (map traces/replay) enables analyzing when and why spatial representations emerge or fail during exploration.
3. Tool use matters: Effective geometric tool use (triangulation) correlated strongly with localization accuracy — GPT-6 Astra used triangulation far more effectively than DeepSeek-V4.1-Flash despite capturing fewer images.

---

Conclusion

Key Takeaways

• SpatialScout is the first benchmark to evaluate VLM agents through active object-level 3D mapping with explicit 3D coordinate scoring.
• Frontier VLMs can recover partial scene layouts but fail to achieve both comprehensive coverage and precise localization simultaneously.
• The strongest model (Claude Opus 5.5) achieves only 41.71% mean IoU — substantial room for improvement remains.

Future Directions

• Advancing active perception strategies for viewpoint selection.
• Improving cross-view reasoning and correspondence reliability.
• Enhancing persistent spatial memory to maintain coherent maps over long horizons.
• Developing agents capable of constructing accurate and complete 3D representations through sustained autonomous interaction.

Key Quote

"Recognizing objects or making coarse spatial judgments does not necessarily translate into constructing reliable metric maps."

The authors position SpatialScout as a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception, bridging the gap between passive spatial understanding and active spatial intelligence.