# SpatialScout: Can VLM Agents Actively Explore and Map 3D Scenes?

> SpatialScout benchmarks vision-language model agents on active 3D object mapping, revealing frontier models achieve only 41.71% mean IoU, far from precise metric reconstruction.

- **Source:** [arXiv](https://arxiv.org/abs/2610.26659)
- **Published:** 2026-10-23
- **Permalink:** https://picx.dev/p/pF3utt
- **Whiteboard:** https://picx.dev/p/pF3utt/image

## Summary

## Summary (Overview)

- **SpatialScout** is a new benchmark evaluating vision-language model (VLM) agents through **active object-level 3D mapping**, requiring agents to autonomously explore scenes, associate objects across views, and estimate object centers in a shared coordinate frame.
- The benchmark comprises **80 diverse indoor scenes** with **3,140 target objects**, spanning multiple embodied AI simulators (iTHOR, RoboTHOR, ArchitecTHOR, Replica-CAD, ProcTHOR).
- Evaluation jointly measures **localization accuracy** (via 3D IoU) and **target coverage**, using object-scale overlap scores that penalize missing predictions.
- Experiments with frontier VLMs (GPT-6 Astra, Claude Opus 5.5, DeepSeek-V4.1-Flash, etc.) reveal substantial limitations: the best model (Claude Opus 5.5) achieves only **41.71% mean IoU**, with just **3.25% of targets exceeding IoU > 0.9**.
- The paper provides **trajectory analysis, tool-use pattern analysis, and failure case studies**, identifying gaps between coarse object layout recovery and precise metric map construction.

---

## Introduction and Theoretical Foundation

### Motivation
- **Spatial intelligence** requires agents to actively acquire visual evidence, resolve occlusion, and integrate observations into coherent 3D representations.
- Existing spatial benchmarks rely on **image/video QA**, which has key limitations:
  - Finite answer choices allow guessing or elimination.
  - Language priors may enable correct answers without spatial understanding.
  - QA does not capture the multi-step process of autonomous evidence gathering and representation building.

### Key Gap
- Recent benchmarks (ESI-Bench, S³-Bench) introduce active interaction but still evaluate answer correctness.
- **Theory of Space** examines discrete cognitive maps, but not continuous metric mapping.
- **SpatialScout advances the field** by making **continuous 3D object map construction through active interaction** the direct evaluation objective.

### Theoretical Foundation
The task requires sustained coordination of:
1. **Viewpoint selection** – choosing where to look next.
2. **Cross-view association** – recognizing the same object across views.
3. **Geometric estimation** – computing 3D positions from 2D observations.
4. **Map revision** – updating estimates as evidence accumulates.

This forms a **closed-loop feedback system**: current spatial estimates determine what to observe next, while new observations refine the estimates.

---

## Methodology

### Task Definition
- Each scene contains N targets with inventory $\mathcal{E} = \{(i, c_i, g_i)\}_{i=1}^N$: instance identifier $i$, category $c_i$, and equivalence group $g_i$.
- **Geometric target**: Each target's center is the midpoint of its tight world-axis-aligned bounding box (AABB).

The reference center in meters is:

$$
\mathbf{p}_i = R_0^\top \left(\frac{\mathbf{b}_i^- + \mathbf{b}_i^+}{2} - \mathbf{o}_0\right)\tag{1}
$$

where $\mathbf{b}_i^-, \mathbf{b}_i^+$ are bounding box corners, $\mathbf{o}_0, R_0$ are initial camera position and rotation.

### Agent Interface (Tools)
| Tool | Effect/Output |
|------|---------------|
| `list_entities` | List targets and metadata |
| `get_camera_pose` | Return camera pose |
| `get_camera_intrinsics` | Return camera calibration |
| `set_camera_position` | Set camera position |
| `move_camera` | Translate camera |
| `rotate_camera` | Rotate camera |
| `capture_image` | Capture RGB frame |
| `triangulate_observations` | Estimate a point from multiple views |
| `submit_positions` | Update center map |
| `finish_mapping` | End episode |

### Geometric Support
The triangulation tool minimizes squared distances to viewing lines:
$$
\widehat{\mathbf{p}}_{\mathrm{tri}} = \arg \min_{\mathbf{p}} \sum_k \|A_k(\mathbf{p} - \mathbf{o}_k)\|_2^2\tag{3}
$$
where $A_k = I - \mathbf{d}_k \mathbf{d}_k^\top$ and $\mathbf{d}_k$ are unit viewing directions.

### Image Coordinate Conversion
Delivered images are resized; pixel conversion from delivered $(u_d, v_d)$ to original grid is:
$$
u = \left(u_d + \frac{1}{2}\right) \frac{W}{W_d} - \frac{1}{2}, \qquad v = \left(v_d + \frac{1}{2}\right) \frac{H}{H_d} - \frac{1}{2}\tag{2}
$$

### Budgets
- **$B = 100$ model calls**
- **$V = 100$ acquired images** (including initial image)

### Scene Composition
- **80 scenes**: 32 iTHOR, 12 RoboTHOR, 10 ArchitecTHOR, 6 Replica-CAD, 30 ProcTHOR
- 30–40 targets per scene (12 scenes with 35, 68 with 40 = 3,410 instances)
- **1,489 targets (43.3%)** belong to equivalence groups with multiple instances per scene

---

## Empirical Validation / Results

### Quantitative Results (Object-Micro Localization)

| Model | Mean IoU (All) | >0% | >50% | >90% | Mean IoU (Submitted) | >0% | >50% | >90% |
|-------|---------------|-----|------|------|---------------------|-----|------|------|
| **Claude Opus 5.5** | **41.71** | 77.36 | 47.23 | 3.25 | **46.69** | 86.60 | 52.87 | 3.64 |
| GPT-6 Astra (xhigh) | 33.16 | 67.79 | 35.15 | 1.62 | 41.16 | 83.86 | 42.69 | 2.09 |
| GPT-6 Astra (medium) | 31.18 | 67.73 | 31.55 | 1.27 | 37.16 | 80.72 | 37.60 | 1.52 |
| Claude Sonnet 5.5 | 32.26 | 67.13 | 33.60 | 1.40 | 38.99 | 81.14 | 40.61 | 1.69 |
| GPT-6 Sol | 20.83 | 51.85 | 18.95 | 0.48 | 28.65 | 71.31 | 26.06 | 0.66 |
| Qwen3.8 Max | 6.80 | 24.36 | 4.52 | 0.00 | 12.14 | 43.49 | 8.07 | 0.00 |
| Kimi-K3 | 4.22 | 19.36 | 2.77 | 0.00 | 6.55 | 30.03 | 4.29 | 0.00 |
| DeepSeek V4.1 Flash | 3.48 | 19.04 | 0.92 | 0.00 | 5.80 | 31.74 | 1.54 | 0.00 |

### Tool-Use Patterns
- **Claude Opus 5.5**: most tool calls, balanced use of observation, movement, and mapping.
- **GPT-6 Astra**: competitive performance with fewer calls; balanced strategy.
- **DeepSeek-V4.1-Flash/Qwen3.8-Max**: more camera movement but less observation; difficulty with viewpoint selection.

### Success/Failure Analysis (40 trajectories)

| Outcome | Factor | Count |
|---------|--------|-------|
| Success | Sufficient multi-view grounding | 6 |
| Success | Incremental submission with continued coverage expansion | 6 |
| Success | Continued refinement | 6 |
| **Failure** | **Incomplete object coverage** | **26** |
| Failure | Suspected correspondence errors despite low residuals | 21 |
| Failure | Unresolved geometric inconsistencies | 13 |
| Failure | Early termination before coverage | 13 |
| Failure | Position estimates lacking triangulation support | 6 |
| Failure | Extensive observation with limited localization progress | 5 |
| Failure | Late-stage refinement without recovering missing objects | 3 |

### Geometric Grounding Comparison
| Metric | GPT-6 Astra | DeepSeek-V4.1 |
|--------|-------------|---------------|
| Submitted objects with prior successful triangulation | 99.33% | 63.48% |
| Triangulation with geometric warnings | 2.69% | 38.65% |
| Mean image captures per task | 32.41 | 39.63 |
| **Median center error (cm)** | **7.08** | **68.17** |

### Cost Analysis
- Claude Opus 5.5 achieves highest IoU but highest cost (**$28.85/task**).
- Open-source models offer limited accuracy-cost advantages; Kimi-K3 and Qwen3.8-Max are both expensive and low-scoring.

---

## Theoretical and Practical Implications

### Theoretical Implications
1. **Spatial QA ≠ Spatial Construction**: The gap between recognition and metric mapping suggests that language-based spatial reasoning does not reflect true 3D understanding.
2. **Geometric consistency alone is insufficient**: Low triangulation residuals do not guarantee correct object-center estimates, since correspondence errors and center-selection errors remain undetected by geometric checks alone.
3. **Active perception is critical**: Models that maintain balanced observation, movement, and mapping behavior (Claude Opus 5.5, GPT-6 Astra) substantially outperform models that explore too much or observe too little, suggesting exploration *strategy* matters as much as raw perception ability.

### Practical Implications
1. **Benchmark as a testbed**: SpatialScout provides quantitative evaluation for developing agents with reliable 3D spatial representations.
2. **Diagnostic capability**: The scoring of intermediate map snapshots (map traces/replay) enables analyzing *when* and *why* spatial representations emerge or fail during exploration.
3. **Tool use matters**: Effective geometric tool use (triangulation) correlated strongly with localization accuracy — GPT-6 Astra used triangulation far more effectively than DeepSeek-V4.1-Flash despite capturing fewer images.

---

## Conclusion

### Key Takeaways
- **SpatialScout** is the first benchmark to evaluate VLM agents through **active object-level 3D mapping** with explicit 3D coordinate scoring.
- Frontier VLMs can recover partial scene layouts but **fail to achieve both comprehensive coverage and precise localization simultaneously**.
- The strongest model (Claude Opus 5.5) achieves only **41.71% mean IoU** — substantial room for improvement remains.

### Future Directions
- Advancing **active perception** strategies for viewpoint selection.
- Improving **cross-view reasoning** and correspondence reliability.
- Enhancing **persistent spatial memory** to maintain coherent maps over long horizons.
- Developing agents capable of constructing **accurate and complete 3D representations** through sustained autonomous interaction.

### Key Quote
> "Recognizing objects or making coarse spatial judgments does not necessarily translate into constructing reliable metric maps."

The authors position SpatialScout as a **quantitative testbed** for developing agents that build reliable 3D spatial representations through active perception, bridging the gap between passive spatial understanding and active spatial intelligence.

---

_Markdown view of https://picx.dev/p/pF3utt, served by PicX — AI-generated visual whiteboard summaries of research papers._