# Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

> Urbex reframes city-scale 3D grounding as active spatial search, achieving state-of-the-art accuracy with a 10x speedup over ranking baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2610.18356)
- **Published:** 2026-10-15
- **Permalink:** https://picx.dev/p/s2DK3Q
- **Whiteboard:** https://picx.dev/p/s2DK3Q/image

## Summary

# Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

## Summary (Overview)

- **New paradigm**: Urbex reformulates city-scale 3D grounding from static candidate ranking to active spatial search, where a vision-language model (VLM) agent interactively explores a multi-view 3D scene environment through tool calls.
- **Interactive environment**: The framework exposes city-scale 3D scenes as a BEV-aligned multi-view environment with three complementary tools: Landmark Query (global anchoring), Zoom-in (local detail), and Oblique Rendering (3D verification).
- **RL-optimized policy**: The tool-use policy is trained with Group Relative Policy Optimization (GRPO) using a smooth terminal reward (IoU + center distance + step cost) plus lightweight tool-level shaping rewards.
- **State-of-the-art results**: Urbex achieves best Hit@1 (25.8 ND, 23.9 NO) and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is ~10× faster than the strongest ranking baseline (3.2s vs 32.1s per example).
- **Zero-shot transfer**: Demonstrates strong cross-scene robustness on STPLS3D-Refer (27.2 Hit@1) without any fine-tuning, suggesting the learned search strategy is reusable.

## Introduction and Theoretical Foundation

### Background and Motivation

City-scale 3D grounding—localizing objects in large outdoor urban scenes from natural language descriptions—is critical for embodied intelligence applications including autonomous driving, robotic navigation, and aerial inspection. Unlike indoor scenes, city-scale environments pose unique challenges:

- **Scale**: Outdoor scenes contain hundreds of thousands to tens of millions of points, making full point cloud processing computationally prohibitive.
- **Abstract referring expressions**: Queries often reference named buildings, road directions, or distant landmarks, e.g., *"a white car parked next to the orange car to the left of the West Midlands Police Custody Suite."*
- **Long-range spatial reasoning**: Solving such queries requires both fine-grained visual recognition and reasoning over long-range spatial cues.

### Limitations of Existing Approaches

The paper identifies two dominant paradigms and their shortcomings:

1. **Downsampling-based methods** (indoor heritage): Downsample raw point clouds to a fixed size (e.g., 50,000 points), which discards substantial fine-grained geometry when applied to city-scale scenes.

2. **Candidate-ranking methods** (current city-scale standard): Formulate grounding as static retrieval over precomputed instance proposals:
   - Accuracy depends on proposal quality
   - Inference cost grows with candidate count
   - Reduces grounding to one-shot retrieval, ignoring the inherently sequential and exploratory nature of localization in vast environments

### Theoretical Foundation

The paper frames city-scale grounding as an **active spatial search problem**—a sequential decision process where the agent must:
1. Parse target attributes and spatial cues from the referring expression
2. Decide where to inspect next
3. Acquire new evidence through tool interactions
4. Determine when evidence is sufficient to commit an answer

This formulation leverages recent advances in **agentic vision-language models** that can reason over visual observations and interact with external tools, combined with **reinforcement learning** to optimize the search policy.

## Methodology

### Interactive Environment Design

Urbex converts a city-scale 3D scene $\mathcal{S}$ into a VLM-operable multi-view space:

**View representation**: The initial view is a global bird's-eye-view (BEV) image $I_{\mathrm{global}}$. Each BEV-aligned view $v \in \mathcal{V}$ contains:
- An image observation $I_v$
- A mapping $T_v$ from view coordinates to world coordinates

The agent expresses boxes in normalized 2D coordinates, which the environment projects back to world coordinates—providing a simple, consistent spatial interface while preserving metric grounding.

**Three city-aware tools**:

| Tool | Function | Purpose |
|------|----------|---------|
| **Landmark Query** | Resolves named landmarks (buildings, roads) to world positions | Global spatial anchoring for proper nouns |
| **Zoom-in** | Maps a selected box to high-resolution BEV (long side >10,000 px) | Recovers detail lost in downsampled global view |
| **Oblique Rendering** | Renders candidate region from a cardinal direction | 3D-aware verification of shape, height, placement |

### Agentic Reasoning Loop

The policy $\pi_\theta$ is a VLM receiving current observation $o_t$ and interaction history $h_t$:

$$
(\mathrm{thought}_t, a_t) \sim \pi_\theta(o_t, h_t) \tag{1}
$$

The action $a_t$ is either a tool call with structured arguments or a final answer. The reasoning loop decomposes into:
1. **Global search**: Parse expression, identify anchors in BEV
2. **Local observation**: Zoom into promising regions for small objects
3. **Candidate confirmation**: Render oblique views when top-down evidence is insufficient

### Reinforcement Learning Objective

**Terminal reward** (smooth, no threshold cliff):

$$
R_{\mathrm{term}} = \lambda_{\mathrm{iou}} \operatorname{IoU}(\hat{b}, b^{\star}) + \lambda_{\mathrm{dist}} \exp\left(-\frac{d(\hat{b}, b^{\star})}{\tau}\right) - \lambda_{\mathrm{step}} \frac{T}{T_{\mathrm{max}}} \tag{2}
$$

where $d(\hat{b}, b^{\star})$ is center distance in the ground plane, $T$ is assistant turns, $T_{\mathrm{max}}$ is the interaction budget. The distance term provides continuous learning signal even when overlap is zero (common early in training).

**Tool-level shaping rewards**:

$$
R(\tau) = R_{\mathrm{term}} + \sum_{t=1}^{T} R_{\mathrm{tool}}(s_t, a_t) \tag{3}
$$

- **Zoom-in**: Positive when zoom center moves closer to target, negative when away (capped)
- **Oblique Rendering**: Proportional to overlap between rendered region and target (capped)
- **Invalid actions**: Negative reward for malformed arguments, impossible boxes, or execution errors

**Optimization**: Trained entirely with GRPO (no supervised trajectory demonstrations):

$$
A_i = \frac{R(\tau_i) - \mu_R}{\sigma_R + \epsilon} \tag{4}
$$

$$
\mathcal{J}_{\mathrm{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{T_i} \sum_{t=1}^{T_i} \left[ \min(r_{i,t}(\theta) A_i, \operatorname{clip}(r_{i,t}(\theta), 1-\epsilon_c, 1+\epsilon_c) A_i) - \beta D_{\mathrm{KL}}^{i,t} \right] \tag{5}
$$

where $r_{i,t}(\theta) = \pi_\theta(a_{i,t}|s_{i,t}) / \pi_{\theta_{\mathrm{old}}}(a_{i,t}|s_{i,t})$, $\epsilon_c$ is the clipping threshold, and $\beta$ controls the KL penalty.

## Empirical Validation / Results

### Main Results on CityRefer

**With GT setting** (candidate-ranking baselines may use ground-truth candidates; Urbex always outputs free-form boxes):

| Method | ND Hit@1 | ND Hit@3 | ND Hit@5 | NO Hit@1 | NO Hit@3 | NO Hit@5 | Time (s) |
|--------|----------|----------|----------|----------|----------|----------|----------|
| GPT-5.4 (w/ tools) | 9.3 | 12.4 | 15.7 | 13.1 | 18.6 | 20.1 | n/a |
| Qwen3.6-plus (w/ tools) | 6.4 | 9.6 | 11.9 | 11.5 | 16.2 | 16.8 | n/a |
| InstanceRefer | 1.6 | 5.0 | 10.5 | 4.9 | 15.9 | 24.5 | 0.6 |
| EDA | 0.1 | 0.1 | 0.2 | 0.2 | 0.3 | 0.4 | 3.0 |
| CityRefer | 1.6 | 3.9 | 4.8 | 0.6 | 2.1 | 3.4 | 0.7 |
| CityAnchor | 23.4 | 35.3 | 45.4 | 14.2 | 22.4 | 31.0 | 32.1 |
| **Urbex** | **25.8** | **39.2** | 42.3 | **23.9** | **36.9** | **40.7** | **3.2** |

**Without GT setting** (candidate-free localization):

| Method | ND 2D IoU | ND 3D IoU | ND Point F1 | NO 2D IoU | NO 3D IoU | NO Point F1 |
|--------|-----------|-----------|-------------|-----------|-----------|-------------|
| GPT-5.4 | 4.4 | 2.2 | 9.4 | 4.6 | 1.9 | 10.4 |
| Qwen3.6-plus | 3.0 | 1.8 | 9.5 | 4.3 | 2.1 | 12.1 |
| EDA | 0.2 | 0.2 | 0.3 | 0.5 | 0.4 | 0.8 |
| **Urbex** | **15.6** | **11.0** | **20.1** | **14.6** | **7.7** | **17.8** |

**Key findings**:
- Urbex achieves ~10× speedup over CityAnchor (3.2s vs 32.1s per example)
- A full proposal-based pipeline (SoftGroup + CityAnchor) reaches only 4.2/2.4/4.7 IoU/F1 on NO vs 14.6/7.7/17.8 for Urbex

### Zero-Shot Generalization on STPLS3D-Refer

| Method | Hit@1 | Hit@3 | Hit@5 | 2D IoU | 3D IoU | Point F1 | Time (s) |
|--------|-------|-------|-------|--------|--------|----------|----------|
| GPT-5.4 (w/ tools) | 16.3 | 17.8 | 19.1 | 8.0 | 5.3 | 12.1 | n/a |
| Qwen3.6-plus (w/ tools) | 12.0 | 13.8 | 14.3 | 8.1 | 6.4 | 12.2 | n/a |
| InstanceRefer | 3.8 | 7.8 | 12.3 | n/a | n/a | n/a | 0.6 |
| EDA | 0.0 | 0.0 | 0.5 | 0.4 | 0.2 | 0.4 | 2.9 |
| CityRefer | 1.4 | 2.5 | 3.8 | n/a | n/a | n/a | 1.4 |
| CityAnchor | 2.1 | 4.2 | 6.0 | n/a | n/a | n/a | 47.2 |
| **Urbex** | **27.2** | **32.1** | **33.4** | **13.0** | **10.0** | **16.2** | **4.8** |

Urbex also achieves 23.3 Hit@1 on 30 manually verified landmark-free re-annotations, demonstrating transfer to queries expressed purely through visual attributes and local spatial relations.

### Ablation Studies

**Tool and budget ablation** (NO split):

| Zoom | Landmark | Oblique | Turns | Hit@3 | 2D IoU |
|------|----------|---------|-------|-------|--------|
| - | - | - | 0 | 10.5 | 2.1 |
| ✓ | - | - | 8 | 15.5 | 4.1 |
| ✓ | ✓ | - | 8 | 28.2 | 8.9 |
| ✓ | ✓ | ✓ | 7 | 34.8 | 11.0 |
| ✓ | ✓ | ✓ | 8 | **36.9** | **14.6** |
| ✓ | ✓ | ✓ | 9 | 36.8 | 13.2 |

**Reward ablation** (NO split):

| Reward setting | Hit@3 | 2D IoU |
|----------------|-------|--------|
| IoU only | 22.8 | 6.7 |
| IoU + Distance | 30.6 | 10.9 |
| IoU + Tool-level | 28.4 | 9.6 |
| **Full reward** | **36.9** | **14.6** |

### Key Analysis Insights

- **Landmark importance**: Named landmarks are central to city-scale referring expressions—adding Landmark Query more than doubles 2D IoU (4.1 → 8.9). Moderate coordinate noise (10m) lowers Hit@1 by only 1.2 points, but missing landmarks cause drops exceeding ten points.
- **Fixed workflow comparison**: RL-trained Urbex reaches 10.0 Hit@1 vs 2.7 for base Qwen under a fixed Landmark → Zoom → Oblique → Answer workflow, showing training improves spatial decision-making.
- **Box quality**: Predicted-to-target area ratio has median 1.40 and P90 9.92, indicating some boxes are considerably larger than targets—refining boxes after target selection could further improve localization.

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Paradigm shift**: The paper challenges the dominant candidate-ranking formulation for city-scale grounding, arguing that localization in vast environments is inherently sequential and exploratory rather than a static retrieval problem.

2. **Agentic formulation**: Provides a principled framework for treating 3D grounding as a multi-step decision process with tool use, connecting the agentic VLM literature to large-scale 3D scene understanding.

3. **Reward design**: The smooth terminal reward (IoU + center distance + step cost) avoids threshold cliffs common in discrete success bonuses, while tool-level shaping provides intermediate guidance without dominating the terminal objective.

### Practical Implications

1. **Efficiency**: The ~10× inference speedup over ranking baselines (3.2s vs 32.1s) makes city-scale grounding feasible for real-time applications like autonomous driving and aerial inspection.

2. **Candidate-free deployment**: Urbex does not require precomputed instance proposals, making it deployable in scenarios where accurate instance segmentation is unavailable.

3. **Cross-scene robustness**: Zero-shot transfer to scenes with different geometry and visual appearance suggests the learned search strategy generalizes beyond dataset-specific statistics.

## Conclusion

Urbex presents a fundamental reformulation of city-scale 3D grounding: from exhaustive candidate ranking to active spatial search. By exposing city-scale scenes through an interactive multi-view environment with city-aware tools (Landmark Query, Zoom-in, Oblique Rendering), and optimizing the tool-use policy with reinforcement learning, Urbex achieves:

- **Best accuracy** on CityRefer (25.8/23.9 Hit@1 on ND/NO) and STPLS3D-Refer zero-shot (27.2 Hit@1)
- **Candidate-free localization** without ground-truth boxes
- **Order-of-magnitude efficiency gains** over the strongest ranking baseline

**Future directions** suggested by the analysis include:
- Improving region selection when the first zoom excludes the target
- Enhancing local verification for distinguishing nearby similar instances
- Refining box boundaries after target identification (median area ratio 1.40 indicates room for improvement)
- Exploring whether the 8-turn budget can be dynamically adapted per query complexity

---

_Markdown view of https://picx.dev/p/s2DK3Q, served by PicX — AI-generated visual whiteboard summaries of research papers._