Urbex: Agentic Spatial Grounding for City-Scale 3D Scenes

Summary (Overview)

• New paradigm: Urbex reformulates city-scale 3D grounding from static candidate ranking to active spatial search, where a vision-language model (VLM) agent interactively explores a multi-view 3D scene environment through tool calls.
• Interactive environment: The framework exposes city-scale 3D scenes as a BEV-aligned multi-view environment with three complementary tools: Landmark Query (global anchoring), Zoom-in (local detail), and Oblique Rendering (3D verification).
• RL-optimized policy: The tool-use policy is trained with Group Relative Policy Optimization (GRPO) using a smooth terminal reward (IoU + center distance + step cost) plus lightweight tool-level shaping rewards.
• State-of-the-art results: Urbex achieves best Hit@1 (25.8 ND, 23.9 NO) and IoU on CityRefer, supports candidate-free grounding without ground-truth boxes, and is ~10× faster than the strongest ranking baseline (3.2s vs 32.1s per example).
• Zero-shot transfer: Demonstrates strong cross-scene robustness on STPLS3D-Refer (27.2 Hit@1) without any fine-tuning, suggesting the learned search strategy is reusable.

Introduction and Theoretical Foundation

Background and Motivation

City-scale 3D grounding—localizing objects in large outdoor urban scenes from natural language descriptions—is critical for embodied intelligence applications including autonomous driving, robotic navigation, and aerial inspection. Unlike indoor scenes, city-scale environments pose unique challenges:

• Scale: Outdoor scenes contain hundreds of thousands to tens of millions of points, making full point cloud processing computationally prohibitive.
• Abstract referring expressions: Queries often reference named buildings, road directions, or distant landmarks, e.g., "a white car parked next to the orange car to the left of the West Midlands Police Custody Suite."
• Long-range spatial reasoning: Solving such queries requires both fine-grained visual recognition and reasoning over long-range spatial cues.

Limitations of Existing Approaches

The paper identifies two dominant paradigms and their shortcomings:

1. Downsampling-based methods (indoor heritage): Downsample raw point clouds to a fixed size (e.g., 50,000 points), which discards substantial fine-grained geometry when applied to city-scale scenes.

2. Candidate-ranking methods (current city-scale standard): Formulate grounding as static retrieval over precomputed instance proposals:

   • Accuracy depends on proposal quality
   • Inference cost grows with candidate count
   • Reduces grounding to one-shot retrieval, ignoring the inherently sequential and exploratory nature of localization in vast environments

Theoretical Foundation

The paper frames city-scale grounding as an active spatial search problem—a sequential decision process where the agent must:

1. Parse target attributes and spatial cues from the referring expression
2. Decide where to inspect next
3. Acquire new evidence through tool interactions
4. Determine when evidence is sufficient to commit an answer

This formulation leverages recent advances in agentic vision-language models that can reason over visual observations and interact with external tools, combined with reinforcement learning to optimize the search policy.

Methodology

Interactive Environment Design

Urbex converts a city-scale 3D scene $\mathcal{S}$ into a VLM-operable multi-view space:

View representation: The initial view is a global bird's-eye-view (BEV) image $I_{\mathrm{global}}$. Each BEV-aligned view $v \in \mathcal{V}$ contains:

• An image observation $I_v$
• A mapping $T_v$ from view coordinates to world coordinates

The agent expresses boxes in normalized 2D coordinates, which the environment projects back to world coordinates—providing a simple, consistent spatial interface while preserving metric grounding.

Three city-aware tools:

Agentic Reasoning Loop

The policy $\pi_\theta$ is a VLM receiving current observation $o_t$ and interaction history $h_t$:

$$(\mathrm{thought}_t, a_t) \sim \pi_\theta(o_t, h_t) \tag{1}$$

The action $a_t$ is either a tool call with structured arguments or a final answer. The reasoning loop decomposes into:

1. Global search: Parse expression, identify anchors in BEV
2. Local observation: Zoom into promising regions for small objects
3. Candidate confirmation: Render oblique views when top-down evidence is insufficient

Reinforcement Learning Objective

Terminal reward (smooth, no threshold cliff):

$$R_{\mathrm{term}} = \lambda_{\mathrm{iou}} \operatorname{IoU}(\hat{b}, b^{\star}) + \lambda_{\mathrm{dist}} \exp\left(-\frac{d(\hat{b}, b^{\star})}{\tau}\right) - \lambda_{\mathrm{step}} \frac{T}{T_{\mathrm{max}}} \tag{2}$$

where $d(\hat{b}, b^{\star})$ is center distance in the ground plane, $T$ is assistant turns, $T_{\mathrm{max}}$ is the interaction budget. The distance term provides continuous learning signal even when overlap is zero (common early in training).

Tool-level shaping rewards:

$$R(\tau) = R_{\mathrm{term}} + \sum_{t=1}^{T} R_{\mathrm{tool}}(s_t, a_t) \tag{3}$$

• Zoom-in: Positive when zoom center moves closer to target, negative when away (capped)
• Oblique Rendering: Proportional to overlap between rendered region and target (capped)
• Invalid actions: Negative reward for malformed arguments, impossible boxes, or execution errors

Optimization: Trained entirely with GRPO (no supervised trajectory demonstrations):

$$A_i = \frac{R(\tau_i) - \mu_R}{\sigma_R + \epsilon} \tag{4}$$ $$\mathcal{J}_{\mathrm{GRPO}}(\theta) = \frac{1}{G} \sum_{i=1}^{G} \frac{1}{T_i} \sum_{t=1}^{T_i} \left[ \min(r_{i,t}(\theta) A_i, \operatorname{clip}(r_{i,t}(\theta), 1-\epsilon_c, 1+\epsilon_c) A_i) - \beta D_{\mathrm{KL}}^{i,t} \right] \tag{5}$$

where $r_{i,t}(\theta) = \pi_\theta(a_{i,t}|s_{i,t}) / \pi_{\theta_{\mathrm{old}}}(a_{i,t}|s_{i,t})$, $\epsilon_c$ is the clipping threshold, and $\beta$ controls the KL penalty.

Empirical Validation / Results

Main Results on CityRefer

With GT setting (candidate-ranking baselines may use ground-truth candidates; Urbex always outputs free-form boxes):

Without GT setting (candidate-free localization):

Key findings:

• Urbex achieves ~10× speedup over CityAnchor (3.2s vs 32.1s per example)
• A full proposal-based pipeline (SoftGroup + CityAnchor) reaches only 4.2/2.4/4.7 IoU/F1 on NO vs 14.6/7.7/17.8 for Urbex

Zero-Shot Generalization on STPLS3D-Refer

Urbex also achieves 23.3 Hit@1 on 30 manually verified landmark-free re-annotations, demonstrating transfer to queries expressed purely through visual attributes and local spatial relations.

Ablation Studies

Tool and budget ablation (NO split):

Reward ablation (NO split):

Key Analysis Insights

• Landmark importance: Named landmarks are central to city-scale referring expressions—adding Landmark Query more than doubles 2D IoU (4.1 → 8.9). Moderate coordinate noise (10m) lowers Hit@1 by only 1.2 points, but missing landmarks cause drops exceeding ten points.
• Fixed workflow comparison: RL-trained Urbex reaches 10.0 Hit@1 vs 2.7 for base Qwen under a fixed Landmark → Zoom → Oblique → Answer workflow, showing training improves spatial decision-making.
• Box quality: Predicted-to-target area ratio has median 1.40 and P90 9.92, indicating some boxes are considerably larger than targets—refining boxes after target selection could further improve localization.

Theoretical and Practical Implications

Theoretical Contributions

1. Paradigm shift: The paper challenges the dominant candidate-ranking formulation for city-scale grounding, arguing that localization in vast environments is inherently sequential and exploratory rather than a static retrieval problem.

2. Agentic formulation: Provides a principled framework for treating 3D grounding as a multi-step decision process with tool use, connecting the agentic VLM literature to large-scale 3D scene understanding.

3. Reward design: The smooth terminal reward (IoU + center distance + step cost) avoids threshold cliffs common in discrete success bonuses, while tool-level shaping provides intermediate guidance without dominating the terminal objective.

Practical Implications

1. Efficiency: The ~10× inference speedup over ranking baselines (3.2s vs 32.1s) makes city-scale grounding feasible for real-time applications like autonomous driving and aerial inspection.

2. Candidate-free deployment: Urbex does not require precomputed instance proposals, making it deployable in scenarios where accurate instance segmentation is unavailable.

3. Cross-scene robustness: Zero-shot transfer to scenes with different geometry and visual appearance suggests the learned search strategy generalizes beyond dataset-specific statistics.

Conclusion

Urbex presents a fundamental reformulation of city-scale 3D grounding: from exhaustive candidate ranking to active spatial search. By exposing city-scale scenes through an interactive multi-view environment with city-aware tools (Landmark Query, Zoom-in, Oblique Rendering), and optimizing the tool-use policy with reinforcement learning, Urbex achieves:

• Best accuracy on CityRefer (25.8/23.9 Hit@1 on ND/NO) and STPLS3D-Refer zero-shot (27.2 Hit@1)
• Candidate-free localization without ground-truth boxes
• Order-of-magnitude efficiency gains over the strongest ranking baseline

Future directions suggested by the analysis include:

• Improving region selection when the first zoom excludes the target
• Enhancing local verification for distinguishing nearby similar instances
• Refining box boundaries after target identification (median area ratio 1.40 indicates room for improvement)
• Exploring whether the 8-turn budget can be dynamically adapted per query complexity