Summary (Overview)

• GST-Bench is a new video-based VQA benchmark for evaluating global spatial awareness in Vision-Language Models (VLMs), requiring models to integrate long-horizon egocentric video observations, reason from novel viewpoints unseen in the input, and map observations onto explicit top-down scene representations.
• The benchmark comprises 12 subtasks across three core competencies (self localization, object localization, scene structure understanding) with 2,762 human-verified questions derived from 6,790 minutes of synthetically generated video in 50 diverse indoor scenes.
• A comprehensive evaluation of 22 state-of-the-art VLMs reveals a striking gap: the strongest zero-shot model (Gemini-3-Pro) achieves only 42.68, far below the human baseline of 79.08.
• Controlled local-variant experiments show that proprietary models fail primarily at cross-frame spatial integration (not single-image perception), while open-source models struggle at both local perception and global integration.
• The authors release GST-Train, a large-scale training dataset; fine-tuning Qwen3-VL-8B on it improves the score from 25.89 to 53.52, surpassing all zero-shot proprietary models.

---

Introduction and Theoretical Foundation

Background and Motivation

Spatial intelligence is fundamental for embodied agents (e.g., household robots) that must:

• Incrementally build scene representations from egocentric exploration
• Remember where objects were encountered
• Infer relative locations even when objects leave the field of view

While VLMs have made rapid progress in visual perception and reasoning, existing spatial benchmarks are confined to:

• Single-image settings (e.g., CV-Bench, 3DSRBench, Spatial457) — testing local spatial properties like object size and relative depth
• Multi-image settings (e.g., MM-Spatial, MMSI-Bench) — typically 2–3 views, still limited to localized understanding

Gaps in Existing Video-Based Benchmarks

Recent video benchmarks (VSI-Bench, MMSI-Video-Bench, OST-Bench) begin to explore global spatial understanding but have three key limitations:

1. No disentanglement of single-frame solvable tasks from those requiring cross-frame global reasoning
2. Coarse-grained spatial metrics (e.g., front/back/left/right) rather than precise angles
3. Lack of explicit global scene representations (e.g., top-down images) to evaluate global-local correspondence

Design Principles of GST-Bench

1. Global reasoning by construction: Target objects are guaranteed invisible from the query viewpoint; single-frame answerable questions are excluded
2. Precise numerical answers: Metrics computed over exact distances, angles, and point locations rather than coarse categorical judgments
3. Top-down images: Probe the model's ability to map cross-view video onto a global scene representation
4. Independent query viewpoints: Query images sampled from off-trajectory camera positions

---

Methodology

Task Inputs

GST-Bench includes five types of visual inputs:

Task Taxonomy

Self Localization (2 tasks):

• Global Orientation: Infer camera orientation of current view on top-down image
• Global Position: Infer camera position of current view on top-down image

Object Localization (6 tasks, with semantic/visual modalities):

• Egocentric Direction: Infer direction from current viewpoint to target object (semantic: by category name; visual: by bounding box)
• Egocentric Distance: Estimate metric distance from current viewpoint to target object
• Global Position: Predict target object's position on top-down image

Scene Structure Understanding (4 tasks):

• Top-Down Selection (easy/medium/hard): Select the matching top-down image among four candidates
• Trajectory Selection: Match egocentric motion to one of four candidate trajectories on a top-down image

Data Generation Pipeline

The pipeline (built on OmniGibson simulation with BEHAVIOR-1K, HyperSim, ArtVIP) operates in six stages:

1. Scene Preparation: 50 diverse indoor scenes; evaluation scenes held out from GST-Train
2. Exploration Video Generation: Spatially distributed viewpoints connected into efficient traversal trajectories
3. Object-Annotated Video Generation: Instance-level bounding boxes from rendered segmentation masks
4. Top-Down Image Generation: Three abstraction levels rendered from scene geometry
5. Current View Generation: Off-trajectory viewpoints with independent camera orientations; filtered for sufficient visual overlap
6. QA Generation & Filtering: Template-based QA pairs with answers computed directly from 3D geometry; automatic filtering removes invalid samples

Human Verification: Annotators verify answerability (target recognizability in video, current-view localizability) — yielding 2,762 human-verified questions.

Evaluation Metrics

For distance prediction, Mean Relative Accuracy (MRA) with thresholds $\mathcal{C}_d = \{0.50, 0.55, \dots, 0.95\}$:

$$\mathcal{MRA} = \frac{1}{|\mathcal{C}_d|} \sum_{\theta \in \mathcal{C}_d} \mathbb{1}\left(\frac{|\hat{y} - y|}{y} < 1 - \theta\right) \tag{1}$$

For angle prediction, circular error with tolerance thresholds $\mathcal{C}_a = \{15^\circ, 30^\circ, 45^\circ\}$:

$$e_a = \min\left(|\hat{\alpha} - \alpha|, 360^\circ - |\hat{\alpha} - \alpha|\right) \tag{2}$$ $$\mathrm{Acc}_{\mathrm{angle}} = \frac{1}{|\mathcal{C}_a|} \sum_{\tau \in \mathcal{C}_a} \mathbb{1}(e_a < \tau) \tag{3}$$

For point prediction, Euclidean distance with thresholds $\mathcal{C}_p = \{100, 150, 200, 250, 300\}$ pixels:

$$\mathrm{Acc}_{\mathrm{point}} = \frac{1}{|\mathcal{C}_p|} \sum_{\tau \in \mathcal{C}_p} \mathbb{1}(e_p < \tau) \tag{4}$$

Overall score is the arithmetic mean across all 12 subtasks:

$$\mathrm{Score} = \frac{1}{12} \sum_{i=1}^{12} s_i \tag{5}$$

---

Empirical Validation / Results

Main Results

Key findings from evaluating 22 VLMs:

1. Huge gap between VLMs and humans: Human average = 79.08 vs. best model (Gemini-3-Pro) = 42.68 (gap of 36.4 points). The gap is most pronounced on orientation estimation (21.52 vs. 85.00) and global position estimation (42.23 vs. 93.00).

2. Open-source models struggle: Best open-source models (Qwen3-VL-32B: 30.43, InternVL3.5-38B: 30.71) remain near random on 10 of 12 subtasks. Four models score below random guessing.

3. Embodied-tuned models inherit the blind spot: RoboBrain2.5-8B (24.61) and Cosmos-Reason2-8B (21.64) fall below general-purpose Qwen3-VL-8B (25.89) at the 8B scale.

Table 1: Key Results on GST-Bench

Disentangling Global vs. Local Reasoning

Three controlled settings were constructed: Global (target absent from current view), Local-Video (target visible in current view, video redundant), and Local-Image (video removed, single-image task).

Proprietary models see locally but fail to integrate globally:

• Gemini-3-Pro on ED: 22.11 → 61.20 (Local-Image), nearly tripling
• GPT-5 on ED: 31.09 → 65.61 (Local-Image), +34.52 points

Open-source models fail at both perception and integration:

• Gains are inconsistent; some models degrade in easier settings (e.g., Qwen3-VL-8B on ED: 12.50 → 2.82 in Local-Video)
• Even in Local-Image, open-source scores remain far below proprietary counterparts

Bridging the Gap with Targeted Training

Fine-tuning Qwen3-VL-8B on GST-Train (with general-purpose instruction data mixing):

• Score improvement: 25.89 → 53.52 (+27.63 points)
• Surpasses all zero-shot proprietary models (Gemini-3-Pro: 42.68)
• Notable gains: ED_v: 18.75 → 53.72, GP_s: 13.01 → 57.64, Ori: 17.64 → 44.39

---

Theoretical and Practical Implications

Diagnostic Insights

1. Two qualitatively different bottlenecks exist between model families:

   • Proprietary models have strong single-image spatial perception but fail at cross-frame spatial integration — the bottleneck is memory and integration
   • Open-source models lack both local perception and global integration — the bottleneck is more fundamental

2. Egocentric distance estimation is intrinsically difficult: Human baseline itself is low (~41 MRA), suggesting metric distance recovery is a hard problem for both humans and machines.

3. Embodied training recipes are insufficient: Despite explicit tuning for grounded spatial reasoning, embodied models (RoboBrain, Robix, Cosmos-Reason2) show no advantage — current embodied post-training emphasizes local affordances rather than long-horizon spatial memory.

Practical Implications

• GST-Train demonstrates tractability: Targeted supervision can significantly narrow the gap (+27.63 points), suggesting the bottleneck is addressable with appropriate training data.
• Benchmark design matters: The strict "target-invisible" constraint and precise numerical metrics provide a sharper diagnostic tool than coarse categorical evaluations.
• Simulation-based generation scales: The automatic pipeline (with human verification) enables large-scale, controllable benchmark construction.

---

Conclusion

GST-Bench reveals that global spatial awareness — the ability to build, maintain, and reason over globally consistent spatial representations from long-horizon video — remains a major unsolved challenge for current VLMs. Even the strongest proprietary models achieve less than 55% of human performance, while most open-source models perform near random.

The authors' three main contributions are:

1. GST-Bench: A rigorously constructed benchmark that enforces global reasoning by design (target-invisible constraint, off-trajectory viewpoints, precise metrics, top-down representations)
2. Diagnostic analysis: Controlled local variants that disentangle perception failures from integration failures, revealing distinct bottlenecks for proprietary vs. open-source models
3. GST-Train: A complementary training dataset demonstrating that targeted supervision can substantially improve global spatial reasoning

Future directions include: developing architectures with explicit spatial memory mechanisms, exploring better training recipes for cross-frame integration, and extending the benchmark to more complex, real-world scenarios.

"We hope GST-Bench and GST-Train will facilitate future research toward VLMs that can build, maintain, and reason over globally consistent spatial representations."