# Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

> Image-generation models excel at pixel-level spatial evidence while text VLMs dominate compositional reasoning, revealing complementary strengths in spatial AI.

- **Source:** [arXiv](https://arxiv.org/abs/2607.21072)
- **Published:** 2026-07-25
- **Permalink:** https://picx.dev/p/kDmUaz
- **Whiteboard:** https://picx.dev/p/kDmUaz/image

## Summary

## Summary (Overview)

- **Answer-Interface Mismatch**: Existing spatial reasoning benchmarks require text/coordinate answers, creating a compatibility gap for image-generation models that can express spatial judgments directly in pixel space. This forces continuous visual judgments into unnatural symbolic formats.
- **ProVisE Framework**: A benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original task metrics. It includes an Agentic builder for automatic protocol construction on new benchmarks.
- **SpatialGen-Bench**: A curated diagnostic benchmark with 470 samples across 14 spatial subtasks organized into four capability levels (perception, understanding, reasoning, interaction) and diverse answer forms (discrete choices, counts, points, masks, trajectories, etc.).
- **Key Findings**: Image-generation models are competitive when answers can be externalized directly as pixel-space evidence (e.g., depth fields, grounded locations), while text-output VLMs remain stronger in compositional reasoning requiring transformations beyond visible evidence. Neither interface uniformly dominates.
- **Parser Sensitivity**: Substituting the task-specific parser with a universal VLM parser changes both scores and rankings, showing that how generated evidence is decoded significantly affects evaluation outcomes.

## Introduction and Theoretical Foundation

Spatial intelligence is essential for agents to interact with the physical world, spanning perception, understanding, reasoning, and interaction. While significant progress has been made in evaluating vision-language models (VLMs) through text-based spatial benchmarks, existing evaluation lines suffer from an **answer-interface mismatch**:

- Text-to-image benchmarks assess prompt fidelity, not spatial question answering grounded in input scenes.
- Spatial reasoning benchmarks for VLMs require symbolic outputs (coordinates, options, textual descriptions), which are unnatural for spatial tasks where continuous visual judgments are better expressed by pointing, marking, or drawing.
- Image-generation models can express spatial judgments directly in pixel space (e.g., marking a region, drawing a trajectory), but their responses cannot be scored by benchmarks expecting textual/coordinate predictions.

Using an auxiliary VLM judge to read generated images introduces model-dependent uncertainty and misaligns with original task metrics. The paper addresses this gap by introducing **ProVisE** (Protocolized Visual Evaluation) and **SpatialGen-Bench**, enabling metric-compatible comparison between text-output VLMs and image-generation models under shared task semantics.

## Methodology

### ProVisE Framework
The pipeline (Figure 2 in paper) preserves source-task metrics while replacing the response interface with a constrained **visual protocol**:

1. **Visual Protocol Definition**: A pair consisting of a **guidance prompt** (constrains how the model expresses its answer in pixel space) and a **parser** (converts generated image into structured prediction). Also specifies answer format, invalid-output conditions, and metric mapping.

2. **Protocol Routing**: For each task, a protocol is selected from a pool of predefined visual answer formats (instance/point marking, direction grids, relative-depth maps, region masks, state matching, trajectory drawing, etc.) with corresponding parsers.

3. **Agentic Protocol Construction**: For previously unconfigured benchmarks, three stages:
   - Normalize benchmark into task-level contracts (input, answer format, evaluation metrics).
   - Agent selects one of three construction routes: **Reuse** (existing compatible protocol), **Build** (assembles task-specific protocol from registered components), or **Fallback** (constrained auxiliary parser when deterministic components cannot recover required answer form).
   - Validation on small sample set (smoke validation for generation, parsing, metric compatibility).

4. **Execution and Parsing**: Image-generation model produces visual response under protocol constraints. Deterministic parsing preferred; Fallback uses a protocol-constrained generated-image-only parser (no source image or original question).

5. **Metric-Compatible Evaluation**: Parsed predictions are scored using original task metrics, same as for text-output VLMs.

### SpatialGen-Bench Construction
- **Data sources**: Drawn from multiple public spatial benchmarks (BLINK, MindCube, OmniSpatial, etc.) across perception, understanding, reasoning, interaction.
- **Quality-aware selection**: Manual inspection; remove near-duplicates, unclear cases, ambiguous answers.
- **Capability hierarchy** (Figure 5):
  - **Spatial Perception** (145 samples): counting, relative depth, orientation, object size.
  - **Spatial Understanding** (140 samples): relationship verification, perspective judgment, mental modeling, spatial grounding.
  - **Spatial Reasoning** (90 samples): multi-hop reasoning, state prediction, geometric feasibility.
  - **Spatial Interaction** (95 samples): affordance grounding, navigation, trajectory planning.
- **Answer forms**: Discrete choices, counts, points, masks, candidate states, continuous trajectories.
- **Evaluation metrics**: Exact-match accuracy (counting, discrete answers), point-in-mask accuracy (grounding), mean mask precision (affordance), partial-credit endpoint score (prediction), DFD-success@0.4 based on Discrete Fréchet Distance (trajectory).

## Empirical Validation / Results

### Main Results on SpatialGen-Bench (Table 2)

| Model | Perception | Understanding | Reasoning | Interaction | Overall |
|-------|-----------|--------------|-----------|------------|---------|
| **Human** | 97.41 | 85.71 | 77.92 | 87.60 | 87.79 |
| **Text Answering** | | | | | |
| GPT-5.4 | 74.61 | 44.29 | 56.64 | 69.70 | **61.04** |
| Cosmos3-Nano | 76.82 | 45.00 | 55.20 | 63.15 | 60.17 |
| **Visual Answering** | | | | | |
| GPT Image 2 | 69.58 | 47.14 | 41.83 | 56.80 | **54.49** |
| Nano Banana 2 | 72.92 | 42.14 | 24.92 | 48.61 | 48.63 |

- Current models remain well below human performance (largest gap: Understanding, 38.57 points).
- Spatially specialized models (SpaceR, RynnBrain, RoboBrain) show strong performance on specific tasks but uneven profiles across broader capabilities.

### Interface Strengths and Complementarity (Figure 6)
- **Visual advantage**: Depth (+18.85 in model-group comparison), Relationship (+6.74), Navigation (+8.0). Tasks where answers remain directly visible as pixel-level spatial states.
- **Text advantage**: Size, Feasibility, Prediction, Counting. Tasks requiring transformation beyond visible evidence (compositional reasoning, constraint composition, state updates).
- **Sample-level complementarity** (Table 3): GPT Image 2 solves 71 items missed by GPT-5.4 (37.0% recovery); SenseNova visual mode solves 78 items missed by its text mode (30.6% recovery). Visual successes are not a subset of text successes.

### Agentic Cross-Benchmark Generalization (Table 4)
Tested on EmbSpatial-Bench, OmniSpatial, Q-Spatial+, RoboSpatial-Home, SAT, RoboAfford (50-record pilots each). GPT Image 2 achieves competitive scores (highest on SAT and RoboAfford), demonstrating protocol transfer across heterogeneous task contracts.

| Model | EmbSpatial | OmniSpatial | Q-Spatial | RoboSpatial | SAT | RoboAfford | Avg. |
|-------|-----------|-------------|-----------|-------------|-----|------------|------|
| GPT Image 2 (Visual) | 42.00 | 44.00 | 68.00 | 46.00 | **74.00** | **64.00** | 56.33 |
| GPT-5.4 (Text) | **82.00** | 38.00 | 72.00 | **58.00** | 66.00 | 48.00 | **60.67** |

### Universal VLM Parser Sensitivity (Table 5)
Replacing task-specific parsers with universal VLM parsers (Qwen3-VL-8B, Qwen2.5-VL-72B, Llama 4 Scout) changes both scores and rankings:
- ProVisE and Qwen3-VL-8B rank GPT Image 2 first; Qwen2.5-VL-72B ranks JoyAI-Image first; Llama 4 Scout ranks Seedream 4.5 first.
- Rank correlations: 0.60, 0.31, 0.60; mean absolute score changes: 5.87, 5.63, 5.26 points.

### Failure Attribution (Figure 7)
Among non-correct outcomes:
- **88.03%** are valid but incorrect spatial predictions (correctly parsed, wrong answer)
- 8.46% protocol noncompliance, 3.45% parser failure, 0.06% generation failure

The visual communication channel usually completes; the bottleneck is incorrect reasoning, not expression. Failures concentrate in tasks requiring hidden transformations (Size, Feasibility) or precise visual symbols (Prediction, Grounding).

## Theoretical and Practical Implications

- **Modality-Specific Strengths**: The division of strengths suggests that spatial AI systems should not force every task into a single output modality. Visual externalization preserves continuous geometry and makes evidence inspectable; text/latent representations handle constraints and planning; deterministic operators verify results.
- **Evaluation Methodology**: ProVisE enables metric-compatible comparison across model families without relying on post-hoc VLM judges, reducing evaluation bias. The parser sensitivity experiment warns that judge-based evaluations may introduce additional model-dependent interpretation layers.
- **Diagnostic Coverage**: SpatialGen-Bench provides a hierarchical capability diagnosis (perception→understanding→reasoning→interaction) that reveals uneven profiles in spatially specialized models and identifies specific bottlenecks (relationship semantics, viewpoint transformation).
- **Practical Applications**: For embodied agents, the results suggest coordinated use of visual generation for grounding and geometry, and text reasoning for constraint satisfaction and planning. The Agentic builder allows rapid adaptation to new benchmarks, supporting broader generative evaluation.

## Conclusion

ProVisE and SpatialGen-Bench establish a metric-compatible testbed for evaluating spatial cognition in image-generation models alongside text-output VLMs. The key takeaway is a division of strengths: image-generation models excel when answers can be externalized directly as pixel-level spatial evidence (depth fields, grounded locations), while text-output VLMs remain stronger for compositional reasoning requiring transformations beyond visible evidence. Most failed visual responses are parseable but encode wrong answers—showing does not replace reasoning.

**Limitations**: Agentic protocol construction depends on its planning/validation backends (GPT-5.4, GPT Image 2), potentially favoring some generators. VLM-based Fallback introduces provider-dependent variability. Current framework targets static-image benchmarks; extending to video, simulation, and closed-loop embodied tasks is future work.

**Future Directions**: Expand reusable recipes, operators, and validation tests for the open-source framework. Investigate coordinated use of text and visual representations where "show" and "tell" exchange intermediate states and cross-check results. Extend evaluation to dynamic and interactive spatial tasks.

---

_Markdown view of https://picx.dev/p/kDmUaz, served by PicX — AI-generated visual whiteboard summaries of research papers._