S-Space: Exploring Spatial Workspace in Multimodal Language Models

Summary (Overview)

• Discovery of S-Space: The paper identifies a three-dimensional linear subspace in multimodal language models (MLLMs) that encodes continuous spatial coordinates (horizontal, vertical, and distance axes) of objects in viewer-centered reference frames.
• Gradient-based identification method: The authors propose a simple gradient-based procedure to identify S-Space directions by differentiating spatial answer scores with respect to object-token activations, yielding directions that generalize across datasets, tasks, and prompts.
• Causal role demonstrated: Through causal interventions (editing coordinates in S-Space), the authors show that S-Space selectively and causally shapes spatial judgments while largely preserving non-spatial responses.
• Diagnostic utility: S-Space is used as a diagnostic tool to decompose spatial reasoning failures into perception vs. computation bottlenecks, revealing that MLLMs often possess accurate spatial representations yet fail at viewpoint transformations.
• Emergence from language training: Case studies show S-Space extends beyond visible objects to unseen entities, the observer, and even abstract concepts (e.g., political left/right), suggesting it emerges from predicting spatial language during large-scale training.

---

Introduction and Theoretical Foundation

Background and Motivation

Multimodal large language models (MLLMs) have demonstrated promising spatial intelligence capabilities in applications such as reconstruction, planning, navigation, and manipulation. However, robust spatial reasoning remains a fundamental challenge, with substantial performance gaps compared to human capabilities. The key problem is that external behaviors (answers, actions) reveal little about where failures arise—the internal mechanisms supporting spatial reasoning remain poorly understood.

Theoretical Foundation

The research draws inspiration from human visual cognition, specifically the concept of a mental workspace in which internal representations are maintained and manipulated in the human brain. The central hypothesis is:

MLLMs possess an internal spatial workspace—a linear representation subspace—in which object-token representations encode continuous coordinates along horizontal, vertical, and distance axes.

This builds on the widely-held hypothesis that large language models represent concepts linearly in their internal representations. The authors hypothesize that an object's spatial coordinates, defined in a viewer-centered (egocentric) reference frame, are linearly encoded in the hidden representation of its object-name token.

---

Methodology

2.1 S-Space Formulation

For a given model and layer $\ell$, S-Space is defined as a three-dimensional linear subspace of hidden representations $h_{\ell,t} \in \mathbb{R}^{d_{\text{model}}}$. It is specific to the model and layer, but shared across all token positions within that layer. The authors focus on middle layers, as early layers remain anchored to input representations and later layers become aligned with next-token prediction.

2.2 Identifying S-Space

The identification procedure uses relative spatial question examples where each example $p$ consists of an image and a prompt comparing a query object with a reference along one axis $g \in \{x, y, z\}$ (e.g., "Is the apple to the left or right of the cup?").

Step 1: Answer contrast definition. For each axis, an answer contrast is defined as the logit difference:

$$s_{p,x} = \mathrm{logit(right)} - \mathrm{logit(left)}, \quad s_{p,y} = \mathrm{logit(above)} - \mathrm{logit(below)}, \quad s_{p,z} = \mathrm{logit(close)} - \mathrm{logit(far)}. \tag{1}$$

Step 2: Gradient computation. The gradient $\nabla_{h_t} s_{p,g}$ specifies the direction along which perturbing the object representation would most increase the corresponding answer contrast.

Step 3: Role-antisymmetric averaging. A role-antisymmetric gradient difference is formed and averaged across prompt templates, original/swapped object orders, and examples:

$$a_{p,g} = \frac{\nabla_{h_q} s_{p,g} - \nabla_{h_r} s_{p,g}}{2}, \quad v_g = \mathrm{unit}(\mathbb{E}_p[a_{p,g}]). \tag{2}$$

The resulting unit vectors establish S-Space: $\mathcal{S} = \mathrm{span}\{v_x, v_y, v_z\}$.

Theoretical intuition: (1) Relative spatial judgments treat query and reference oppositely—a change making the query more right should favor "right," while the same change to the reference should favor "left." (2) Averaging across templates and examples cancels object-identity, prompt-phrasing, and scene-context effects.

2.3 Reading Out S-Space

Reading out requires only a forward pass. The directions are stacked into matrix $V$, and each object token $t$ is projected:

$$V = [v_x, v_y, v_z] \in \mathbb{R}^{d_{\text{model}} \times 3}, \qquad c_t = V^{\top} h_t \in \mathbb{R}^3. \tag{3}$$

For relative spatial judgment, the margin along axis $g$ is computed as:

$$m_g = v_g^{\top}(h_q - h_r) = c_{q,g} - c_{r,g}. \tag{4}$$

The sign of $m_g$ selects the predicted endpoint (positive = right/above/close; negative = left/below/far).

Models Evaluated

• Molmo2-ER
• Qwen3.5-4B
• Qwen3.6-27B

S-Space is identified using constructed relative spatial questions from COCO and evaluated on CV-Bench, EmbSpatial, and Spatial-Tunnel.

---

Empirical Validation / Results

3.1 Accuracy

• Object coordinates: S-Space coordinates read from layer 21 of Molmo2-ER strongly correlate with ground-truth object positions (Spearman correlations, least-squares fits).
• Pairwise differences: Coordinate differences between objects also correlate strongly with ground truth.
• Spatial QA benchmark accuracy: The coordinate-based readout achieves accuracy comparable to the models' direct verbal answers across CV-Bench, EmbSpatial-Bench, and SpatialTunnel, including benchmarks outside the training distribution.

3.2 Causality

Causal interventions involve reading out coordinates, editing them, writing modified values back into hidden representations, and resuming generation.

• Editing S-Space coordinates changes spatial judgments: Coordinate editing substantially alters spatial judgments compared to random editing. The exception is relative questions under common shifts (shifting both objects together preserves relative positions).
• Non-spatial judgments are largely preserved: The same intervention has little effect on unrelated color judgments, indicating S-Space selectively modifies spatial information without broadly disrupting object representations.

3.3 Persistence

S-Space remains readable across different prompt formats:

• Pairwise spatial prompts (e.g., "Is the {query} to the left or right of the {reference}?")
• Single-object prompts (e.g., "Where is the {query}?")
• Non-spatial prompts (e.g., "What colors are the {query} and the {reference}?")

Despite substantial changes in prompt structure and computational context, the coordinates continue to support accurate spatial judgments, demonstrating that S-Space is an intrinsic and persistent organization of the model's representations.

5.1 Perception-Computation Decomposition

Using SpinBench (perspective-taking questions), the authors decompose spatial reasoning into perception and computation:

Key finding: Explicit transformation over S-Space substantially improves perspective-taking accuracy (up to +37.5 percentage points), demonstrating that models encode sufficient spatial information but fail at the computational transformation step.

5.2 Tracking S-Space Through Reasoning

Controlled chain-of-thought analysis on Qwen3.6-27B shows:

• As CoT unfolds, object coordinates become increasingly aligned with the requested viewpoint.
• There is a progressive shift from advantage for rotated readout (early stages) to advantage for direct readout (later stages), suggesting the model internally simulates part of the perspective transformation—though imperfectly.

5.3 Reference-Frame Coupling

Using MMSI-Bench positional tasks, cardinal-direction tokens (north, south, etc.) read from Qwen3.6-27B's chain-of-thought reveal:

• East aligns with right in S-Space, north with above, and northeast reflects both directions.
• This systematic coupling suggests the model partially shares spatial features between cardinal and image-relative reference frames, potentially reflecting biases in web-scale pretraining data.

---

Theoretical and Practical Implications

Theoretical Implications

1. Linear spatial encoding: The discovery of S-Space provides strong evidence that MLLMs encode spatial information linearly in their internal representations, supporting the broader hypothesis of linear concept representation in LLMs.

2. Perception vs. computation distinction: The perception-computation decomposition reveals a critical insight: MLLMs can form capable spatial representations yet fail to draw reliable spatial conclusions from them. This separates two distinct sources of error in spatial reasoning.

3. Emergence from language training: S-Space appears to emerge from the objective of predicting spatial language, binding visible entities to relative positions. The structure inherited from language extends beyond physical space to abstract concepts (political left/right), illustrating both capabilities and limitations of large-scale language training.

4. Reference-frame coupling: The systematic coupling between cardinal and image-relative directions suggests models do not maintain fully independent coordinate systems, potentially explaining reference-frame confusion errors.

Practical Implications

1. Diagnostic tool: S-Space can serve as a diagnostic instrument to identify where spatial reasoning fails (perception vs. computation), enabling targeted improvements.

2. Architectural guidance: The findings suggest future systems need not rely on a monolithic MLLM to perform every spatial operation internally. Instead, agentic spatial reasoning can connect dedicated tools for transformation, simulation, and memory, turning capable perception into reliable reasoning and action.

3. Intervention opportunities: The ability to edit S-Space coordinates offers a potential mechanism for correcting spatial errors or injecting spatial priors.

---

Conclusion

The paper makes several key contributions:

1. First systematic identification of an internal spatial workspace in MLLMs, revealing how spatial information is encoded, updated, and causally used within internal representations.

2. A simple gradient-based method for identifying S-Space, enabling accurate and generalizable continuous coordinate readouts along fixed spatial directions.

3. S-Space as a diagnostic tool for spatial reasoning failures, separating spatial perception from subsequent computation and exposing limitations in how models use available spatial information.

The main takeaway is that MLLMs often possess accurate perceptual spatial representations yet fail in the spatial transformations required to reason reliably across viewpoint changes. This suggests that scaling alone may not resolve spatial reasoning failures; instead, targeted improvements in spatial computation capabilities, or hybrid systems that combine perception with external transformation tools, may be necessary. Future work could explore whether S-Space generalizes to other modalities, whether it can be used to improve spatial reasoning through targeted interventions, and how the biases inherited from language training can be mitigated.