RoTA-Bench: Evaluating 3D Spatial Reasoning through Interactive Projection Alignment

Summary (Overview)

• RoTA-Bench is a new benchmark that evaluates whether vision-language models (VLMs) can convert 2D observations into 3D understanding that supports precise action, through an interactive 3D-to-2D projection alignment task.
• The benchmark contains 674 canonical tasks with a stratified 100-task test set, automatically constructed via a pipeline that extrudes 2D icons into 3D objects and uses reversible initialization to mathematically guarantee solvability.
• Key finding: A substantial human–model gap persists—GPT-5.6-Sol succeeds on only 17% of tasks, whereas 20 undergraduate participants achieved a 100% success rate across 400 task runs.
• Key finding: Image-space and pose-space progress can diverge—at matched states, GPT-5.6-Sol more often improves silhouette overlap while worsening pose alignment than humans (19.4% vs. 10.9%).
• Key finding: Fine-grained control is sensitive to interaction history—removing it leaves 96.8% of the model's actions at a coarse 30° step size and sharply reduces task success.

---

Introduction and Theoretical Foundation

Background and Motivation

The paper is grounded in classical cognitive science research establishing spatial perception as a foundational cognitive ability (Carroll, 1993), essential for:

• Maintaining object constancy across viewpoints (Shepard & Metzler, 1971)
• Recovering three-dimensional geometry from two-dimensional projections (Richards et al., 1987)

As vision-language models are increasingly deployed in interactive and embodied settings (visual question answering, interactive agents, geometry-aware navigation), they must translate spatial understanding into physical actions. However, current models struggle with reasoning about 3D positions, orientations, and spatial relationships.

Gap in Existing Benchmarks

Design Rationale

The authors formulate 3D orientation alignment as a projection-matching task: rotating a 3D object until its 2D silhouette matches a target image. This design offers two advantages:

1. Out-of-plane rotations induce nonlinear silhouette deformations, aspect changes, and self-occlusion—an inherently visual challenge resistant to textual representation.
2. Pure rotation cleanly decouples spatial orientation reasoning from extraneous physical dynamics (translational drift, grasp affordances, contact friction).

To reduce reliance on semantic shortcuts, objects are synthesized from 2D vector icons rather than common 3D CAD repositories, producing abstract, novel 3D geometries that avoid pre-trained priors.

---

Methodology

Task Formulation

Each task is defined by a rigid 3D object $G \subset \mathbb{R}^3$ centered at the origin, an initial pose $R_0 \in \mathrm{SO}(3)$, and a canonical target pose $R^\star = I_3$. The environment provides an observation triplet at each step $t$:

$$O_t = (I^\star, V_t, I_t), \quad I^\star = \Pi(G, R^\star), \quad I_t = \Pi(G, R_t) \tag{1}$$

where $I^\star$ is the invariant target silhouette, $I_t$ is the current projection silhouette, and $V_t$ is a depth-shaded auxiliary view from the same viewpoint.

Action Space

The agent selects from 24 discrete rotation actions spanning:

• Three principal camera axes: pitch (up/down), yaw (left/right), in-plane roll (clockwise/counterclockwise)
• Four step sizes: $\Delta\theta \in \{1^\circ, 5^\circ, 10^\circ, 30^\circ\}$
• Plus a terminal STOP command

The pose updates via:

$$R_{t+1} = Q(a_t) R_t \tag{2}$$

Success Criterion

Projection IoU between current and target silhouettes:

$$s_t = \operatorname{IoU}(I_t, I^\star) = \frac{|I_t \cap I^\star|}{|I_t \cup I^\star|} \tag{3}$$

Success requires $s_t \geq \tau$ with $\tau = 0.90$. The goal set accounts for geometric symmetries:

$$\mathcal{R}_{\mathrm{goal}}(G, I^\star) = \{R \in \mathrm{SO}(3) \mid \mathrm{IoU}(\Pi(G, R), I^\star) \geq \tau\} \tag{4}$$

Dataset Construction Pipeline

1. Task Generation: 2D vector icons from the Game-icons library are filtered, rasterized, and extruded into 3D objects under a spherical envelope:

$$h(x, y) = \sqrt{\max(r^2 - x^2 - y^2, 0)}, \quad z \in [-h(x, y), h(x, y)] \tag{5}$$

2. Reversible Initialization: Initial poses are generated via $L \sim \text{Uniform}\{6, \ldots, 16\}$ discrete 30° rotations. Because every rotation in $\mathrm{SO}(3)$ is invertible, executing the reversed sequence guarantees 100% reachability and solvability.

3. Reference Search: Pose-guided beam search and 30°-restricted BFS discover shorter reference paths.

4. Dataset Construction: A stratified 100-task test set is frozen via Hamilton's method and deterministic hash sorting.

Evaluation Metrics

• Task Success Rate: Fraction of episodes reaching IoU ≥ 0.90
• Mean Final IoU and mean rotation steps
• Pose error (diagnostic, not provided to model):

$$e_t = \min_{S \in \mathcal{G}} d_{\mathrm{SO}(3)}(R_t, R^\star S) = \min_{S \in \mathcal{G}} \arccos\left(\frac{\operatorname{Tr}(R_t (R^\star S)^\top) - 1}{2}\right) \tag{6}$$

• Effective Direction Rate: Fraction of rotations choosing a direction that can reduce pose error
• Best Legal Angle Rate, Angle Gap, Angle Loss

Evaluated Systems

• Nine VLMs: GPT-5.6-Sol, GPT-5.6-Terra, GPT-5.6-Luna, Gemini 3.6 Flash, Claude Opus 5, Claude Sonnet 5, Qwen 3.7 Plus, GPT-6-Astra, SpatioLM-Understanding-InternVL3.5
• Two agent frameworks: ReAct and Magentic (based on GPT-5.6-Sol)
• 20 human participants (each completing a consecutive 20-task block)
• Baselines: Uniform random, Random with IoU rollback, IoU-greedy, Pose-greedy

---

Empirical Validation / Results

Main Results

Table 1 (abbreviated, sorted by Final IoU):

Key observations:

• Claude Sonnet 5, Claude Opus 5, and Qwen 3.7 Plus fail entirely (0.0% success), with final IoU barely exceeding uniform random exploration.
• SpatioLM suffers a catastrophic policy collapse: 91.12% of its actions degenerate into YAW_POS 10°, yielding the lowest final IoU (0.463).
• The IoU-greedy baseline succeeds on only 39% of tasks, showing greedy projection overlap maximization is insufficient.
• The pose-greedy baseline solves every task (100%), showing reliable 3D pose error access is sufficient.

Diagnostic Analysis

• Direction selection: GPT-5.6-Sol selects effective directions only 58.75% of the time vs. 83.91% for humans, barely above uniform random (53.78%).
• Angle precision: GPT-5.6-Sol has a mean angle gap of 6.49° vs. 2.99° for humans, with an angle loss of 2.25° per step vs. 0.96° for humans.

Ablation Study

Table 2 — Input ablation for GPT-5.6-Sol on 30 test tasks:

Key finding: Removing interaction history reduces success from 5/30 to 0/30 tasks and final IoU to near-random levels. The effective-direction rate remains nearly unchanged (54.85%–56.52%), but angle selection changes markedly—30° rotations rise to 96.76% and 1° adjustments fall to 0.16%. History helps the model choose how far to rotate more than which way to rotate.

Action Association Analysis

• GPT-5.6-Sol's rotation angles correlate more strongly with projection IoU ($\rho = -0.536$) than with 3D pose error ($\rho = 0.368$).
• Human angles correlate strongly with both ($\rho = -0.696$ for IoU, $\rho = 0.751$ for pose error).
• At 211 matched states (same pose, similar visual coverage), humans execute more DC actions (33.18% vs. 23.70%—accepting transient IoU drops to reduce pose error), while GPT-5.6-Sol executes nearly twice as many UF actions (19.43% vs. 10.90%—improving IoU at the cost of pose error).

---

Theoretical and Practical Implications

Theoretical Implications

1. 2D-to-3D inference remains a fundamental challenge for VLMs: The results demonstrate that even frontier models struggle to infer 3D orientation from 2D projections, despite strong performance on static spatial reasoning benchmarks. This suggests current models may rely on pattern matching rather than genuine mental rotation or 3D reconstruction.

2. Image-space and pose-space dissociation: The divergence between IoU improvement and pose error reduction reveals that models optimize visible metrics without maintaining a coherent 3D internal model—a critical distinction for embodied AI.

3. Action granularity requires temporal memory: The ablation showing that interaction history is crucial for fine-grained angle selection suggests that models lack an implicit sense of "how much correction is needed" without explicit memory of prior actions and observations.

Practical Implications

1. Benchmark design: RoTA-Bench provides a scalable, automatically generated evaluation protocol that isolates 3D spatial reasoning from high-level planning, offering a cleaner signal for model improvement.

2. Agent development: The failure of ReAct and Magentic frameworks suggests that adding reasoning scaffolding alone does not solve perceptual-motor challenges; the bottleneck is perceptual inference, not planning.

3. Training signal: The 574 reserved tasks provide a scalable environment for reinforcement learning fine-tuning to study whether interactive training improves closed-loop 3D control.

---

Conclusion

RoTA-Bench evaluates whether vision-language models can turn 2D observations into 3D understanding that supports precise action through interactive projection alignment. The benchmark's automated pipeline constructs silhouette-preserving 3D objects with mathematically guaranteed solvability, yielding 674 tasks and a stratified 100-task test set.

The evaluations reveal three main findings:

1. A substantial human–model success gap (17% vs. 100% success)
2. Image-space and pose-space progress can diverge, with models more often improving silhouette overlap while worsening pose alignment
3. Fine-grained control is sensitive to interaction history, with its removal causing models to default to coarse 30° rotations

These findings highlight the fundamental challenge of converting visual evidence into precise 3D control—a capability that goes beyond recognizing or matching individual images.

Future Directions

• Expanding task and icon diversity via generative models
• Introducing continuous rotation control, external occlusions, and multi-object alignment/assembly tasks
• Using the automatically generated tasks for RL fine-tuning to study whether interactive training improves the use of visual feedback for precise 3D control