# Learning to Think and Explore in 3D World

> ACT3D learns active 3D exploration via outcome-driven reinforcement learning, achieving 53.2% average accuracy across six benchmarks and surpassing Gemini 3.6 Flash with 50–70% fewer input frames.

- **Source:** [arXiv](https://arxiv.org/abs/2610.21783)
- **Published:** 2026-10-20
- **Permalink:** https://picx.dev/p/JQ9pUC
- **Whiteboard:** https://picx.dev/p/JQ9pUC/image

## Summary

## Summary (Overview)

- **ACT3D** is a vision-language model (VLM) that learns to *actively explore* 3D environments through sequential egocentric interactions, rather than receiving complete 3D representations as input. The model iteratively reasons, navigates, and reorients its viewpoint before committing to a final answer.
- The framework uses a **two-stage training procedure**: trajectory-supervised fine-tuning (SFT) to establish basic interaction competence, followed by **outcome-driven reinforcement learning** (GRPO) where rewards are driven primarily by final-answer correctness — with no direct supervision over intermediate viewpoints or actions.
- ACT3D achieves an **average accuracy of 53.2%** across six 3D benchmarks, improving its base Qwen model by **22.0%** and surpassing proprietary **Gemini 3.6 Flash** by 1.0%, while using 50–70% fewer input frames than existing open-source 3D methods.
- The model exhibits **adaptive, question-conditioned exploration**: performance gains over the base model increase from 0.6% on questions answerable from the initial view to 16.3% on those requiring extensive exploration.
- RL-trained policies demonstrate emergent behaviors including searching for occluded targets, avoiding unnecessary exploration when evidence is visible, and acquiring complementary viewpoints for spatial comparisons.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Traditional 3D vision-language models assume observations are **specified a priori**, providing models with complete RGB-D scans, multi-view images, videos, or explicit 3D representations (point clouds, meshes). However, under **partial observability**, a fundamental challenge remains: the model must not only reason over currently available evidence, but also determine **how to act** to acquire missing evidence.

The central research question is: *Can a vision-language model learn to actively explore and acquire task-relevant visual evidence as an integral part of the 3D reasoning process?*

### Problem Formulation as POMDP

Reasoning in 3D is formalized as a **partially observable Markov decision process (POMDP)**. A task instance $w = ( \mathcal { M } , q , y ^ { \star } ) \sim \mathcal { D }$ consists of:

- A reconstructed 3D scene $\mathcal{M}$
- A question $q$
- Ground-truth answer $y^{\star} = g(\mathcal{M}, q) \in \mathcal{Y}$

At step $t$, the environment state is $\boldsymbol{s}_t = (\mathcal{M}, \boldsymbol{c}_t)$, where $c_t = (e_t, f_t)$ specifies camera position $e_t \in \mathbb{R}^3$ and viewing direction $f_t \in \mathbb{S}^2$. The camera up vector is held parallel to $\hat{z}$ (vertical axis), giving $c_t$ **five degrees of freedom** (no roll).

The policy observes rendered egocentric views $o_t = \mathrm{Render}(\mathcal{M}, c_t)$ and the resulting camera configuration $c_t$, while the scene $\mathcal{M}$ remains fixed and hidden throughout the episode.

### Action Space

**Relative parameterization** (adopted): A navigation action $a_t = (\xi_t, n_t) \in \mathcal{A}_{\mathrm{nav}} = \mathcal{U} \times \{1, \dots, 8\}$ pairs a motion primitive with an integer multiplier, giving $|\mathcal{A}_{\mathrm{nav}}| = 80$ navigation actions. The set $\mathcal{U}$ contains:
- Six translations: forward/backward, left/right, up/down
- Four rotations: turn_left/turn_right, turn_up/turn_down

Each primitive has a unit step $\delta_\xi$ (1 m for forward/backward, 0.6 m for other translations, 30° for rotations). Camera dynamics are deterministic:

$$
(e_{t+1}, f_{t+1}) = \left\{ \begin{array}{l l} (\Pi_{\mathcal{B}} (e_{t} + n_{t} \delta_{\xi_{t}} u_{t}), f_{t}), & a_{t} \text{ is a translation}, \\ (e_{t}, \mathrm{Rot}_{k_{t}}(n_{t} \delta_{\xi_{t}}) f_{t}), & a_{t} \text{ is a rotation}, \end{array} \right.\tag{1}
$$

where $\mathrm{Rot}_k(\alpha)$ rotates a vector by angle $\alpha$ about unit axis $k$, and $\Pi_{\mathcal{B}}$ projects onto the scene's bounding box (no collision checking).

**Absolute parameterization** (compared but not adopted): actions name target pose directly as $(p_t, \ell_t)$, requiring prediction of coordinates in a global reference frame that is not directly observed.

The full action set augments navigation with terminal answer actions:

$$
\bar{\mathcal{A}} = \mathcal{A}_{\mathrm{nav}} \cup \mathcal{A}_{\mathrm{stop}}.\tag{3}
$$

### Objective

The policy conditions on the complete interaction history $h_t$ (including intermediate reasoning $z_t$) and is optimized for expected finite-horizon, undiscounted return:

$$
\max_{\theta} J(\theta) = \mathbb{E}_{w \sim \mathcal{D}} \mathbb{E}_{\tau \sim \pi_{\theta} \otimes \mathcal{E}} [R(\tau)],\tag{5}
$$

where $\mathcal{E}$ is the deterministic interactive environment and $R$ is the reward function.

---

## Methodology

### Stage 1: Trajectory-Supervised Policy Initialization (SFT)

- **Trajectory curation**: GPT-5.4 generates reason–act–observe trajectories in the environment. After filtering (correct answers, valid interactions, well-formed responses, sufficient visible geometry), the corpus contains **5,712 trajectories, 20,603 rendered observations, and 830 scenes**.
- **Supervised objective**: Minimize negative log-likelihood over model-generated tokens only (reasoning, actions, final answer), masking environment observations and prompt tokens:

$$
\mathcal{L}_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{\tau^{\star} \sim \mathcal{D}_{\mathrm{traj}}} \sum_{\ell \in \mathcal{I}(\tau^{\star})} \log \pi_{\theta}(\tau_{\ell}^{\star} \mid \tau_{<\ell}^{\star}).\tag{7}
$$

### Stage 2: Outcome-Driven Reinforcement Learning (GRPO)

**Training data**: RL requires only question–answer pairs and valid initial viewpoints, enabling broader data collection: **51,543 questions** from ScanQA, MP3D, and generated QAs over ScanNet and ARKitScenes.

**Verifiable Reward**: For a rollout $\tau$ with final answer $y$:

$$
R(\tau) = \underbrace{\lambda_{\text{judge}} R_{\text{judge}} + \lambda_{\text{EM}} R_{\text{EM}}}_{\text{answer correctness}} + \underbrace{\lambda_{\text{int}} R_{\text{judge}} c(\tau)}_{\text{interaction reward}} - \underbrace{\lambda_{\text{fmt}} R_{\text{fmt}}}_{\text{output format}},\tag{8}
$$

where:
- $R_{\text{judge}} \in \{0,1\}$: LLM judge verdict on semantic equivalence of answer to ground truth
- $R_{\text{EM}} \in \{0,1\}$: exact match with annotated GT answers
- $R_{\text{fmt}} \in \{0,1\}$: malformed output indicator

The interaction credit term:

$$
c(\tau) = \left\{ \begin{array}{l l} N_{\max}, & R_{\text{gnd}} = 1, \\ \min\{n_{\text{int}}, N_{\max}\}, & \text{otherwise}, \end{array} \right.\tag{9}
$$

where $n_{\text{int}}$ is the number of interactions and $R_{\text{gnd}}$ indicates whether the annotated answer object was visible in some rendered view. The design **pays for evidence rather than for looking**: a correct rollout that observes the answer object receives full interaction credit at once, while further calls earn nothing once the needed view is obtained.

**Policy optimization** uses GRPO with trajectory-level advantages:

$$
\hat{A}_i = \frac{R(\tau_i) - \mathrm{mean}_j R(\tau_j)}{\mathrm{std}_j R(\tau_j)}.\tag{10}
$$

The same advantage is assigned to all model-generated tokens, with environment observations excluded from optimization.

### Training Configuration

- Base model: **Qwen3.5-9B**
- Two configurations: ScanQA-only (25,563 questions) and mixed-data (51,543 questions from ScanQA, MP3D, and curated QA)
- SFT: 2 epochs, batch size 16, learning rate $10^{-5}$
- GRPO: 64 prompts/step, mini-batch 16, $G=8$ rollouts per prompt, learning rate $2 \times 10^{-6}$, temperature 0.7, $\epsilon = 0.2$, $\beta = 0.005$
- Reward weights: $(\lambda_{\text{judge}}, \lambda_{\text{EM}}, \lambda_{\text{fmt}}, \lambda_{\text{int}}) = (0.3, 0.2, 0.2, 0.05)$, $N_{\max} = 12$
- Maximum 12 interaction steps per episode, context length 24,576 tokens

---

## Empirical Validation / Results

### Main Results

ACT3D (mixed-data) achieves the highest average accuracy of **53.2%** across six benchmarks, surpassing all evaluated baselines:

| Model | ScanQA | SQA3D | SPB MV | SPB SI | MP3D | ARKit | Avg. |
|---|---|---|---|---|---|---|---|
| **Passive perception** | | | | | | | |
| Video-R1-7B | 30.8 | 41.1 | 42.0 | 39.6 | 42.6 | 63.4 | 43.2 |
| ViLaSR-7B | 33.3 | 47.6 | 52.6 | 43.4 | 34.6 | 66.0 | 46.2 |
| SpatialLadder-3B | 29.3 | 40.4 | 61.2 | 45.2 | 45.2 | 58.7 | 46.7 |
| InternVL3.5-38B | 36.2 | 47.0 | 36.9 | 38.7 | 51.1 | 75.4 | 47.6 |
| SpaceR-7B | 31.6 | 44.7 | 52.6 | 41.0 | 52.0 | 70.1 | 48.7 |
| Ross3D | 50.7 | 62.6 | 32.0 | 31.9 | 55.2 | 71.5 | 50.7 |
| **Active interaction** | | | | | | | |
| Qwen2.5-VL-72B | 16.8 | 26.8 | 20.8 | 19.1 | 26.7 | 46.8 | 26.2 |
| Qwen3.5-9B | 36.7 | 33.5 | 21.3 | 24.7 | 30.9 | 40.1 | 31.2 |
| Qwen3.5-27B | 27.5 | 40.2 | 33.2 | 31.4 | 33.8 | 33.8 | 33.3 |
| InternVL3.5-38B | 30.9 | 41.5 | 43.2 | 41.6 | 50.1 | 55.8 | 43.9 |
| Gemini 3.6 Flash | 43.3 | 53.8 | 45.5 | 44.4 | 53.4 | 73.0 | 52.2 |
| **ScanQA only** | | | | | | | |
| SFT init | 39.4 | 46.0 | 47.3 | 41.2 | 44.7 | 68.3 | 47.8 |
| **ACT3D (ours)** | **45.5** | **49.8** | **47.6** | **42.3** | **50.7** | **67.6** | **50.6** |
| **Mixed data** | | | | | | | |
| SFT init | 39.3 | 47.4 | 39.2 | 37.7 | 61.8 | 65.9 | 48.6 |
| **ACT3D (ours)** | **42.1** | **49.0** | **47.3** | **37.4** | **63.3** | **80.0** | **53.2** |

### Learning Dynamics

- RL improves average accuracy from 48.6% to 53.2% (+4.6%) over SFT init in mixed-data setting, with largest gains on ARKit (+14.1%) and SPBench-MV (+8.1%).
- Average number of interactions per episode increases from ~3.3 to ~8.3 during mixed-data training.
- **Cross-domain generalization**: ScanQA-only training improves average accuracy on other five benchmarks from 49.5% to 51.6% (+2.1%), with substantial gains on MP3D (+6.0%) and SQA3D (+3.8%).

### Adaptive Exploration

Performance gains stratified by estimated exploration demand (geometry-based measure independent of model behavior):
- Combined SFT+RL improvements increase from **0.6%** (lowest-demand group) to **16.3%** (highest-demand group)
- RL contributes an additional **10.9%** over SFT in the most demanding group

### Ablation Results

**Initial observation**: Single rendered view achieves 41.0% on ScanQA, closely matching 41.4% with BEV + 4 rendered frames, despite using only one image instead of five.

**Rendering vs. retrieval**: Pose-aligned rendering consistently outperforms nearest-view real-image retrieval across all 13 evaluation checkpoints, achieving **10.6% higher accuracy on average**, highlighting the importance of action–observation consistency.

**Action space**: Relative actions achieve 41.4% accuracy vs. 36.7% for absolute pose prediction.

**Supervised initialization**: Without SFT, RL alone improves accuracy from 28.9% to 41.4% after 470 steps but with only ~2.0 interactions/episode. SFT-initialized RL achieves 42.7% with 8.7 interactions/episode, suggesting SFT facilitates adaptation and accelerates RL optimization.

**Interaction reward**: Adding per-interaction reward increases average interactions from 5.1–6.3 to 9.4–10.1 per episode, improving held-out benchmarks (SPBench-MV +6.4%) but slightly reducing training-domain performance (0.3–2.3%).

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Outcome-based RL suffices for learning exploration**: The paper demonstrates that task-level supervision alone — without any intermediate viewpoint or action supervision — can induce task-directed visual information acquisition during 3D reasoning. This addresses a fundamental question about the sufficiency of outcome-based credit assignment for learning active perception policies.

2. **Emergent question-conditioned exploration**: The model learns adaptive strategies that are conditioned on the question and current evidence, including searching for occluded targets, avoiding unnecessary exploration, and acquiring complementary viewpoints — behaviors not explicitly programmed or supervised.

3. **Partial observability as a training signal**: By treating the 3D scene as an external interactive environment rather than providing complete representations, the formulation preserves native visual encoding of VLMs (single images) while enabling learning of evidence acquisition as an integral part of reasoning.

### Practical Implications

1. **Efficiency**: ACT3D achieves superior accuracy while using 50–70% fewer input frames and lower-resolution observations (640×480) than existing methods, suggesting that *active* evidence acquisition can be more efficient than *passive* provision of complete observations.

2. **Generalization**: Gains extend beyond training domains, with an average improvement of 5.9% over supervised initialization across four out-of-domain benchmarks, indicating transferable exploration capabilities.

3. **Interface design principles**: The ablation studies provide practical guidance — action–observation consistency matters more than raw visual fidelity (rendering > retrieval), relative camera-centric actions outperform absolute coordinate prediction, and single-view initialization is sufficient.

---

## Conclusion

ACT3D presents a framework for learning to think and explore in 3D by treating 3D representations as **interactive environments** rather than assumed complete inputs. Starting from a single egocentric observation, the model iteratively reasons, acts, and acquires visual evidence through interaction. The two-stage training — trajectory-supervised initialization followed by verifiable RL — enables the model to learn not only *how to reason* over visual evidence, but also *which evidence to acquire* and *when*.

Key contributions:
1. A POMDP formulation for VLM-based 3D reasoning under partial observability with a relative, camera-centric action space
2. A two-stage training procedure combining SFT initialization with outcome-driven GRPO
3. Empirical demonstration of emergent, question-conditioned exploration behaviors
4. State-of-the-art results across six 3D benchmarks with improved efficiency

**Future directions** suggested by this work include:
- Extending beyond direct camera control to full embodied navigation with physical constraints
- Scaling to larger, more diverse environments and question types
- Investigating whether RL can learn even more sophisticated exploration strategies with longer horizons or richer action spaces
- Exploring the interaction between exploration and higher-level spatial reasoning capabilities

---

_Markdown view of https://picx.dev/p/JQ9pUC, served by PicX — AI-generated visual whiteboard summaries of research papers._