# Player-1: Learning to Play from Synthetic Experience

> PLAYER-1's synthetic-experience RL training lifts game-agent success on MineDojo-300 from 10.54% to 38.33%, tripling the strongest baseline and generalizing to unseen games.

- **Source:** [arXiv](https://arxiv.org/abs/2610.20584)
- **Published:** 2026-10-16
- **Permalink:** https://picx.dev/p/OauYde
- **Whiteboard:** https://picx.dev/p/OauYde/image

## Summary

# PLAYER-1: Learning to Play from Synthetic Experience

## Summary (Overview)

- **PLAYER-1** is a post-training framework that enables game-control agents to learn beyond supervised demonstrations through synthetic experience, combining curated gameplay trajectories with reinforcement learning in synthetic environments.
- The framework extends the computer-use action space with **relative pointer motion, sustained inputs, and composite actions**, enabling dynamic game control capabilities.
- **PLAYER-1-GYM** provides sixteen synthetic 2D and 3D environments organized around five shared control-demand families (aiming, combat, driving, navigation, UI interaction) with programmatically verifiable rewards and extensive randomization.
- Results show **Player-1-9B achieves 38.33% success on MineDojo-300** (up from 10.54% after SFT only), surpassing the strongest evaluated baseline (12.63%) by 3x, with gains also generalizing to ViZDoom-10 and GameWorld outside the RL training suite.
- Learning from synthetic experience is shown to be scalable with environment count, with composite-action usage rising from 64.7% to 93.2% during RL training.

## Introduction and Theoretical Foundation

The paper addresses a fundamental limitation in current game-agent learning pipelines: **demonstrations cover only a limited portion of states** an agent may encounter in open-ended tasks, while **scalable environments for interactive training remain difficult to obtain**. Despite differences in appearance and rules, the authors observe that many games share **recurring control demands** such as precise aiming, sustained movement, and coordinated inputs.

The central research question is: *Can synthetic experience advance game-control capabilities beyond supervised initialization, with gains that generalize to games outside the RL training suite?*

Three key challenges motivate the approach:

1. **Action space unification**: Gameplay demonstrations recorded with heterogeneous input conventions must be converted into a shared action space paired with task instructions and visual reasoning.
2. **Feedback beyond demonstrations**: Learning requires feedback on states reached by the policy's own actions, but programmatic rewards across diverse commercial games are difficult to scale.
3. **Multi-environment generalization**: Experience from environments with different control demands and dynamics must be incorporated into a shared policy that generalizes.

The theoretical foundation builds on prior work in computer-use agents (CUAs), vision-language models (VLMs), and reinforcement learning with verifiable rewards (RLVR), positioning PLAYER-1 as advancing beyond systems like SIMA 2, GameWAM, and NitroGen by providing open availability and systematic evaluation both in-domain and on held-out games.

## Methodology

### Extended Action Space

The policy receives task instruction $q$, screenshot $I_t$, and interaction history $C_H(h_{<t})$:

$$(z_t, \hat{a}_t) \sim \pi_\theta(\cdot | q, I_t, C_H(h_{<t}))$$

Three key extensions to the computer-use action space:

**1. Relative pointer motion:**

$$\mathcal{A}_{\text{pointer}} = \underbrace{\{\text{absolute}(x, y)\}}_{\text{Retained CUA coordinates}} \cup \underbrace{\{\text{relative}(\Delta x, \Delta y)\}}_{\text{Relative-motion extension}}$$

Absolute coordinates satisfy $x, y \in [0, 1000]$, while relative displacements $\Delta x, \Delta y \in [-1000, 1000]$ are normalized by screen dimensions.

**2. Sustained inputs:**

$$\mathcal{A}_{\text{press}}(b) = \underbrace{\{\text{tap/click}(b)\}}_{\text{Retained CUA taps/clicks}} \cup \underbrace{\{\text{hold}(b; \Delta t)\}}_{\text{Sustained extension}}$$

**3. Composite actions:**

$$\mathcal{A}_{\text{step}} = \underbrace{\{\text{wait}\} \cup \mathcal{A}_{\text{keyboard}} \cup \mathcal{A}_{\text{mouse}}}_{\text{Individual CUA inputs}} \cup \underbrace{\{\text{input}_1 \| \cdots \| \text{input}_m\}}_{\text{Composite-action extension}}$$

### Supervised Initialization

- **6,044 curated gameplay trajectories** comprising 295,097 interaction turns from public datasets and in-house recordings
- Controls aligned to shared action intervals with source-specific mouse calibration
- A VLM annotates recorded actions with **task instructions and per-turn visual reasoning**
- SFT examples take the form $(q, I_t, C_H(h_{<t}), z_t, a_t)$ with a $2+2$ history setting (two recent visual-action turns, two earlier action-only turns)
- **Action-balanced sampling** increases exposure to less frequent controls and composite actions

### PLAYER-1-GYM: Multi-Environment RL

Sixteen synthetic environments across five task families:
- **Aiming (2)**: 2D aiming, 3D aiming
- **Combat (5)**: 2D shooting, 3D combat arena, 3D spear duel, 3D sword duel, 3D melee arena
- **Driving (4)**: 2D lane driving, 2D circuit driving, 3D road driving, 3D lane driving
- **Navigation (4)**: 2D collect dodge, 2D platforming, 3D navigation, 3D chase
- **UI interaction (1)**: 2D UI crafting

Four complementary randomization dimensions: **appearance** (colors, lighting, assets), **camera** (projection, placement, orientation), **scene** (layouts, spawns, configurations), and **control** (input-response gains).

**Per-environment advantage normalization** for PPO:

$$\widetilde{A}_{i,j} = \frac{A_{i,j} - \mu_e}{\sqrt{\sigma_e^2 + \epsilon}}, \quad \mu_e = \frac{\sum_{(i,j) \in \mathcal{S}_e} A_{i,j}}{N_e}, \quad \sigma_e^2 = \frac{\sum_{(i,j) \in \mathcal{S}_e} (A_{i,j} - \mu_e)^2}{N_e - 1}$$

where $i$ indexes turns, $j$ indexes response tokens, and $\mathcal{S}_e$ contains response-token indices in the rollout batch for environment $e$.

## Empirical Validation / Results

### MineDojo-300 Results

**Table 2: Task success and coverage on MineDojo-300** (8 trials per task):

| Method | Combat (%) | Harvest (%) | Overall (%) | ≥1 of 8 | ≥2 of 8 | ≥4 of 8 |
|--------|-----------|------------|------------|---------|---------|---------|
| GameWAM | 14.00 | 9.88 | 12.63 | 155/300 | 79/300 | 16/300 |
| Qwen3.5-9B | 11.75 | 4.13 | 9.21 | 106/300 | 59/300 | 13/300 |
| Player-1-9B-SFT | 12.50 | 6.63 | 10.54 | 120/300 | 63/300 | 20/300 |
| Player-1-9B | **52.31** | **10.38** | **38.33** | 209/300 | 173/300 | **136/300** |
| Player-1-35B-A3B | **55.56** | **12.00** | **41.04** | 224/300 | 182/300 | **142/300** |

Key findings: SFT raises success from 9.21% to 10.54% (9B), RL further raises it to 38.33% — a **27.79 percentage point gain over SFT**. Tasks completed in ≥4 of 8 trials increase from 20 to 136.

### ViZDoom-10 Results

**Table 3: First-person control on ViZDoom-10** (mean scaled return):

| Method | Precision shooting | Goal navigation | Mobile combat | Survival | Overall |
|--------|-------------------|-----------------|---------------|----------|---------|
| GameWAM | 0.1175 | -0.1285 | 1.1860 | 0.1424 | 0.3938 |
| Player-1-9B-SFT | 0.4403 | 0.1839 | 0.5293 | 0.5423 | 0.4361 |
| **Player-1-9B** | **1.0772** | **0.5974** | 0.3033 | **0.7927** | **0.6922** |
| **Player-1-35B-A3B** | **1.0423** | 0.2628 | **0.5960** | **0.8979** | **0.7236** |

Player-1-35B-A3B surpasses GameWAM (strongest external baseline) by **83.7%** on overall scaled return.

### GameWorld Results

**Table 4: Browser-game control on GameWorld** (3 trials × 170 tasks × 34 games):

| Method | Success (%) | Progress (%) |
|--------|------------|--------------|
| Qwen3.5-35B-A3B | 13.14 | 33.49 |
| Player-1-9B | 15.88 | 34.45 |
| **Player-1-35B-A3B** | **17.25** | **38.79** |

### Learning Dynamics

- Both mean episode return and game score rise across RL training steps, with gains largest early in training
- **SFT→RL trajectory** rises from 17.67% to 51.67% Minecraft success, while **Base→RL** rises only from 12.67% to 16.67% (a 35.00-percentage-point gap at step 100)
- Different environments show varied learning curves: simple tasks (2D aiming) improve rapidly then level off, while 2D circuit driving shows delayed gains

### Action Composition Evolution

Composite-action usage rises from **64.7% at RL start to 85.4% and 93.2%** after 50 and 100 RL steps respectively — occurring without explicit reward for composition, suggesting task-driven RL reinforces coordinated inputs.

## Theoretical and Practical Implications

**For game-agent research:** The work establishes synthetic experience as an effective resource for learning transferable game control without online training in each target game. The shared action space and environment-suite design provide a blueprint for scalable post-training of game agents.

**For computer-use agents:** Extending the CUA action space with relative pointer motion, sustained inputs, and composite actions demonstrates a path toward general digital control in dynamic, real-time environments beyond traditional GUI interaction.

**For RL methodology:** The per-environment advantage normalization (Eq. 1) addresses reward-scale heterogeneity across multi-environment training, while environment scaling analysis (2→16 environments) shows broader coverage improves transfer.

**For practical deployment:** The demonstration of control in Watch Dogs 2 — a commercial game unseen during both SFT and RL — shows the framework's potential for real-world game automation and testing applications.

## Conclusion

PLAYER-1 demonstrates that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite. Key contributions include:

1. A **post-training framework** combining SFT on curated demonstrations with multi-environment RL
2. **PLAYER-1-GYM**: sixteen synthetic environments with verifiable rewards and extensive randomization
3. **Empirical gains**: 27.8 and 17.7 percentage-point improvements on MineDojo-300 for 9B and 35B-A3B models respectively, with generalization to ViZDoom and GameWorld
4. **Scalability**: broader synthetic environment coverage improves transfer
5. **Emergent behavior**: composite-action usage increases during RL without explicit reward

Future directions suggested by the work include expanding synthetic environment coverage, exploring additional control-demand families, and investigating further scaling of both models and environment suites. The findings support synthetic experience as an effective and scalable resource for developing transferable game-control capabilities.

---

_Markdown view of https://picx.dev/p/OauYde, served by PicX — AI-generated visual whiteboard summaries of research papers._