PLAYER-1: Learning to Play from Synthetic Experience

Summary (Overview)

• PLAYER-1 is a post-training framework that enables game-control agents to learn beyond supervised demonstrations through synthetic experience, combining curated gameplay trajectories with reinforcement learning in synthetic environments.
• The framework extends the computer-use action space with relative pointer motion, sustained inputs, and composite actions, enabling dynamic game control capabilities.
• PLAYER-1-GYM provides sixteen synthetic 2D and 3D environments organized around five shared control-demand families (aiming, combat, driving, navigation, UI interaction) with programmatically verifiable rewards and extensive randomization.
• Results show Player-1-9B achieves 38.33% success on MineDojo-300 (up from 10.54% after SFT only), surpassing the strongest evaluated baseline (12.63%) by 3x, with gains also generalizing to ViZDoom-10 and GameWorld outside the RL training suite.
• Learning from synthetic experience is shown to be scalable with environment count, with composite-action usage rising from 64.7% to 93.2% during RL training.

Introduction and Theoretical Foundation

The paper addresses a fundamental limitation in current game-agent learning pipelines: demonstrations cover only a limited portion of states an agent may encounter in open-ended tasks, while scalable environments for interactive training remain difficult to obtain. Despite differences in appearance and rules, the authors observe that many games share recurring control demands such as precise aiming, sustained movement, and coordinated inputs.

The central research question is: Can synthetic experience advance game-control capabilities beyond supervised initialization, with gains that generalize to games outside the RL training suite?

Three key challenges motivate the approach:

1. Action space unification: Gameplay demonstrations recorded with heterogeneous input conventions must be converted into a shared action space paired with task instructions and visual reasoning.
2. Feedback beyond demonstrations: Learning requires feedback on states reached by the policy's own actions, but programmatic rewards across diverse commercial games are difficult to scale.
3. Multi-environment generalization: Experience from environments with different control demands and dynamics must be incorporated into a shared policy that generalizes.

The theoretical foundation builds on prior work in computer-use agents (CUAs), vision-language models (VLMs), and reinforcement learning with verifiable rewards (RLVR), positioning PLAYER-1 as advancing beyond systems like SIMA 2, GameWAM, and NitroGen by providing open availability and systematic evaluation both in-domain and on held-out games.

Methodology

Extended Action Space

The policy receives task instruction $q$, screenshot $I_t$, and interaction history $C_H(h_{<t})$:

$$(z_t, \hat{a}_t) \sim \pi_\theta(\cdot | q, I_t, C_H(h_{<t}))$$

Three key extensions to the computer-use action space:

1. Relative pointer motion:

$$\mathcal{A}_{\text{pointer}} = \underbrace{\{\text{absolute}(x, y)\}}_{\text{Retained CUA coordinates}} \cup \underbrace{\{\text{relative}(\Delta x, \Delta y)\}}_{\text{Relative-motion extension}}$$

Absolute coordinates satisfy $x, y \in [0, 1000]$, while relative displacements $\Delta x, \Delta y \in [-1000, 1000]$ are normalized by screen dimensions.

2. Sustained inputs:

$$\mathcal{A}_{\text{press}}(b) = \underbrace{\{\text{tap/click}(b)\}}_{\text{Retained CUA taps/clicks}} \cup \underbrace{\{\text{hold}(b; \Delta t)\}}_{\text{Sustained extension}}$$

3. Composite actions:

$$\mathcal{A}_{\text{step}} = \underbrace{\{\text{wait}\} \cup \mathcal{A}_{\text{keyboard}} \cup \mathcal{A}_{\text{mouse}}}_{\text{Individual CUA inputs}} \cup \underbrace{\{\text{input}_1 \| \cdots \| \text{input}_m\}}_{\text{Composite-action extension}}$$

Supervised Initialization

• 6,044 curated gameplay trajectories comprising 295,097 interaction turns from public datasets and in-house recordings
• Controls aligned to shared action intervals with source-specific mouse calibration
• A VLM annotates recorded actions with task instructions and per-turn visual reasoning
• SFT examples take the form $(q, I_t, C_H(h_{<t}), z_t, a_t)$ with a $2+2$ history setting (two recent visual-action turns, two earlier action-only turns)
• Action-balanced sampling increases exposure to less frequent controls and composite actions

PLAYER-1-GYM: Multi-Environment RL

Sixteen synthetic environments across five task families:

• Aiming (2): 2D aiming, 3D aiming
• Combat (5): 2D shooting, 3D combat arena, 3D spear duel, 3D sword duel, 3D melee arena
• Driving (4): 2D lane driving, 2D circuit driving, 3D road driving, 3D lane driving
• Navigation (4): 2D collect dodge, 2D platforming, 3D navigation, 3D chase
• UI interaction (1): 2D UI crafting

Four complementary randomization dimensions: appearance (colors, lighting, assets), camera (projection, placement, orientation), scene (layouts, spawns, configurations), and control (input-response gains).

Per-environment advantage normalization for PPO:

$$\widetilde{A}_{i,j} = \frac{A_{i,j} - \mu_e}{\sqrt{\sigma_e^2 + \epsilon}}, \quad \mu_e = \frac{\sum_{(i,j) \in \mathcal{S}_e} A_{i,j}}{N_e}, \quad \sigma_e^2 = \frac{\sum_{(i,j) \in \mathcal{S}_e} (A_{i,j} - \mu_e)^2}{N_e - 1}$$

where $i$ indexes turns, $j$ indexes response tokens, and $\mathcal{S}_e$ contains response-token indices in the rollout batch for environment $e$.

Empirical Validation / Results

MineDojo-300 Results

Table 2: Task success and coverage on MineDojo-300 (8 trials per task):

Key findings: SFT raises success from 9.21% to 10.54% (9B), RL further raises it to 38.33% — a 27.79 percentage point gain over SFT. Tasks completed in ≥4 of 8 trials increase from 20 to 136.

ViZDoom-10 Results

Table 3: First-person control on ViZDoom-10 (mean scaled return):

Player-1-35B-A3B surpasses GameWAM (strongest external baseline) by 83.7% on overall scaled return.

GameWorld Results

Table 4: Browser-game control on GameWorld (3 trials × 170 tasks × 34 games):

Learning Dynamics

• Both mean episode return and game score rise across RL training steps, with gains largest early in training
• SFT→RL trajectory rises from 17.67% to 51.67% Minecraft success, while Base→RL rises only from 12.67% to 16.67% (a 35.00-percentage-point gap at step 100)
• Different environments show varied learning curves: simple tasks (2D aiming) improve rapidly then level off, while 2D circuit driving shows delayed gains

Action Composition Evolution

Composite-action usage rises from 64.7% at RL start to 85.4% and 93.2% after 50 and 100 RL steps respectively — occurring without explicit reward for composition, suggesting task-driven RL reinforces coordinated inputs.

Theoretical and Practical Implications

For game-agent research: The work establishes synthetic experience as an effective resource for learning transferable game control without online training in each target game. The shared action space and environment-suite design provide a blueprint for scalable post-training of game agents.

For computer-use agents: Extending the CUA action space with relative pointer motion, sustained inputs, and composite actions demonstrates a path toward general digital control in dynamic, real-time environments beyond traditional GUI interaction.

For RL methodology: The per-environment advantage normalization (Eq. 1) addresses reward-scale heterogeneity across multi-environment training, while environment scaling analysis (2→16 environments) shows broader coverage improves transfer.

For practical deployment: The demonstration of control in Watch Dogs 2 — a commercial game unseen during both SFT and RL — shows the framework's potential for real-world game automation and testing applications.

Conclusion

PLAYER-1 demonstrates that learning from synthetic experience substantially improves game control beyond supervised initialization, with gains that generalize beyond the RL training suite. Key contributions include:

1. A post-training framework combining SFT on curated demonstrations with multi-environment RL
2. PLAYER-1-GYM: sixteen synthetic environments with verifiable rewards and extensive randomization
3. Empirical gains: 27.8 and 17.7 percentage-point improvements on MineDojo-300 for 9B and 35B-A3B models respectively, with generalization to ViZDoom and GameWorld
4. Scalability: broader synthetic environment coverage improves transfer
5. Emergent behavior: composite-action usage increases during RL without explicit reward

Future directions suggested by the work include expanding synthetic environment coverage, exploring additional control-demand families, and investigating further scaling of both models and environment suites. The findings support synthetic experience as an effective and scalable resource for developing transferable game-control capabilities.