SPADE ♠: Self-Play in Adaptive Synthetic Executable Environments

Summary (Overview)

• SPADE (Self-Play in Adaptive Synthetic Executable Environments) is a novel self-play reinforcement learning (RL) framework where a single LLM plays two roles: an Environment Designer that writes executable training environments as Python code (Gym-style reset()/step() interface) and a Reasoning Agent that learns to act within them.
• The framework introduces a hint-based regret reward for the Environment Designer, which targets environments at the frontier of the Reasoning Agent's capability—solvable with hints but challenging without them—grounded in minimax regret theory (PAIRED).
• SPADE is corpus-grounded and memory-augmented, using documents from pretraining corpora to sustain environment diversity and an environment memory to avoid re-posing mastered tasks.
• At 30B-parameter scale, SPADE achieves +5.3 average improvement over the strongest fixed-environment baseline across eight held-out benchmarks (math, science, code, reasoning), and +13.9 on ACEBench-Agent and +5.7 on BFCL v4 multi-turn in tool-use settings.
• The framework demonstrates that making environment design itself a learnable component enables open-ended self-improvement, with gains that grow with model scale.

Introduction and Theoretical Foundation

Background and Motivation

Agentic AI systems have become broadly capable—reasoning over long horizons, using tools, searching the web, and operating computers. However, these gains increasingly come from learning through experience rather than pretraining alone. The key bottleneck is the supply of training environments: interactive tasks with verifiable rewards. High-quality human text is a finite resource, and existing approaches to environment creation have fundamental limitations:

• Harness engineering improves inference-time behavior but changes no weights
• Human-curated scaling scales only as fast as people can write environments
• Synthetic generation uses fixed generators that don't adapt as the agent improves
• Un-grounded self-play is bounded by information symmetry and prone to amplifying errors

Theoretical Foundation

SPADE builds on Unsupervised Environment Design (UED) principles, particularly the minimax regret objective from PAIRED (Dennis et al., 2020). The framework draws from:

1. Markov Decision Processes (MDPs): An MDP is a tuple $(S, A, T, R, \rho_0)$ with state space $S$, action space $A$, transition function $T(s'|s,a)$, reward function $R(s,a)$, and initial state distribution $\rho_0$.

2. Group Relative Policy Optimization (GRPO): The policy $\pi_\theta$ is trained via RLVR with group-normalized advantages:

$$\hat{A}^i = \frac{r^i - \text{mean}(\{r^j\}_{j=1}^{G})}{\text{std}(\{r^j\}_{j=1}^{G})} \tag{1}$$

3. Self-play paradigms: From TD-Gammon through AlphaZero to modern LLM self-play methods (SPIRAL, SPICE, AZR), the dual-role pattern enables capability growth.

Methodology

Dual-Role Self-Play and Code-as-Environment

SPADE trains a single LLM $\pi_\theta$ in two roles via role-specific system prompts:

• Environment Designer (role=D): Produces executable environments $e \in \mathcal{E}$ as Python programs implementing the Gym-style reset()/step() API, plus a privileged hint $h$ for each environment
• Reasoning Agent (role=A): Interacts with the environment via sequential actions, receiving observations and rewards

The code-as-environment representation unifies single-turn settings and multi-turn agentic tasks under a single interface. Any computable MDP can be expressed as a Python program, enabling an unbounded environment space.

Hint-Based Regret Reward

The Environment Designer's reward for producing environment $e$ is:

$$r_D(e) = \bar{r}_A(e \mid h) - \bar{r}_A(e) \tag{3}$$

where $\bar{r}_A(e \mid h)$ is the Reasoning Agent's average return over $G$ rollouts with the hint, and $\bar{r}_A(e)$ is the average without it. Three regimes emerge:

• High regret: Environment at the learning frontier (solvable with hints, not without)
• Low regret with high returns: Mastery achieved
• Low regret with low returns: Intractable environment

Environment Design Pipeline

Corpus grounding: The Environment Designer conditions on freshly sampled documents from pretraining corpora (DCLM, MegaScience for games; Nemotron code corpus for tool-use), preventing mode collapse and sustaining diversity.

Environment memory: A buffer of previously generated environments annotated with regret scores and skill tags, giving the Environment Designer high-regret seeds to vary and too-easy/too-hard examples to avoid.

Stabilizing joint training: Per-role advantage normalization, upweighting of less frequent Environment Designer trajectories, delayed Environment Designer updates with truncated importance sampling, and asymmetric clipping ranges $(\varepsilon_{low} = 0.2, \varepsilon_{high} = 0.28)$.

Empirical Validation / Results

Games Setting Results

Training on diverse synthetic games improves held-out benchmarks at every backbone scale:

Tool-Use Setting Results

Key Qualitative Findings

1. Emergent curricula: SPADE progresses from simple single-skill tasks to complex multi-constraint environments requiring long-horizon interaction (Figure 2)
2. Corpus grounding sustains diversity: Vendi/n score of 0.68 with corpus vs. 0.04 without (Figure 7)
3. Environment Designer training makes environments harder: Formula-reveal rate falls from 25% to 5% over training; rewards become more finely graded (3.7 → 5.8 distinct levels)
4. Reasoning Agent shifts from front-loaded derivation to evidence-first interaction (Figure 9)

Ablation Results

Hint-based regret vs. EMA-based learning potential: Hint-based regret achieves +8.1 vs. base (58.3), while EMA-based learning potential reaches +5.7 (55.9), with the signals separating after ~50 steps.

Theoretical and Practical Implications

Theoretical Contributions

1. Hint-based regret as a practical UED objective: SPADE implements the minimax regret principle of PAIRED without a separate antagonist—the hint-equipped Reasoning Agent serves as the upper-bound policy. Under idealized assumptions, every pure Nash equilibrium yields hint-free optimality on every valid environment.

2. Code-as-environment representation: By representing environments as executable Python programs, SPADE extends UED from parameterized design spaces to an unbounded, code-defined environment space.

3. Co-evolution dynamics: The Environment Designer and Reasoning Agent co-evolve, with the environment distribution shifting as the Reasoning Agent improves—a step toward open-ended self-improvement.

Practical Implications

1. Scaling behavior: SPADE's gains grow with model size (+5.2 at 4B, +5.7 at 8B, +8.1 at 30B-A3B), while fixed-environment baselines plateau near +1.2 at every size.

2. Curriculum diversity matters: A six-skill curriculum captures significantly more gains than a two-skill version (53.7 vs. 58.3 on the suite average).

3. Cross-domain transfer: Skills learned in synthetic games (planning, constraint satisfaction, strategic thinking) generalize to held-out mathematics, science, and code benchmarks.

4. Unified interface: One code-as-environment interface spans both single-turn reasoning and multi-turn agentic tool use, simplifying the training pipeline.

Conclusion

SPADE demonstrates that a single LLM can design its own training environments and improve from them, achieving:

• +8.1 average improvement over base at 30B-A3B across eight held-out benchmarks
• +5.3 over the strongest fixed-environment baseline at 30B-A3B
• +13.9 on ACEBench-Agent and +5.7 on BFCL v4 multi-turn in tool-use settings
• Gains that retain after fixed-environment baselines begin to saturate

Key Limitations

1. Complexity bounded by scale: The Environment Designer cannot produce environments more complex than its base model can express
2. Human-designed optimizer: Both roles use a fixed, human-authored RL algorithm (GRPO)
3. No formal optimality proof: Hint-based regret is motivated by PAIRED but not proven to yield an optimal curriculum

Future Directions

• In-context Environment Designer evolution without weight updates
• Automating the remaining stages of post-training (data curation through learning rules)
• Extending co-adaptive self-play across the whole training pipeline

SPADE takes a concrete step from fixed benchmarks toward open-ended, continual self-improvement—a single model that improves not only how it reasons and acts, but the worlds it builds to learn in.