Summary (Overview)

• Skill Self-Play (Skill-SP) is a co-evolutionary framework that resolves the fundamental tension between task diversity and verification reliability in LLM self-evolution by using a dynamically evolving library of modular skills.
• The framework consists of three components: a Proposer (generates challenging tasks conditioned on skills), a Solver (learns to solve tasks), and a Skill Controller (manages skill library via refinement, pruning, and induction).
• Empirically, Skill-SP achieves absolute gains of up to +42.9 points on tool-use (over unguided self-play) and +12.0 points on logical reasoning across diverse backbones (3B-14B parameters), consistently outperforming unguided self-play.
• Unguided self-play fails to produce valid tasks for complex reasoning domains and plateaus early, while Skill-SP maintains a steady frontier-tracking curriculum by co-evolving the skill library with the proposer and solver.
• The skill library actively expands (e.g., from 0 to 86 active skills over 5 iterations) and provides a balanced, diverse curriculum, unlocking a sustainable pathway for open-ended LLM self-evolution.

Introduction and Theoretical Foundation

The paper addresses a core dilemma in LLM self-evolution: environment-bound methods (e.g., code executors, game simulators) provide reliable verification but confine learning to narrow domains, while unguided generation with post-hoc filtering broadens the task space but suffers from residual errors, ill-posed tasks, and synthetic data collapse. The authors argue that agent skills—modular units of procedural knowledge that bundle task-specific instructions, examples, and validators—serve as a powerful middle ground. Skills provide structured, verifiable execution in specific scenarios while dynamic routing across skills maintains open-ended task variety. The concept formalizes a task-pattern interface that actively orchestrates the self-play loop: injecting structural priors before generation, enabling rigorous verification after response, and tracking difficulty for curriculum progression.

Methodology

Skill-SP formalizes a verifiable agent task as a tuple $(\boldsymbol{x}, \boldsymbol{c})$ where $\boldsymbol{x}$ is the solver prompt and $\boldsymbol{c}$ is a hidden machine-readable verification contract. The environment returns a reward $R_{\text{solve}}(\boldsymbol{x}, \boldsymbol{y}, \boldsymbol{c}) \in [0,1]$. The solver's expected success rate is:

$$v_{\text{solve}}(\boldsymbol{x}, \boldsymbol{c}; \pi_{\text{solve}}) = \mathbb{E}_{\boldsymbol{y} \sim \pi_{\text{solve}}(\cdot | \boldsymbol{x})} [R_{\text{solve}}(\boldsymbol{x}, \boldsymbol{y}, \boldsymbol{c})] \tag{1}$$

The proposer reward is gated by a validity filter:

$$R_{\text{propose}}(\boldsymbol{x}, \boldsymbol{c}; \pi_{\text{solve}}) = \mathbb{1}\{(\boldsymbol{x}, \boldsymbol{c}) \text{ is valid}\} \cdot \left(1 - 2\left|v_{\text{solve}}(\boldsymbol{x}, \boldsymbol{c}; \pi_{\text{solve}}) - \frac{1}{2}\right|\right) \tag{2}$$

The overall objective is a bi-level optimization:

$$\max_{\pi_{\text{propose}}, \mathcal{S}} \mathbb{E}_{\boldsymbol{s} \sim \mathcal{S}, (\boldsymbol{x}, \boldsymbol{c}) \sim \pi_{\text{propose}}(\cdot | \boldsymbol{s})} \left[ R_{\text{propose}}(\boldsymbol{x}, \boldsymbol{c}; \pi_{\text{solve}}^*) \right] \quad \text{s.t.} \quad \pi_{\text{solve}}^* = \arg\max_{\pi} \mathbb{E}[R_{\text{solve}}] \tag{3}$$

Key components:

• Skill-Conditioned Generation: Skills $\boldsymbol{s} = \langle m, r, h, e, \nu, \sigma \rangle$ include routing metadata, rules, hints, examples, validators, and tracking statistics. The proposer generates tasks via a skill stream $(\boldsymbol{x}, \boldsymbol{c}) \sim \pi_{\text{propose}}(\cdot | \boldsymbol{s})$ and an open-ended exploration stream $(\boldsymbol{x}, \boldsymbol{c}) \sim \pi_{\text{propose}}(\cdot | \emptyset)$.
• Validity Verification: Three conditions: schema compliance, contract validity (using skill-specific validators), and probe consistency ($K$ rollouts from the solver yield a unique majority answer matching the reference).
• Policy Updates: Both proposer and solver are updated via GRPO (Group Relative Policy Optimization). The solver curriculum is constructed by ranking valid tasks from both streams by $R_{\text{propose}}$ and selecting top-scoring tasks with a blending ratio $\alpha$.

Skill Library Evolution:

• Refinement: Update statistics $\sigma$ and refine skill content from failure traces.
• Pruning: Remove saturated skills $\mathcal{S}^{(t)}_{\text{prune}} = \{\boldsymbol{s} \in \mathcal{S}^{(t)} | \mathbb{E}[R_{\text{propose}}] < \gamma_{\text{prune}}\}$.
• Induction: Extract novel candidates $\mathcal{T}^{(t)}_{\text{induce}} = \{(\boldsymbol{x}, \boldsymbol{c}) \in \mathcal{T}^{(t)}_{\text{explore}} | R_{\text{propose}} \geq \gamma_{\text{induce}}\}$, then abstract into new skills via $\pi_{\text{control}}$ with integrity and novelty checks.

Empirical Validation / Results

The framework is evaluated on tool-call prediction (API-Bank Levels 1–3, BFCL across JS/Python/Java/Live) and logical reasoning (ZebraLogic grid puzzles) using five backbones: Qwen3-4B-Ins, Qwen3-8B, Ministral-3-8B, Ministral-3-14B, and Granite-4.1-3B.

Table 1: Main tool-call prediction results (avg@8)

Skill-SP consistently improves all backbones, with massive gains (up to +42.9) for initially misaligned models like Ministral-3-8B. Unguided SP often stagnates or degrades.

Table 2: Main logical reasoning results (avg@8) on ZebraLogic

Unguided SP cannot bootstrap reasoning tasks. Skill-SP delivers up to +12.0 overall and +35.3 on small puzzles for weak models.

Ablations (Table 3, 4) confirm that:

• Removing skill orchestration (Unguided SP) causes a 2.6-point drop.
• Exclusive skill-only generation (no exploration stream) underperforms the mixed pool.
• Static routing (Uniform, Frozen skills) reduces accuracy by 1.9–2.3 points.
• Freezing the proposer or feedback solver (disabling co-evolution) degrades performance by 2.1–3.2 points.

Data-loop diagnostics (Figure 5, 6) show that the skill-stream maintains a mean solver success rate of ~0.57 (near the frontier), while exploration and Unguided SP drift to 0.70–0.75. The skill library grows to 86 active skills over 5 iterations, with continuous induction, refinement, and pruning, ensuring broad and balanced curriculum coverage.

Theoretical and Practical Implications

• Theoretical: Skill-SP provides a principled resolution to the diversity–verification tension by using modular skills as proactive task-pattern interfaces. It formalizes self-play as a co-evolutionary process where the skill library, proposer, and solver jointly advance, avoiding context bloat and passive filtering limitations.
• Practical: The framework is broadly applicable to any verifiable agent task (tool-use, reasoning, etc.) and can rescue models that initially fail to generate valid tasks, dramatically expanding their capabilities. The open-source code enables reproducibility and further research.

Conclusion

Skill Self-Play demonstrates that co-evolving skills serve as powerful training-time scaffolds, enabling sustainable, structured, and truly open-ended LLM self-evolution. The framework outperforms unguided self-play across diverse backbones and tasks, with consistent improvements and striking turnarounds for misaligned models. Future work may explore extending the skill library to multimodal domains, incorporating stronger teacher models for skill induction, and scaling to larger backbones.