Summary (Overview)

• Novel Framework: Proposes Skill-α, a reinforcement learning (RL) framework that formulates agent skill generation as a progressive, sequential editing process, unifying document-to-skill and experience-to-skill generation under a single learning paradigm.
• Key Innovation - Rollback Reward: Introduces a rollback reward mechanism that provides execution-grounded credit assignment for local skill edits. It evaluates each edit by comparing the downstream task performance of a fixed worker agent using the original skill versus the edited skill on an anchored query.
• Strong Empirical Performance: Under a GPT-4o worker, Skill-α improves average downstream success rates over the strongest baselines by 3.3 points on CL-Bench and 6.7 points on tau2-bench. The learned skills also transfer effectively to a different worker (Claude-Sonnet-4.5).
• Unified Approach: Unlike existing heuristic or pipeline-based methods that are tailored to specific evidence types, Skill-α uses a single learned policy for both document and experience sources.
• Comprehensive Analysis: Ablation studies validate the importance of the rollback reward and the progressive generation strategy. Analysis shows the method is robust to evidence reordering but sensitive to evidence granularity (batch size).

Introduction and Theoretical Foundation

The paper addresses the challenge of automatically generating high-quality "skills" for Large Language Model (LLM) agents. Skills are reusable procedural modules (e.g., a SKILL.md file) that condition an agent's inference, helping it decompose tasks, use tools, and check results without retraining the underlying model. The core problem is that skill generation lacks a natural, direct supervision signal. Unlike tasks with clear right/wrong answers (e.g., math, web search), a skill's quality is primarily determined by its downstream effect on agent behavior.

The authors argue that existing methods, which rely on heuristics or pipeline-style consolidation (e.g., Ctx2Skill, Trace2Skill), are source-specific and do not learn a unified generation process. The central challenge is to provide "execution-grounded credit assignment" – linking a change in the skill text to a change in agent performance.

The paper formalizes skill generation with the following objective, which aims to minimize the KL divergence between a high-quality teacher behavior distribution $p^*_M$ and the worker agent's behavior $\pi_\psi$ conditioned on a generated skill $z$:

$$\min_{\phi} \mathbb{E}_{M, q', x_{1:T} \sim p^*_M(\cdot|q'), q, z \sim \bar{\pi}_\phi(\cdot|x_{1:T}, z_0)} [ D_{KL}(p^*_M(\tau | q) \parallel \pi_\psi(\tau | z, q)) ]$$

Where $\pi_\phi$ is the skill-editing policy, $x_{1:T}$ are evidence units, $z_0$ is the initial skill, and $q$ is a held-out target query. Since $p^*_M$ is unavailable, this objective must be approximated.

Methodology

Skill-α decomposes the monolithic task of skill generation into a sequential decision-making process. The methodology has three key components:

1. Progressive Skill Generation: Instead of generating a skill in one shot, the generator ($\pi_\phi$) reads evidence sequentially and makes a series of local edits. Starting from an initial skill state $z_0$, at each step $t$, it samples an action $A_t$ based on the current skill and new evidence $x_t$:

   $$A_t \sim \pi_\phi(\cdot | z_{t-1}, x_t), \quad z_t = \text{Edit}(z_{t-1}, A_t)$$

   This decomposes the complex generation task into simpler, evaluable local decisions.

2. Action Space and Rollback Reward: The action space $\mathcal{A}$ consists of five structured edits:

   • CREATE: Adds missing rules or procedures.
   • UPDATE: Fixes incomplete or inaccurate rules.
   • MERGE: Consolidates overlapping content.
   • PRUNE: Removes misleading or redundant content.
   • NOOP: Leaves the skill unchanged.

   The core training signal is the rollback reward. For a candidate edit $A_t^{(i)}$, the reward $R_{rb}$ is defined by comparing the performance of the fixed worker agent ($\pi_\psi$) on the same anchored query ($q^{anc}_t$) using the old skill ($z_{t-1}$) and the new skill ($z_t^{(i)}$). A verifier ($V_t$) provides scalar feedback ($r^{ctrl}_t, r^{edit}_{t,i}$).

   $$R_{rb}(A_t^{(i)}) = \begin{cases} 1, & \text{if } A_t^{(i)} \neq \text{NOOP and } r^{edit}_{t,i} > r^{ctrl}_t \\ 1, & \text{if } A_t^{(i)} = \text{NOOP and no other edit improves the score} \\ 0, & \text{otherwise} \end{cases}$$

   This reward provides a local, execution-grounded signal for credit assignment.

3. Training Implementation: The skill-editing policy $\pi_\phi$ is initialized from an instruction-tuned Qwen3-8B model. Training proceeds in two stages:

   • Supervised Fine-Tuning (SFT): Warm-up on a dataset of 6,481 synthetic skill-editing examples generated by DeepSeek-V4-Pro.
   • Reinforcement Learning (RL): Optimization using Group Relative Policy Optimization (GRPO) with the proposed rollback reward. The fixed worker agent during RL is GPT-4o. The GRPO objective is: $$J_{\text{GRPO}}(\phi) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \left( \min(\rho_i(\phi) \hat{A}_i, \text{clip}(\rho_i(\phi), 1-\epsilon, 1+\epsilon) \hat{A}_i) - \beta D_{KL}(\pi_\phi \parallel \pi_{\text{ref}}) \right) \right]$$ where $\hat{A}_i$ is the group-relative advantage calculated from the rollback rewards.

Empirical Validation / Results

The paper evaluates Skill-α on three benchmarks, comparing it against a range of strong baselines.

• Document-to-Skill (CL-Bench): Skill-α achieves the highest average pass rate of 10.38% with a GPT-4o backbone, significantly outperforming the best baseline (AutoSkill at 6.37%). The largest gains are on "Procedural Task Execution" (9.68% vs. 5.38% for Anthropic Skill-Creator). The skills also transfer well to a Claude-Sonnet-4.5 backbone.

• Experience-to-Skill (SpreadsheetBench & tau2-bench): Skill-α shows even clearer advantages. With GPT-4o, it achieves the best or tied-best score on every metric, with a tau2-bench average of 55.83%, compared to the best baseline (SkillPro at 49.17%). On SpreadsheetBench, it improves from 18.00% (No Skill) to 27.50%.

Table 3: Experience-to-skill results on SpreadsheetBench and tau2-bench.

• Ablation Study: The full model outperforms all ablated versions, confirming the necessity of each component.

Table 4: Ablation study results.

• Analysis: The method is robust to evidence order but highly sensitive to evidence batch size, with a batch of 4 units per step performing best. The training dynamics show the full model learns a balanced policy, while ablations lead to conservative (dominated by NOOP) or unstable behavior.

Theoretical and Practical Implications

• Theoretical Implications: The paper provides a theoretical analysis of the rollback reward, showing that under a calibrated verifier, the expected rollback reward preserves the ideal preference ranking of edits. For a binary success/failure task, the expected reward is proportional to the improvement in success probability. This provides a sound theoretical basis for using this reward in a group-relative optimization framework like GRPO.

• Practical Implications: Skill-α presents a practical, unified, and trainable framework for automatic skill generation. Its key practical advantages are:

  • Unified Solution: It replaces a collection of source-specific heuristics with a single learning-based system.
  • Execution-Grounded Quality: It optimizes skills based on their actual impact on task performance, not just textual fluency.
  • Transferability: The skills generated by Skill-α are not overfitted to a single worker model and can improve performance when used with different LLMs.
  • Robustness: The progressive generation strategy is robust to the order in which evidence is presented.

Conclusion

Skill-α is a reinforcement learning framework that reframes agent skill generation as a progressive editing process. By introducing the rollback reward, it provides a powerful, execution-grounded training signal for local skill edits. Experiments across multiple benchmarks demonstrate that Skill-α consistently outperforms heuristic and pipeline-based methods, generating more effective skills that transfer across different worker agents. The work successfully addresses the core challenge of credit assignment in skill learning.

Future Work identified by the authors includes extending the method to more general and stronger verifiers, richer multimodal skill formats, and longer-horizon training signals that go beyond local rollback comparisons. The current dependency on benchmark-specific verifiers is noted as a limitation.