# Progressive Agent Skill Generation via Reinforcement Learning

> Skill-α uses a rollback reward to train an RL policy that generates agent skills via progressive editing, outperforming all baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2608.01678)
- **Published:** 2026-08-05
- **Permalink:** https://picx.dev/p/s4hUjj
- **Whiteboard:** https://picx.dev/p/s4hUjj/image

## Summary

## Summary (Overview)

- **Novel Framework**: Proposes Skill-α, a reinforcement learning (RL) framework that formulates agent skill generation as a progressive, sequential editing process, unifying document-to-skill and experience-to-skill generation under a single learning paradigm.
- **Key Innovation - Rollback Reward**: Introduces a *rollback reward* mechanism that provides execution-grounded credit assignment for local skill edits. It evaluates each edit by comparing the downstream task performance of a fixed worker agent using the original skill versus the edited skill on an anchored query.
- **Strong Empirical Performance**: Under a GPT-4o worker, Skill-α improves average downstream success rates over the strongest baselines by 3.3 points on CL-Bench and 6.7 points on tau2-bench. The learned skills also transfer effectively to a different worker (Claude-Sonnet-4.5).
- **Unified Approach**: Unlike existing heuristic or pipeline-based methods that are tailored to specific evidence types, Skill-α uses a single learned policy for both document and experience sources.
- **Comprehensive Analysis**: Ablation studies validate the importance of the rollback reward and the progressive generation strategy. Analysis shows the method is robust to evidence reordering but sensitive to evidence granularity (batch size).

## Introduction and Theoretical Foundation

The paper addresses the challenge of automatically generating high-quality "skills" for Large Language Model (LLM) agents. Skills are reusable procedural modules (e.g., a `SKILL.md` file) that condition an agent's inference, helping it decompose tasks, use tools, and check results without retraining the underlying model. The core problem is that skill generation lacks a natural, direct supervision signal. Unlike tasks with clear right/wrong answers (e.g., math, web search), a skill's quality is primarily determined by its downstream effect on agent behavior.

The authors argue that existing methods, which rely on heuristics or pipeline-style consolidation (e.g., Ctx2Skill, Trace2Skill), are source-specific and do not learn a unified generation process. The central challenge is to provide "execution-grounded credit assignment" – linking a change in the skill text to a change in agent performance.

The paper formalizes skill generation with the following objective, which aims to minimize the KL divergence between a high-quality teacher behavior distribution $p^*_M$ and the worker agent's behavior $\pi_\psi$ conditioned on a generated skill $z$:

$$
\min_{\phi} \mathbb{E}_{M, q', x_{1:T} \sim p^*_M(\cdot|q'), q, z \sim \bar{\pi}_\phi(\cdot|x_{1:T}, z_0)} [ D_{KL}(p^*_M(\tau | q) \parallel \pi_\psi(\tau | z, q)) ]
$$

Where $\pi_\phi$ is the skill-editing policy, $x_{1:T}$ are evidence units, $z_0$ is the initial skill, and $q$ is a held-out target query. Since $p^*_M$ is unavailable, this objective must be approximated.

## Methodology

Skill-α decomposes the monolithic task of skill generation into a sequential decision-making process. The methodology has three key components:

1.  **Progressive Skill Generation**: Instead of generating a skill in one shot, the generator ($\pi_\phi$) reads evidence sequentially and makes a series of local edits. Starting from an initial skill state $z_0$, at each step $t$, it samples an action $A_t$ based on the current skill and new evidence $x_t$:
    $$A_t \sim \pi_\phi(\cdot | z_{t-1}, x_t), \quad z_t = \text{Edit}(z_{t-1}, A_t)$$
    This decomposes the complex generation task into simpler, evaluable local decisions.

2.  **Action Space and Rollback Reward**: The action space $\mathcal{A}$ consists of five structured edits:
    - **CREATE**: Adds missing rules or procedures.
    - **UPDATE**: Fixes incomplete or inaccurate rules.
    - **MERGE**: Consolidates overlapping content.
    - **PRUNE**: Removes misleading or redundant content.
    - **NOOP**: Leaves the skill unchanged.

    The core training signal is the **rollback reward**. For a candidate edit $A_t^{(i)}$, the reward $R_{rb}$ is defined by comparing the performance of the fixed worker agent ($\pi_\psi$) on the same anchored query ($q^{anc}_t$) using the old skill ($z_{t-1}$) and the new skill ($z_t^{(i)}$). A verifier ($V_t$) provides scalar feedback ($r^{ctrl}_t, r^{edit}_{t,i}$).

    $$R_{rb}(A_t^{(i)}) = \begin{cases} 1, & \text{if } A_t^{(i)} \neq \text{NOOP and } r^{edit}_{t,i} > r^{ctrl}_t \\ 1, & \text{if } A_t^{(i)} = \text{NOOP and no other edit improves the score} \\ 0, & \text{otherwise} \end{cases}$$

    This reward provides a local, execution-grounded signal for credit assignment.

3.  **Training Implementation**: The skill-editing policy $\pi_\phi$ is initialized from an instruction-tuned Qwen3-8B model. Training proceeds in two stages:
    - **Supervised Fine-Tuning (SFT)**: Warm-up on a dataset of 6,481 synthetic skill-editing examples generated by DeepSeek-V4-Pro.
    - **Reinforcement Learning (RL)**: Optimization using Group Relative Policy Optimization (GRPO) with the proposed rollback reward. The fixed worker agent during RL is GPT-4o. The GRPO objective is:
        $$J_{\text{GRPO}}(\phi) = \mathbb{E} \left[ \frac{1}{G} \sum_{i=1}^G \left( \min(\rho_i(\phi) \hat{A}_i, \text{clip}(\rho_i(\phi), 1-\epsilon, 1+\epsilon) \hat{A}_i) - \beta D_{KL}(\pi_\phi \parallel \pi_{\text{ref}}) \right) \right]$$
        where $\hat{A}_i$ is the group-relative advantage calculated from the rollback rewards.

## Empirical Validation / Results

The paper evaluates Skill-α on three benchmarks, comparing it against a range of strong baselines.

- **Document-to-Skill (CL-Bench)**: Skill-α achieves the highest average pass rate of 10.38% with a GPT-4o backbone, significantly outperforming the best baseline (AutoSkill at 6.37%). The largest gains are on "Procedural Task Execution" (9.68% vs. 5.38% for Anthropic Skill-Creator). The skills also transfer well to a Claude-Sonnet-4.5 backbone.

- **Experience-to-Skill (SpreadsheetBench & tau2-bench)**: Skill-α shows even clearer advantages. With GPT-4o, it achieves the best or tied-best score on every metric, with a tau2-bench average of 55.83%, compared to the best baseline (SkillPro at 49.17%). On SpreadsheetBench, it improves from 18.00% (No Skill) to 27.50%.

**Table 3: Experience-to-skill results on SpreadsheetBench and tau2-bench.**

| Method | SpreadsheetBench ↑ | tau2-bench (Airline) ↑ | tau2-bench (Retail) ↑ | tau2-bench (Telecom) ↑ | tau2-bench Avg. ↑ |
| :--- | :--- | :--- | :--- | :--- | :--- |
| **Backbone: GPT-4o** | | | | | |
| No Skill | 18.00 | 40.00 | 47.50 | 12.50 | 33.33 |
| Anthropic Skill-Creator | 26.00 | 55.00 | 60.00 | 7.50 | 40.83 |
| SkillPro | 15.50 | 55.00 | 72.50 | 20.00 | 49.17 |
| **Skill-α** | **27.50** | **65.00** | **80.00** | **22.50** | **55.83** |
| **Backbone: Claude-Sonnet-4.5** | | | | | |
| No Skill | 26.00 | 65.00 | 85.00 | 40.00 | 63.33 |
| SkillX | 31.50 | 75.00 | 85.00 | 35.00 | 65.00 |
| **Skill-α** | **31.50** | **80.00** | **86.00** | **45.00** | **70.33** |

- **Ablation Study**: The full model outperforms all ablated versions, confirming the necessity of each component.

**Table 4: Ablation study results.**

| Variant | CL-Bench Avg. ↑ | SpreadsheetBench ↑ | tau2-bench Avg. ↑ |
| :--- | :--- | :--- | :--- |
| **Skill-α (Full)** | **10.38** | **27.50** | **55.83** |
| SFT only | 3.46 | 15.50 | 44.17 |
| w/o rollback reward | 3.68 | 17.00 | 46.67 |
| w/o MERGE/PRUNE | 4.74 | 20.00 | 39.17 |
| w/o NOOP | 9.55 | 22.00 | 53.33 |

- **Analysis**: The method is robust to evidence order but highly sensitive to evidence batch size, with a batch of 4 units per step performing best. The training dynamics show the full model learns a balanced policy, while ablations lead to conservative (dominated by NOOP) or unstable behavior.

## Theoretical and Practical Implications

- **Theoretical Implications**: The paper provides a theoretical analysis of the rollback reward, showing that under a calibrated verifier, the expected rollback reward preserves the ideal preference ranking of edits. For a binary success/failure task, the expected reward is proportional to the improvement in success probability. This provides a sound theoretical basis for using this reward in a group-relative optimization framework like GRPO.

- **Practical Implications**: Skill-α presents a practical, unified, and trainable framework for automatic skill generation. Its key practical advantages are:
    - **Unified Solution**: It replaces a collection of source-specific heuristics with a single learning-based system.
    - **Execution-Grounded Quality**: It optimizes skills based on their actual impact on task performance, not just textual fluency.
    - **Transferability**: The skills generated by Skill-α are not overfitted to a single worker model and can improve performance when used with different LLMs.
    - **Robustness**: The progressive generation strategy is robust to the order in which evidence is presented.

## Conclusion

Skill-α is a reinforcement learning framework that reframes agent skill generation as a progressive editing process. By introducing the rollback reward, it provides a powerful, execution-grounded training signal for local skill edits. Experiments across multiple benchmarks demonstrate that Skill-α consistently outperforms heuristic and pipeline-based methods, generating more effective skills that transfer across different worker agents. The work successfully addresses the core challenge of credit assignment in skill learning.

**Future Work** identified by the authors includes extending the method to more general and stronger verifiers, richer multimodal skill formats, and longer-horizon training signals that go beyond local rollback comparisons. The current dependency on benchmark-specific verifiers is noted as a limitation.

---

_Markdown view of https://picx.dev/p/s4hUjj, served by PicX — AI-generated visual whiteboard summaries of research papers._