Summary (Overview)

• Skill1 is a unified framework that trains a single policy to co-evolve three coupled capabilities of skill-augmented LLM agents: skill selection, skill utilization, and skill distillation, all toward a shared task-outcome objective.
• The key innovation is decomposing the single task-outcome reward signal $r(\tau)$ into a low-frequency trend (crediting selection) and high-frequency variation (crediting distillation), eliminating the need for separate auxiliary reward sources.
• Skill1 achieves 97.5% average success rate on ALFWorld (surpassing the previous best RetroAgent by 2.6 points) and 82.9% success rate on WebShop, outperforming all prior skill-based and RL baselines.
• Training dynamics confirm the mutual reinforcement of the three capabilities: selection precision converges first, which accelerates utilization and distillation.
• Ablation studies show that removing any single credit-assignment signal degrades all three capabilities, evidencing their coupled evolution.

---

Introduction and Theoretical Foundation

Background and Motivation

Reinforcement learning (RL) has become a key paradigm for training LLM agents in complex environments. However, standard RL treats each task as an isolated episode—successful strategies are absorbed only implicitly into policy parameters and cannot be explicitly reused. The paper addresses this by augmenting agents with a persistent skill library that accumulates reusable strategies.

The workflow of skill-augmented agents follows a three-stage lifecycle:

1. Skill selection: selecting a relevant skill from the library
2. Skill utilization: executing guided by the selected skill
3. Skill distillation: deriving new reusable skills from trajectories

Open Problems

Two fundamental questions motivate the work:

1. How can an agent evolve all three capabilities simultaneously? Existing methods optimize only a subset, leaving at least one capability unoptimized.
2. How can the three capabilities co-evolve toward a shared objective? Prior designs draw rewards from different sources, creating conflicting pressures.

Task Formulation

The problem is formulated as a POMDP $\mathcal{M} = (S, \mathcal{A}, \mathcal{O}, T, \Omega, R, \gamma)$ where:

• State $S = (x, e, B)$ comprises task instruction $x$, environment state $e$, and skill library $B = \{s_1, s_2, \ldots\}$
• Observation $o_t = (x, e_t, z)$ where $z$ is the selected skill via a frozen encoder $\mathcal{E}$

The training objective is:

$$\max_{\theta} \mathbb{E}_{x \sim \mathcal{D}, \tau \sim \pi_{\theta}(\cdot | x)} [r(\tau)]$$

A skill $s \in B$ consists of a strategy description $s.\text{strat}$ and a scenario description $s.\text{desc}$. Actions are generated conditioned on the skill:

$$a_t \sim \pi_{\theta}(\cdot | x, z.\text{strat}, o_{\leq t})$$

---

Methodology

Agent Workflow

For each task $x \sim \mathcal{D}$, the policy performs three stages producing trajectory $\tau = (q, z, a_1, o_1, \ldots, a_T, o_T, s_{\text{new}})$:

1. Skill Selection:

• Generate query $q \sim \pi_{\theta}(\cdot | x)$
• Retrieve top-K candidates via semantic similarity:

$$\mathcal{B}_K = \text{top-K}_{s \in \mathcal{B}} \text{sim}\big(\mathcal{E}(q), \mathcal{E}(s.\text{desc})\big)$$

• Re-rank candidates by generating permutation $\sigma \sim \pi_{\theta}(\cdot | x, \mathcal{B}_K)$

2. Skill Utilization: Multi-turn interaction conditioned on selected skill: $\tau \sim \pi_{\theta}(\cdot | x, z.\text{strat}, o_{\leq t})$

3. Skill Distillation: Policy reflects on trajectory to produce new strategy $s_{\text{new}}.\text{strat}$ and description $s_{\text{new}}.\text{desc}$. New skills admitted only when $r(\tau) = 1$.

Reward Assignment via Signal Decomposition

The core innovation is decomposing the task outcome $r(\tau)$ into:

Crediting utilization (direct outcome):

$$R_i^{\text{util}} = r(\tau_i)$$

Crediting selection via low-frequency trend—per-skill utility updated by exponential moving average:

$$U(s) \leftarrow (1 - \alpha) \cdot U(s) + \alpha \cdot r(\tau_i), \quad \forall s \in \mathcal{B}_K$$

The best available utility $\hat{U}_i = \max_{s \in \mathcal{B}_K^i} U(s)$ serves as library baseline. Re-ranking is supervised via NDCG:

$$R_i^{\text{rerank}} = \text{NDCG}\big(\sigma_i, \text{argsort}(-U(\mathcal{B}_K^i))\big)$$

Crediting distillation via high-frequency variation:

$$R_i^{\text{distill}} = r(\tau_i) - \hat{U}_i$$

Joint Optimization

Three objectives are combined in a single update:

$$\mathcal{J}(\theta) = \mathcal{J}^{\text{util}}(\theta) + \lambda_1 \mathcal{J}^{\text{rerank}}(\theta) + \lambda_2 \mathcal{J}^{\text{distill}}(\theta)$$

• Utilization uses GRPO with group-relative advantages
• Re-ranking uses REINFORCE-style objective (since candidate sets differ across rollouts)
• Distillation uses GRPO with separately normalized advantages

The GRPO advantage is computed as:

$$\hat{A}_i = \frac{r(\tau_i) - \text{mean}(\{r(\tau_1), \ldots, r(\tau_G)\})}{\text{std}(\{r(\tau_1), \ldots, r(\tau_G)\})}$$

---

Empirical Validation / Results

Main Results

Table 1: Main results on ALFWorld and WebShop (Success Rate %)

Key findings:

• Skill1 achieves 97.5% on ALFWorld, surpassing RetroAgent by 2.6 points
• Outperforms GiGPO (best RL-only) by 6.5 points, with largest gains on Look and Pick2
• Performance increases with the degree of co-evolution across lifecycle stages

Ablation Study

Table 2: Ablation study on ALFWorld (Success Rate %)

Key insights:

• Removing the library causes the largest drop (97.5% → 80.9%), especially on Heat and Pick2
• Removing selection or distillation individually reduces performance by ~5 points
• Removing both auxiliary objectives yields a sharper decline (90.2%), worse than removing each stage individually

Co-evolution Dynamics

• Selection precision converges first (reaching 0.95 by step 20), accelerating the other two stages
• Utilization and distillation both reach 0.8 by step 60
• Ablating any credit-assignment signal slows all three capabilities, confirming mutual dependence

Computational Overhead

Table 3: Computational cost on ALFWorld training

• Skill1 adds ~1.3–1.7× overhead over GRPO, stemming from growing library context
• Distillation compresses experience, controlling both quality and computational cost

---

Theoretical and Practical Implications

Theoretical Significance

1. Unified credit assignment: The paper demonstrates that a single task-outcome signal can be decomposed into trend (low-frequency) and variation (high-frequency) components to provide stage-specific gradients, eliminating the need for heterogeneous reward sources that create conflicting optimization pressures.

2. Co-evolution principle: The results establish that the three skill lifecycle capabilities are mutually dependent—optimizing them jointly produces emergent benefits beyond optimizing each in isolation.

3. Skill library as explicit memory: The work shows that explicit skill libraries complement parameter-only RL, with the largest gains on tasks requiring composition of multiple sub-procedures.

Practical Implications

1. Sample efficiency: The framework enables agents to reuse successful strategies across tasks rather than solving each from scratch, improving sample efficiency in interactive environments.

2. Scalable skill management: The trend-based utility scoring and variation-based distillation naturally regulate library growth, compressing experience into concise, reusable skills.

3. Broad applicability: The framework is demonstrated on ALFWorld (embodied household tasks) and WebShop (e-commerce), suggesting applicability to diverse agent environments.

---

Conclusion

Main Takeaways

• Skill1 trains a single policy to co-evolve skill selection, utilization, and distillation toward a shared task-outcome objective
• The low-frequency trend of outcomes credits selection; the high-frequency variation credits distillation
• Achieves state-of-the-art performance on ALFWorld (97.5%) and WebShop (82.9% success)
• Ablations confirm the coupled evolution of all three capabilities

Limitations

1. Environment coverage: Evaluation limited to two text-based environments; generalization to visual or deep-search environments unexplored
2. Scalability of skill library: Fixed capacity of 5,000 entries may become a bottleneck as task diversity grows; more sophisticated eviction or hierarchical organization strategies may be needed

Future Directions

• Extending the co-evolution framework to broader agent settings (visual observations, deep search)
• Developing more sophisticated library management strategies for larger-scale task distributions
• Exploring whether the unified signal decomposition principle generalizes to other multi-stage agent workflows beyond skill management