# CoSkill: Joint Reinforcement Learning of Reasoning and Meta-Skill Agents for Hierarchical Skill Evolution

> CoSkill's multi-agent co-training of reasoning and meta-skill agents over a hierarchical library achieves 98.4% ALFWorld and 90.6% WebShop success, outperforming all baselines.

- **Source:** [arXiv](https://arxiv.org/abs/2609.04865)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/ZvXgpG
- **Whiteboard:** https://picx.dev/p/ZvXgpG/image

## Summary

## Summary (Overview)

- **CoSkill** is a unified multi-agent reinforcement learning (RL) framework that jointly trains a **Reasoning Agent** and a learnable **Meta-Skill Agent** over a hierarchical skill library, enabling fine-grained skill–policy co-evolution.
- The framework casts static meta-skill workflows as an RL-trained agent, closing the skill–policy loop that prior paradigms leave open.
- CoSkill achieves **98.4% success on ALFWorld** (+3.5 pp) and **90.6% success on WebShop** (+6.2 pp), outperforming all prior skill-based and RL baselines.
- The approach demonstrates superior **early-stage sample efficiency, asymptotic performance, and wall-clock efficiency** compared to existing methods.
- Ablations show that RL training of the Meta-Skill Agent reduces destructive edits, improves retrieval reliability (from 12.4% to 0.2% empty step-bundle rate), and cuts redundant library growth by 47.6%.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Skill libraries improve the sample efficiency of agentic RL by enabling LLM agents to reuse procedural knowledge across tasks. However, existing skill-evolution paradigms exhibit structural shortcomings:

1. **Externally Orchestrated Skill Evolution** (e.g., SkillRL, D2Skill, ReSkill, Trace2Skill): Treats the skill library as a standalone knowledge base managed by external LLMs or hand-crafted rules. Skill evolution remains outside the policy-learning objective, leading to stale or misaligned skills.

2. **RL-Optimized Skill-Library Management** (e.g., SAGE, ARISE, Skill1): Incorporates library maintenance into the RL loop but treats skills as atomic units, leaving internal procedures unoptimized.

3. **Meta-Skill-Driven Skill Optimization** (e.g., SkillOpt, EvoSkill, SkillEvolver): Uses execution feedback to optimize individual skill procedures but relies on predefined meta-skill programs (rubric-based prompts, SKILL.md specifications) and typically freezes the reasoning agent.

### Key Design Insight

> "Formulating the meta-skill as an RL-trained agent rather than a fixed workflow allows its update policy to co-adapt with the reasoning policy."

CoSkill models the Reasoning and Meta-Skill Agents as a **cooperative team sharing a single LLM backbone**, jointly trained over a hierarchical skill library. This directly addresses the limitations of prior paradigms by:
- Closing the skill–policy loop through an agentic formulation
- Replacing fixed workflows with learnable meta-skills
- Moving beyond coarse library management to step-level editing

---

## Methodology

### Hierarchical Skill Library

CoSkill structures its persistent library as a collection of K task-indexed skill bundles:

$$\mathcal{B}_k = \left(s_k^{\text{task}}, \mathcal{S}_k^{\text{step}}\right) \tag{3}$$

Each bundle contains an episode-level task skill $s_k^{\text{task}}$ and a child set of step skills $\mathcal{S}_k^{\text{step}}$ for intermediate observations. Hierarchical retrieval proceeds in two stages:

$$
\begin{array}{l} 
k^{*} = \underset{k}{\arg \max} \, \text{sim}(f(x), f(q_k)), \\ 
j_t^{*} = \underset{j \in \mathcal{S}_{k^{*}}^{\text{step}}}{\arg \max} \, \text{sim}(f(o_t), f(o_{k^{*},j})), 
\end{array} \tag{4}
$$

where $f$ is a text encoder, $q_k$ is the task-skill retrieval key, and $o_{k,j}$ is the source observation of a step skill.

### Shared-Parameter Multi-Agent Semi-Markov Decision Process (MSMDP)

The interaction is formalized as an MSMDP:

$$\mathcal{M} = \langle \mathcal{S}, \Omega, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma, \mathcal{T} \rangle \tag{5}$$

**Global State Space**: The shared blackboard state is:

$$g_{\ell} = \left(x, \rho_{\ell}, \mathcal{H}_{\ell}, \mathcal{B}, \widetilde{\mathcal{B}}_{\ell}, \mathcal{Z}_{\ell}\right) \in \mathcal{S} \tag{6}$$

**Role-Specific Observations**:
- Reasoning Agent: $u_t^R = \mathcal{O}_R\left((x, o_t) \cup h_t \cup (Z^{\text{task}}, Z_t^{\text{step}})\right)$
- Meta-Skill Agent: $u_t^S = \mathcal{O}_S\left(\text{Select}_S(u_t^R) \cup a_t \cup (r_t, e_t, o_{t+1})\right)$

**Heterogeneous Action Space**: Both agents share policy $\pi_\theta$ but use role-specific prompts:

$$
\begin{array}{rl}
a_t \sim \pi_\theta(\cdot | u_t^R, p_R), & a_t \in \mathcal{A}_R \subseteq \mathcal{V}^*, \\
z_t \sim \pi_\theta(\cdot | u_t^S, p_S), & z_t \in \mathcal{A}_S \subseteq \mathcal{V}^*.
\end{array} \tag{9}
$$

The Meta-Skill Agent's action space is restricted to structured edits:

$$\mathcal{A}_S = \{\text{INSERT}(\hat{s}_t^{\text{step}}), \text{UPDATE}(s_t^{\text{step}}, \hat{s}_t^{\text{step}}), \text{DELETE}(s_t^{\text{step}}), \text{KEEP}(s_t^{\text{step}})\} \tag{10}$$

### Post-Edit Skill Verification Reward

To evaluate edit sequences through downstream execution, CoSkill runs M verification attempts with the edited bundle and computes:

$$\Delta_i^{\text{skill}} = \frac{1}{M}\sum_{m=1}^{M} R_i^{(m)} - R_i^{(0)} \tag{13}$$

This baseline-subtracted, mean-aggregated improvement score serves as the delayed episode-level reward for the Meta-Skill Agent's edit sequence.

### Joint Optimization via GiGPO

Role-specific credit assignment combines episode-level and step-level relative advantages:

$$A_{i,t}^{\rho} = A_{E,i}^{\rho} + \omega_{\rho} A_{S,i,t}^{\rho} \tag{14}$$

The joint objective updates the shared actor:

$$\mathcal{L}(\theta) = \mathcal{L}_{\text{GiGPO}}^{R}(\theta) + \mathcal{L}_{\text{GiGPO}}^{S}(\theta) \tag{15}$$

---

## Empirical Validation / Results

### Main Results (RQ1)

**Table 1: Main results on ALFWorld and WebShop**

| Method | ALFWorld Avg. (%) | WebShop Score | WebShop Succ. (%) |
|--------|-------------------|---------------|-------------------|
| **CoSkill (Ours)** | **98.4** | **95.9** | **90.6** |
| RetroAgent | 94.9 | 88.9 | 82.3 |
| Skill1 | 93.7 | 91.6 | 75.0 |
| D2Skill (Gemini-3-Flash) | 90.6 | 91.1 | 80.5 |
| GiGPO (no skills) | 90.8 | 84.4 | 72.8 |
| SkillRL | 89.9 | 85.2 | 72.7 |
| **Impv. vs. Best** | **+3.5** | **+4.3** | **+6.2** |

Key findings:
- CoSkill reaches **100% success on 5 of 6 ALFWorld task types**
- **+7.6 pp** over skill-free GiGPO on ALFWorld; **+17.8 pp** on WebShop success
- Outperforms D2Skill variants using Gemini-3-Flash/O3 editors, showing that **reward-aligned skill refinement matters more than editor scale**

### Ablation Study (RQ2)

**Table 2: Ablation results on ALFWorld (success rate %)**

| Method | Step 20 | Step 60 | Step 120 |
|--------|---------|---------|----------|
| **CoSkill (Ours)** | **70.31** | **89.06** | **95.31** |
| w/o Meta-Skill Agent RL | 57.81 (−12.50) | 87.50 (−1.56) | 92.19 (−3.12) |
| w/o Hierarchical Library | 56.25 (−14.06) | 87.50 (−1.56) | 93.75 (−1.56) |
| Alternating Updates = 10 | 62.50 (−7.81) | 64.06 (−25.00) | 78.12 (−17.19) |
| Alternating Updates = 20 | 59.38 (−10.93) | 70.31 (−18.75) | 81.25 (−14.06) |

### Meta-Skill Agent Learning Dynamics (RQ3)

- **With RL**: Edit distribution converges to 97–99% UPDATE; destructive DELETEs nearly disappear
- **Without RL**: Diffuse distribution persists (UPDATE drops to 54%, DELETE rises to 36%)
- **RL reduces cumulative library promotions by 47.6%** (387 vs. 739)
- **Retrieval hit rate**: CoSkill recovers ~100% vs. gradual deterioration without RL
- **Empty step-bundle rate**: Reduced from 12.4% to 0.2%

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Skill evolution as a learnable process**: CoSkill reframes meta-skills from fixed workflows to RL-trained agents, establishing that skill refinement should be co-optimized with reasoning rather than externally orchestrated.

2. **Shared-parameter multi-agent formulation**: The MSMDP formulation demonstrates that two distinct roles (reasoning and editing) can share a single backbone while maintaining role-specific credit assignment, avoiding the need for separate editor models.

3. **Cross-episode credit assignment**: The post-edit verification reward ($\Delta_i^{\text{skill}}$) provides a principled mechanism for assigning delayed execution feedback to edit sequences, drawing on meta-RL concepts.

### Practical Implications

1. **Sample and wall-clock efficiency**: CoSkill achieves higher performance with fewer training steps and less wall-clock time, making skill-augmented RL more practical for real-world deployment.

2. **Backbone independence**: The framework works with a 7B-parameter open-source model (Qwen2.5-7B), outperforming methods that rely on much larger closed-source editors (Gemini-3-Flash, O3).

3. **Stable skill libraries**: RL-aligned editing produces more selective, structurally stable skill representations that maintain high retrieval reliability over time.

---

## Conclusion

CoSkill demonstrates that multi-agent co-training—where a Reasoning Agent and a Meta-Skill Agent share a backbone and are jointly optimized—can simultaneously improve:

- **Sample efficiency** (faster early learning)
- **Asymptotic performance** (98.4% on ALFWorld, 90.6% on WebShop)
- **Training efficiency** (better success–wall-clock trade-off)

The key finding is that relying solely on powerful closed-source LLMs for external skill evolution does not guarantee alignment with the target environment or evolving reasoning policy. Instead, optimizing the Meta-Skill Agent from execution feedback yields more selective, stable, and policy-adaptive skill evolution.

**Future directions** include:
- Extending multi-agent co-training to broader agentic systems
- Developing more efficient interaction, credit-assignment, and update mechanisms
- Reducing training overhead while accelerating co-evolution

---

**Code**: Available at https://github.com/jinyuan-cookie/CoSkill

---

_Markdown view of https://picx.dev/p/ZvXgpG, served by PicX — AI-generated visual whiteboard summaries of research papers._