Summary (Overview)

• CoSkill is a unified multi-agent reinforcement learning (RL) framework that jointly trains a Reasoning Agent and a learnable Meta-Skill Agent over a hierarchical skill library, enabling fine-grained skill–policy co-evolution.
• The framework casts static meta-skill workflows as an RL-trained agent, closing the skill–policy loop that prior paradigms leave open.
• CoSkill achieves 98.4% success on ALFWorld (+3.5 pp) and 90.6% success on WebShop (+6.2 pp), outperforming all prior skill-based and RL baselines.
• The approach demonstrates superior early-stage sample efficiency, asymptotic performance, and wall-clock efficiency compared to existing methods.
• Ablations show that RL training of the Meta-Skill Agent reduces destructive edits, improves retrieval reliability (from 12.4% to 0.2% empty step-bundle rate), and cuts redundant library growth by 47.6%.

---

Introduction and Theoretical Foundation

Background and Motivation

Skill libraries improve the sample efficiency of agentic RL by enabling LLM agents to reuse procedural knowledge across tasks. However, existing skill-evolution paradigms exhibit structural shortcomings:

1. Externally Orchestrated Skill Evolution (e.g., SkillRL, D2Skill, ReSkill, Trace2Skill): Treats the skill library as a standalone knowledge base managed by external LLMs or hand-crafted rules. Skill evolution remains outside the policy-learning objective, leading to stale or misaligned skills.

2. RL-Optimized Skill-Library Management (e.g., SAGE, ARISE, Skill1): Incorporates library maintenance into the RL loop but treats skills as atomic units, leaving internal procedures unoptimized.

3. Meta-Skill-Driven Skill Optimization (e.g., SkillOpt, EvoSkill, SkillEvolver): Uses execution feedback to optimize individual skill procedures but relies on predefined meta-skill programs (rubric-based prompts, SKILL.md specifications) and typically freezes the reasoning agent.

Key Design Insight

"Formulating the meta-skill as an RL-trained agent rather than a fixed workflow allows its update policy to co-adapt with the reasoning policy."

CoSkill models the Reasoning and Meta-Skill Agents as a cooperative team sharing a single LLM backbone, jointly trained over a hierarchical skill library. This directly addresses the limitations of prior paradigms by:

• Closing the skill–policy loop through an agentic formulation
• Replacing fixed workflows with learnable meta-skills
• Moving beyond coarse library management to step-level editing

---

Methodology

Hierarchical Skill Library

CoSkill structures its persistent library as a collection of K task-indexed skill bundles:

$$\mathcal{B}_k = \left(s_k^{\text{task}}, \mathcal{S}_k^{\text{step}}\right) \tag{3}$$

Each bundle contains an episode-level task skill $s_k^{\text{task}}$ and a child set of step skills $\mathcal{S}_k^{\text{step}}$ for intermediate observations. Hierarchical retrieval proceeds in two stages:

$$\begin{array}{l} k^{*} = \underset{k}{\arg \max} \, \text{sim}(f(x), f(q_k)), \\ j_t^{*} = \underset{j \in \mathcal{S}_{k^{*}}^{\text{step}}}{\arg \max} \, \text{sim}(f(o_t), f(o_{k^{*},j})), \end{array} \tag{4}$$

where $f$ is a text encoder, $q_k$ is the task-skill retrieval key, and $o_{k,j}$ is the source observation of a step skill.

Shared-Parameter Multi-Agent Semi-Markov Decision Process (MSMDP)

The interaction is formalized as an MSMDP:

$$\mathcal{M} = \langle \mathcal{S}, \Omega, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma, \mathcal{T} \rangle \tag{5}$$

Global State Space: The shared blackboard state is:

$$g_{\ell} = \left(x, \rho_{\ell}, \mathcal{H}_{\ell}, \mathcal{B}, \widetilde{\mathcal{B}}_{\ell}, \mathcal{Z}_{\ell}\right) \in \mathcal{S} \tag{6}$$

Role-Specific Observations:

• Reasoning Agent: $u_t^R = \mathcal{O}_R\left((x, o_t) \cup h_t \cup (Z^{\text{task}}, Z_t^{\text{step}})\right)$
• Meta-Skill Agent: $u_t^S = \mathcal{O}_S\left(\text{Select}_S(u_t^R) \cup a_t \cup (r_t, e_t, o_{t+1})\right)$

Heterogeneous Action Space: Both agents share policy $\pi_\theta$ but use role-specific prompts:

$$\begin{array}{rl} a_t \sim \pi_\theta(\cdot | u_t^R, p_R), & a_t \in \mathcal{A}_R \subseteq \mathcal{V}^*, \\ z_t \sim \pi_\theta(\cdot | u_t^S, p_S), & z_t \in \mathcal{A}_S \subseteq \mathcal{V}^*. \end{array} \tag{9}$$

The Meta-Skill Agent's action space is restricted to structured edits:

$$\mathcal{A}_S = \{\text{INSERT}(\hat{s}_t^{\text{step}}), \text{UPDATE}(s_t^{\text{step}}, \hat{s}_t^{\text{step}}), \text{DELETE}(s_t^{\text{step}}), \text{KEEP}(s_t^{\text{step}})\} \tag{10}$$

Post-Edit Skill Verification Reward

To evaluate edit sequences through downstream execution, CoSkill runs M verification attempts with the edited bundle and computes:

$$\Delta_i^{\text{skill}} = \frac{1}{M}\sum_{m=1}^{M} R_i^{(m)} - R_i^{(0)} \tag{13}$$

This baseline-subtracted, mean-aggregated improvement score serves as the delayed episode-level reward for the Meta-Skill Agent's edit sequence.

Joint Optimization via GiGPO

Role-specific credit assignment combines episode-level and step-level relative advantages:

$$A_{i,t}^{\rho} = A_{E,i}^{\rho} + \omega_{\rho} A_{S,i,t}^{\rho} \tag{14}$$

The joint objective updates the shared actor:

$$\mathcal{L}(\theta) = \mathcal{L}_{\text{GiGPO}}^{R}(\theta) + \mathcal{L}_{\text{GiGPO}}^{S}(\theta) \tag{15}$$

---

Empirical Validation / Results

Main Results (RQ1)

Table 1: Main results on ALFWorld and WebShop

Key findings:

• CoSkill reaches 100% success on 5 of 6 ALFWorld task types
• +7.6 pp over skill-free GiGPO on ALFWorld; +17.8 pp on WebShop success
• Outperforms D2Skill variants using Gemini-3-Flash/O3 editors, showing that reward-aligned skill refinement matters more than editor scale

Ablation Study (RQ2)

Table 2: Ablation results on ALFWorld (success rate %)

Meta-Skill Agent Learning Dynamics (RQ3)

• With RL: Edit distribution converges to 97–99% UPDATE; destructive DELETEs nearly disappear
• Without RL: Diffuse distribution persists (UPDATE drops to 54%, DELETE rises to 36%)
• RL reduces cumulative library promotions by 47.6% (387 vs. 739)
• Retrieval hit rate: CoSkill recovers ~100% vs. gradual deterioration without RL
• Empty step-bundle rate: Reduced from 12.4% to 0.2%

---

Theoretical and Practical Implications

Theoretical Contributions

1. Skill evolution as a learnable process: CoSkill reframes meta-skills from fixed workflows to RL-trained agents, establishing that skill refinement should be co-optimized with reasoning rather than externally orchestrated.

2. Shared-parameter multi-agent formulation: The MSMDP formulation demonstrates that two distinct roles (reasoning and editing) can share a single backbone while maintaining role-specific credit assignment, avoiding the need for separate editor models.

3. Cross-episode credit assignment: The post-edit verification reward ($\Delta_i^{\text{skill}}$) provides a principled mechanism for assigning delayed execution feedback to edit sequences, drawing on meta-RL concepts.

Practical Implications

1. Sample and wall-clock efficiency: CoSkill achieves higher performance with fewer training steps and less wall-clock time, making skill-augmented RL more practical for real-world deployment.

2. Backbone independence: The framework works with a 7B-parameter open-source model (Qwen2.5-7B), outperforming methods that rely on much larger closed-source editors (Gemini-3-Flash, O3).

3. Stable skill libraries: RL-aligned editing produces more selective, structurally stable skill representations that maintain high retrieval reliability over time.

---

Conclusion

CoSkill demonstrates that multi-agent co-training—where a Reasoning Agent and a Meta-Skill Agent share a backbone and are jointly optimized—can simultaneously improve:

• Sample efficiency (faster early learning)
• Asymptotic performance (98.4% on ALFWorld, 90.6% on WebShop)
• Training efficiency (better success–wall-clock trade-off)

The key finding is that relying solely on powerful closed-source LLMs for external skill evolution does not guarantee alignment with the target environment or evolving reasoning policy. Instead, optimizing the Meta-Skill Agent from execution feedback yields more selective, stable, and policy-adaptive skill evolution.

Future directions include:

• Extending multi-agent co-training to broader agentic systems
• Developing more efficient interaction, credit-assignment, and update mechanisms
• Reducing training overhead while accelerating co-evolution

---

Code: Available at https://github.com/jinyuan-cookie/CoSkill