# SkillDuet: Co-Evolving Models and Skills through Response Feedback

> SkillDuet co-evolves model parameters and reusable textual skills via response-based Strength scoring, achieving state-of-the-art 98.57% on ALFWorld and 92.97% on WebShop.

- **Source:** [arXiv](https://arxiv.org/abs/2611.12109)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/xmtFV1
- **Whiteboard:** https://picx.dev/p/xmtFV1/image

## Summary

# SkillDuet: Co-Evolving Models and Skills through Response Feedback

## Summary (Overview)

- **Novel framework for model–skill co-evolution**: SkillDuet coordinates parameter updates and persistent textual skill evolution within a single reinforcement learning loop, addressing the under-explored coupling between skill internalization (into model parameters) and skill retention (as reusable external guidance).

- **Response-guided skill evolution**: Introduces *Strength*, a response-based score that ranks candidate skills by the magnitude of the shift they induce in the current model's response probabilities, enabling principled selection of skills for long-term consolidation.

- **State-of-the-art results**: With Qwen3-4B, SkillDuet achieves 98.57% success on ALFWorld and 92.97% on WebShop, exceeding the matched SEED-API baseline by 2.86 and 5.47 percentage points respectively.

- **Key empirical finding**: The utility of retained skills diminishes as training progresses—fixed guidance that helps early (+26.98 pp gain over steps 1–50) becomes detrimental later (−2.00 pp over steps 101–150), motivating the need for continuous skill reassessment.

- **Skill value is model-dependent**: The same learned global skill set improves performance when paired with the model it co-evolved with (SkillDuet: 97.62%) but *hurts* a different trained model (SEED-API: 89.76% vs. 93.81% without it), demonstrating that skill utility depends on both content and model compatibility.

---

## Introduction and Theoretical Foundation

### Background and Motivation

LLM agents retain interaction experience through two primary routes:
- **Training-free approaches**: Accumulate experience as external textual skills that guide later behavior
- **Training-based approaches**: Internalize experience into model parameters via SFT or RL

The authors identify a critical gap: these two routes are usually studied separately, yet they are fundamentally interdependent. Textual skills can serve dual roles—as supervision for parameter updates and as guidance for future tasks—but a skill that is useful for the current parameter update may not remain useful for the updated model's subsequent interactions.

### Key Observations Motivating the Work

1. **Skill utility decays during training**: On ALFWorld, guidance extracted from early trajectories substantially improves task completion early in training, but gains diminish as training progresses, turning negative in the final stage.

2. **Skill value is model-specific**: The same learned global skill set has different effects when supplied to two different trained models, showing that retained skill value depends on both content and compatibility with the current model.

### Theoretical Foundation: Recursive Self-Improvement (RSI)

The paper frames SkillDuet within the context of skill-level recursive self-improvement—the idea that a model can iteratively improve its own skills and capabilities through a self-reinforcing loop. The central challenge is that:

> "The model changes at every step, judging a skill's value through task execution is expensive, and the training objective itself provides no criterion for deciding which skills remain worth retaining."

---

## Methodology

### Framework Overview

SkillDuet couples two learning paths within a single interaction loop:

1. **Parameter-learning path**: Local skills provide privileged context for On-Policy Distillation (OPD), yielding token-level supervision for parameter updates at every step.

2. **Skill-evolution path**: Recent Local skills are scored by *Strength*, and the highest-scoring skills are periodically consolidated into a persistent global skill set (*Global*).

The updated model and Global jointly guide subsequent interactions, creating a feedback loop where internalization and consolidation reinforce each other.

### Key Components

#### 1. Local Skill Extraction

- An external Qwen3.8-27B analyzer processes trajectory groups collected at each training step
- **Success–failure pairing**: If a group contains both successful and failed trajectories, the analyzer compares one of each to generate a *shared* Local skill that captures the distinguishing decisions
- **Individual analysis**: Without a success–failure pair, each eligible trajectory yields its own Local skill (procedures from successes, corrections from failures)

#### 2. Local-Guided OPD for Parameter Updates

The combined training objective is:

$$
\begin{array}{c} g _ {i} = \mathbf {s g} [ \sigma (\beta (t _ {i} - s _ {i})) ], \quad \beta = 5, \\ \mathcal {L} _ {\mathrm{OPD}} = \frac {\sum_ {i} M _ {i} g _ {i} (\mathbf {s g} [ t _ {i} ] - s _ {i})}{\sum_ {i} M _ {i}}, \\ \mathcal {L} = \mathcal {L} _ {\mathrm{PPO}} + \lambda_ {\mathrm{OPD}} \mathcal {L} _ {\mathrm{OPD}} + \mathcal {L} _ {\mathrm{reg}}. \end{array}\tag{1}
$$

Where:
- $s_i$ and $t_i$ are student and teacher token log probabilities at position $i$
- $M$ is the supervision mask
- $\text{sg}$ is the stop-gradient operator
- $\mathcal{L}_{\mathrm{PPO}}$ is the proximal policy optimization objective
- $\mathcal{L}_{\mathrm{reg}}$ contains regularization terms
- $\beta = 5$ controls the confidence gating

#### 3. Strength: Response-Based Skill Selection

The *Strength* score ranks Local skills by the magnitude of the response shift they induce in the current model:

$$
\operatorname{Strength} (L) = \frac {1}{| H _ {L} |} \sum_ {r \in H _ {L}} | \Delta_ {r} (L) |.\tag{2}
$$

Where $\Delta_r(L)$ is the mean response log-probability shift over valid tokens of trajectory $r$, and $H_L$ is the scoring set of held-out trajectories not used to generate the skill.

#### 4. Periodic Global Consolidation

- Operates every 10 training steps
- Deduplicates recent Local candidates for each Global
- Retains top 50% by cached Strength score
- A synthesizer integrates selected skills with current Global, using hierarchical compression if context budget is exceeded

### Alternative Selection Criteria (for Comparison)

Three response-based scores were compared:

$$
\begin{array}{l} S (L) = \frac {1}{| H _ {L} |} \sum_ {r \in H _ {L}} | \Delta_ {r} (L) |, \\ U (L) = \frac {1}{| H _ {L} |} \sum_ {r \in H _ {L}} A _ {r} \Delta_ {r} (L), \\ C (L) = \frac {1}{| H _ {L} ^ {+} |} \sum_ {r \in H _ {L} ^ {+}} \Delta_ {r} (L) \\ - \frac {1}{| H _ {L} ^ {-} |} \sum_ {r \in H _ {L} ^ {-}} \Delta_ {r} (L). \end{array}\tag{3}
$$

Where $A_r$ is the trajectory-mean training advantage, and $H_L^+$, $H_L^-$ are successful and failed subsets of the scoring set.

---

## Empirical Validation / Results

### Main Results

**Table 1 | Performance of Qwen3-4B-family agents on ALFWorld and WebShop (%)**

| Method | Pick ↑ | Look ↑ | Clean ↑ | Heat ↑ | Cool ↑ | Pick2 ↑ | ALL ↑ | Score ↑ | Succ. ↑ |
|--------|--------|--------|---------|--------|--------|----------|--------|---------|---------|
| SkillRL (O3) | 90.0 | 63.6 | 92.3 | 42.9 | 52.0 | 40.9 | 67.2 | 89.4 | 75.7 |
| D2Skill (O3) | 89.4 | 54.5 | 72.4 | 60.0 | 66.7 | 50.0 | 72.7 | 89.1 | 75.8 |
| D2Skill (Gemini-3-Flash) | 88.6 | 66.7 | 75.0 | 60.0 | 54.2 | 52.6 | 69.6 | 90.0 | 82.8 |
| D2Skill + teacher SFT | 97.6 | 88.9 | 95.8 | 90.0 | 100.0 | 91.7 | 95.3 | 89.2 | 81.3 |
| OPD (ATOD) | — | — | — | — | — | — | 80.47 | — | 75.00 |
| GSPO (ICRL) | — | — | — | — | — | — | 82.0 | 82.1 | 65.0 |
| Vanilla | 70.83 | 40.00 | 19.75 | 18.52 | 20.00 | 26.67 | 37.38 | 13.85 | 0.00 |
| GRPO | 100.00 | 85.71 | 94.29 | 61.54 | 82.76 | 72.22 | 86.43 | 92.96 | 89.84 |
| SDAR | 100.00 | 100.00 | 100.00 | 54.55 | 70.37 | 80.95 | 87.86 | 89.83 | 86.72 |
| SEED | 100.00 | 100.00 | 100.00 | 100.00 | 90.48 | 87.50 | 96.09 | 86.23 | 75.78 |
| SEED-API | 100.00 | 93.33 | 96.30 | 83.33 | 95.00 | 100.00 | 95.71 | 91.31 | 87.50 |
| **SkillDuet** | **100.00** | **100.00** | **100.00** | **100.00** | **90.00** | **100.00** | **98.57** | **95.18** | **92.97** |

*Underlined entries are from cited papers. ALL is overall ALFWorld success; Score is WebShop's terminal reward.*

### Ablation: Global Consolidation and Response-Based Selection

**Table 2 | WebShop ablations of Global consolidation and Local selection**

| Configuration | Global | Succ. ↑ |
|---------------|--------|---------|
| SEED-API | None | 87.50 |
| Paired Local-only | None | 87.50 |
| SkillDuet: Random | Top50% | 89.06 |
| **SkillDuet: Strength** | **Top50%** | **92.97** |

Key findings:
- Paired Local generation alone does not improve performance (87.50% for both SEED-API and Paired Local-only)
- Response-based selection (Strength) outperforms Random by 3.91 percentage points
- Global consolidation adds value beyond Local-guided parameter updates alone

### Selection Criteria Comparison

In the historical ALFWorld candidate pool, **Strength** achieves the highest peak validation success (97.9%) and highest AUC (88.26%), exceeding Random's AUC by 9.29 percentage points. The paper argues that trajectory rewards and success labels describe behavior *before* the candidate skill is introduced and do not directly measure the skill's future value.

### Retention Ratio

Top50% matches Top25% on ALFWorld and achieves the best WebShop result, while Top75% performs worse on both tasks. Top50% is the default configuration.

### Global Content and Model Compatibility

**Table 3 | Controlled ALFWorld evaluations with fixed model parameters (%)**

| Model | Global | ALL ↑ | Avg ↑ |
|-------|--------|--------|-------|
| Qwen3-4B (Vanilla) | None | 37.38 | 32.63 |
| SEED-API | None | 93.81 | 92.43 |
| SEED-API | Learned Global | 89.76 | 88.42 |
| SkillDuet | None | 76.67 | 73.72 |
| SkillDuet | Length-matched neutral | 75.24 | 72.17 |
| **SkillDuet** | **Learned Global** | **97.62** | **97.50** |

Critical findings:
- SkillDuet retains substantial capability without Global (76.67%) but improves dramatically with it (97.62%)
- Replacing Global with length-matched neutral text reduces performance, confirming the gain depends on skill *content*
- The same skills *lower* SEED-API's success rate (93.81% → 89.76%), showing that skill benefit does **not** transfer automatically to another trained model

### Qualitative Example

The paper provides a paired execution example (Clean task): with Global guidance, the model picks up the knife, goes to the sink, cleans it, then places it—succeeding in 7 actions. Without Global, it places the uncleaned knife first and remains unfinished after 30 actions.

---

## Theoretical and Practical Implications

### Theoretical Implications

1. **Skill-level recursive self-improvement**: SkillDuet provides a concrete, operationalized mechanism for RSI where the model and its skills co-evolve, with each shaping the experience available for the other's improvement.

2. **Response feedback as a unified signal**: The paper demonstrates that model response feedback can serve dual purposes—immediate token-level distillation (OPD) and long-term skill retention (Strength)—unifying two previously separate uses of this signal.

3. **Skill utility is dynamic and context-dependent**: The results challenge static views of skill value. A skill's usefulness depends on (a) its content, (b) the current model's capabilities, and (c) the training stage. This has implications for any system maintaining persistent knowledge bases alongside learned models.

4. **Direction of probability shift ≠ benefit**: The paper argues that increasing the likelihood of successful responses (or decreasing failed ones) does not establish that a skill will improve future decisions. Response-shift *magnitude* proved more effective than reward weighting or success–failure contrast for selection.

### Practical Implications

1. **Architecture for skill management**: SkillDuet offers a practical template for systems that need to balance model internalization with external knowledge retention, with clear roles for extraction, scoring, selection, and synthesis components.

2. **Cost efficiency**: Strength scoring reuses already-sampled trajectories, requiring no additional environment interaction—an important practical consideration for expensive or slow environments.

3. **Model-specific skill curation**: The finding that skills do not transfer across models has practical implications for skill libraries and knowledge bases—they may need to be curated per model, not treated as model-agnostic assets.

4. **Training stability**: The diminishing returns of fixed guidance observed in Section 3.1 suggests that static skill sets may actively harm late-stage training, motivating the need for adaptive skill management in deployed RL systems.

---

## Conclusion

SkillDuet presents a framework for coordinating LLM parameter updates and the evolution of reusable textual skills during reinforcement learning. The core insight is that these processes are interdependent: skills guide model learning, while parameter updates change which skills remain useful. Model response feedback connects them in a shared learning loop.

**Key contributions:**
1. A framework for model–skill co-evolution that coordinates parameter updates and persistent skill updates within a single interaction loop
2. Response-guided skill evolution via the *Strength* score, extending response feedback from immediate distillation to long-term skill retention

**Experimental results** with Qwen3-4B demonstrate substantial gains over matched baselines (98.57% on ALFWorld, 92.97% on WebShop), with ablations confirming the benefits of response-based selection and Global consolidation.

**Future directions** suggested by this work include:
- Exploring alternative response-based signals for skill selection beyond magnitude
- Investigating how the consolidation frequency and retention ratio interact with different task types
- Understanding when skills become redundant or conflicting as models evolve
- Extending the co-evolution framework to other forms of persistent knowledge beyond textual skills

The paper concludes that SkillDuet provides "a concrete mechanism for skill-level recursive self-improvement through model–skill co-evolution," establishing a foundation for systems where models and their external knowledge grow together in a mutually reinforcing loop.

---

_Markdown view of https://picx.dev/p/xmtFV1, served by PicX — AI-generated visual whiteboard summaries of research papers._