Summary

• Core Problem Identified: This paper exposes a fundamental blind spot in skill–policy co-evolution methods for self-improving agents: current evaluations only measure performance on the fixed delivered skill bank, ignoring that deployed skill banks drift (skills are rewritten, added, or removed) over time.
• Theoretical Contribution: The authors prove (Proposition 1) that the shared "cooperative acceptance" rule—retaining only skill updates that help the current policy—places the delivered policy–bank pair at a peak over candidate banks, meaning no covered content-level rewrite can improve the policy and almost always degrades it.
• Empirical Finding: Using a new co-evolution fragility curve, the paper shows that sensitivity to skill drift grows as training proceeds across 4 different methods and 5 benchmarks (6 settings), with 18/18 endpoint pairs confirming the theoretical prediction.
• New Evaluation Protocol (CODA): The paper introduces the CO-evolution Drift Audit, a deployment-oriented protocol with three new metrics: consistency rate (CR), resistance rate (RR), and worst-case success rate (SR_min), along with a net-benefit gate.
• Headline Result: Across all settings, worst-case success under skill drift is 5–13 points below the reported peak, erasing 52–93% of the delivered bank's gain over the no-skill baseline—meaning the reported peak significantly overstates deployable competence.

Introduction and Theoretical Foundation

Background and Motivation

Skill–policy co-evolution is a promising paradigm where an agent's policy model and its skill bank improve each other during training. The agent solves tasks, distills experience into reusable skills, and uses those skills to strengthen its policy—so competence compounds over rounds. Methods like SkillRL, ReSkill, ARISE, and Evolving-RL follow this recipe and report large gains.

The critical gap: Current evaluations assume the skill bank remains fixed after training. However, real skill banks are non-stationary: skills are rewritten, entries are added or removed, and versions coexist—often changing faster than the policy is retrained. A deployed policy typically runs on a bank that has drifted from the one it was delivered with, yet existing protocols do not test whether performance is maintained.

Theoretical Framework

The paper formalizes the problem setting with the expected return:

$$J(\theta, S) = \mathbb{E}_{x \sim \mathcal{D}}[R(\pi_\theta(\cdot | x, S))] \tag{1}$$

where $x$ is a task from distribution $\mathcal{D}$, $S$ is a skill bank, and $\pi_\theta$ is the policy. Co-evolution alternates a policy step (improving $\theta$ against the current bank) and a bank step (proposing new banks and retaining one via an acceptance test).

Key theoretical constructs:

1. Idealized cooperative acceptance (Assumption 1): The retained bank maximizes the current policy's return over the round's candidate set:

$$S_t \in \arg\max_{S \in C_t} J(\theta_t, S) \tag{2}$$

2. Content-level neighborhood (Definition 1): $B(S)$ is the family of banks obtained by rewriting the textual content of existing skills while fixing the skill set and retrieval-side fields.

3. Perturbation response and fragility (Definition 2):

$$\Phi(\theta, S) = -\min \mathcal{R}(\theta, S) = J(\theta, S) - \min_{S' \in B(S)} J(\theta, S') \tag{4}$$

Proposition 1 (Co-evolution peak): Under Assumptions 1 and 2, the delivered pair $(\theta^*, S^*) = (\theta_T, S_T)$ satisfies:

$$\mathcal{R}(\theta^*, S^*) \subseteq (-\infty, 0] \tag{5}$$

This means no covered content-level variant improves the co-evolved policy—the delivered bank is a peak for the co-evolved policy, making it fragile to any drift.

Proposition 2 (The point-mass objective prices degradation at zero):

$$\mathbb{E}_{S \sim P}[J(\theta, S)] = J(\theta, S^*) - \mathbb{E}_{S \sim P}[J(\theta, S^*) - J(\theta, S)] \tag{6}$$

This shows that the training objective never marginalizes over banks—it optimizes at one bank while deployment presents a neighborhood, pricing degradation at zero.

Methodology

Five-Axis Skill-Drift Probe

The paper introduces a systematic perturbation framework with five axes of increasing structural drift:

An external LLM (gpt-4o-2024-08-06, temperature 0.7) generates perturbed variants. For T1–T3, retrieval runs once on the original bank and only selected content fields are replaced, preventing retrieval changes from confounding content changes.

Co-evolution Fragility Curve

The paper traces checkpoints along one training trajectory while holding the delivered bank $S^*$ and its perturbed variants fixed. Two metrics are tracked:

• Absolute drop: \text{Drop}_a(t) = \text{SR}(S^*) - \text{SR}(S^*_a) \tag{7}
• Retention performance (competence-normalized): \text{RP}_a(t) = \frac{\text{SR}(S^*_a)}{\text{SR}(S^*)} \tag{8}

CODA Protocol

CODA (CO-evolution Drift Audit) scores only content-level axes (T1–T3) with three metrics:

1. Gate (net skill benefit): \Delta_{\text{net}} = \text{SR}(S^*) - \text{SR}(\emptyset) \tag{9} — a method must clear this before robustness matters.

2. Consistency rate (CR): CR = \frac{1}{N}|\{i : c_i(S^*_{T1}) = c_i(S^*)\}| \tag{10} — fraction of outcomes unchanged by a meaning-preserving rewrite.

3. Resistance rate (RR): RR_a = \frac{1}{N}|\{i : c_i(\emptyset) = 1 \wedge c_i(S^*_a) = 1\}| \tag{11} — fraction solved both without skills and under corruption (for T2/T3).

4. Worst-case success rate: SR_{\min} = \min_{a \in \{T1,T2,T3\}} \text{SR}(S^*_a) \tag{13}

Empirical Validation / Results

Training Trend: Fragility Grows with Co-evolution

Unperturbed success rises from first to final retained anchor in all six settings:

• SkillRL-ALFWorld: 5.2% → 88.8%
• Evolving-RL-Mind2Web: 16.4% → 25.2%
• ARISE-AMC23: 15.5% → 87.4%

Key findings:

• At least one content-level axis has a significantly positive Drop slope in 5 of 6 settings
• Across 18 axis–setting pairs, 12 are significant (4 each on T1, T2, T3)
• The one-sided prediction holds 18/18: no content-level rewrite beats evaluation noise
• A meaning-preserving rewrite alone costs 10.9 points on ReSkill-ScienceWorld (peak 50.3%)
• The bank still helps under drift: SR_min stays above the no-skill anchor in all settings

CODA Report Card

Headline findings from the report card:

• Worst-case success trails the peak by 4.9–12.6 points across all settings
• Drift consumes 52–93% of the skill benefit in every setting
• The worst case is often the mildest change: in 3/6 settings, T1 (meaning-preserving) falls below the harsher T2/T3 rates
• Consistency rate sits below its noise floor in every setting (77.5–87.0 vs. CR₀ = 91.9–94.4), meaning 13–23% of tasks flip outcome under a meaning-preserving rewrite vs. 6–8% under decoding noise alone

The Cost in Standalone Ability

A pure-RL policy trained with skills disabled reaches 70.6% (last-five mean) and 87.5% (peak), while the co-evolved policy stripped of its bank scores only 57.6%—co-evolution leaves the underlying policy worse at acting alone.

Theoretical and Practical Implications

Theoretical Implications

1. The co-evolution trap is structural, not incidental: The paper proves that the shared acceptance rule (cooperative acceptance) fundamentally ties a policy's competence to the bank it co-evolved with. The selection bias—not co-evolution alone—governs whether robustness emerges.

2. The point-mass objective is blind to fragility (Proposition 4): Under a richness condition, the training objective $J(\cdot, S^*)$ cannot distinguish a fragile policy from a robust one—its maximizers span the full achievable range of fragility.

3. Adversarial co-evolution flips the conclusion: Replacing the cooperative gate with an adversarial one ($S_t \in \arg\min_{S \in C_t} J(\theta_t, S)$) drives fragility to zero, confirming that the sign of the acceptance test determines whether training pressure acts on the worst case.

Practical Implications

1. Reported peaks overstate deployable competence: The majority of the advertised gain from skill–policy co-evolution is contingent on the bank's exact wording and disappears once it drifts.

2. CODA provides a shared audit standard: The protocol reframes the target of co-evolution from peak performance on the co-evolved bank to the ability to harness skills however they evolve.

3. The natural fix direction: Training against a distribution of bank perturbations (the distributional objective of Eq. 6) rather than a single fixed bank—though this remains future work.

Conclusion

This work shifts the evaluation of skill–policy co-evolution from performance on a single delivered pair to robustness under realistic deployment-time skill drift. The key takeaways are:

1. Cooperative acceptance is the source of dependence: The shared acceptance rule places the delivered pair at a peak over candidate banks, making it fragile to any drift.

2. Fragility strengthens over training: The co-evolution fragility curve shows that sensitivity to skill drift grows as co-evolution proceeds—a process that raises peak performance also creates sensitivity to bank drift.

3. CODA operationalizes the findings: The protocol audits whether a delivered policy remains consistent under benign rewrites and resists incomplete or misleading skills, providing a shared standard for future methods.

Future directions:

• Turning the distributional objective of Eq. (6) into a practical training method while keeping the net-benefit gate satisfied
• Studying whether broader exposure (more benchmarks, larger banks, more data) reduces sensitivity to wording or merely raises the peak from which the worst case falls away
• The paper explicitly states: "The delivered peak is therefore informative but insufficient: future methods should preserve its gains while learning to interpret, reject, and adapt to skills as the bank evolves."