Self-Evolving Agents Can Repair Failures but Struggle to Generalize from Them

Summary (Overview)

• Core finding: The paper identifies a consistent repair–generalization gap in self-evolving LLM agents: failure-derived experience can help repair previously failed tasks, but rarely yields sustained improvements on unseen tasks.
• New evaluation setting: Introduces Cold-Start Failure-Only Evolution (CFOE), where agents start with no accumulated experience and learn exclusively from failed interactions, without access to ground-truth answers, expert solutions, or verifier feedback.
• Scope: Evaluates three self-evolving paradigms (ReasoningBank, SkillOpt, Evo-Harness) across 10 LLM backbones and four interactive environments (ALFWorld, τ²-Bench-Verified, WebArena-Verified, WebShop).
• Causal analysis: Traces the gap to (1) unreliable failure interpretation (false-success attribution, misattribution, distorted consolidation, harmful guidance) and (2) limited transferability of even verified-correct experience.
• Key insight: Successful trajectories provide substantially stronger transfer gains than verifier feedback or additional failure exploration, suggesting current agents learn "what went wrong" better than "what reliably works."

---

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a fundamental question in the emerging "era of experience" for LLM agents: Can agents bootstrap new capabilities from failure itself, rather than relying on success as a prerequisite for learning?

Existing self-evolution approaches typically assume access to positive learning signals:

• Successful trajectories
• Verifier feedback
• Already useful experience

However, in challenging open environments, these signals may be scarce precisely when adaptation is most needed. Agents can start without prior experience, repeatedly fail on unfamiliar tasks, and receive little guidance on how to improve.

Research Questions

The paper investigates three research questions:

• RQ1: To what extent can agents learn from failures under CFOE, and does such learning generalize to unseen tasks?
• RQ2: What prevents failure-derived experience from becoming transferable capability?
• RQ3: What learning conditions are missing for effective experience transfer?

Formal Framework

The paper formulates an online self-evolving agent with:

• Fixed backbone policy $\pi_\theta$
• Evolving external state $S_t$
• Updater $U$
• Verifier $V$

Given a sequential task stream $\{x_t\}_{t=1}^T$, the agent executes each task with its current external state:

$$\tau_t \sim \pi_\theta(\cdot | x_t, S_{t-1})$$

where $\theta$ remains fixed throughout evolution. The verifier judgment $y_t = V(x_t, \tau_t) \in \{0, 1\}$ is available only to the experimental controller. The state update follows:

$$S_t = \begin{cases} S_{t-1}, & y_t = 1, \\ U(S_{t-1}, \tau_t), & y_t = 0. \end{cases}$$

Thus, successful executions never contribute to evolution, while success/failure labels, reward scores, verifier explanations, ground-truth answers, and expert solutions remain inaccessible throughout the process.

---

Methodology

Self-Evolving Agent Paradigms

Three representative paradigms are studied, all with frozen backbones:

Backbone Models

Experiments cover 10 LLM backbones spanning multiple families, scales, and architectures:

• Qwen3.5 family: 4B, 9B, 27B (dense); Qwen3.5-35B-A3B (MoE)
• Gemma 4 family: Gemma-4-12B-it, Gemma-4-31B-it (dense); Gemma-4-26B-A4B-it (MoE)
• Selected frameworks: DeepSeek-V4-Pro, GLM-5.1, GPT-5.6 Sol

Environments

Four quality-verified interactive environments:

• ALFWorld: Household embodied task execution
• τ²-Bench-Verified: Long-horizon interactive tool use
• WebArena-Verified: Web navigation (Shopping, Shopping Admin, Reddit, GitLab)
• WebShop: Online shopping

Failure-Dense Stream Construction

For each backbone and benchmark, a challenging task stream is constructed where each task has failed in 3 independent preliminary rollouts, inducing high failure density during online self-evolution.

CFOE Constraints

1. Cold start: Evolution starts from empty external state
2. Failure-only: Only failed trajectories used for updating
3. Reward-blind: All verifier feedback hidden from agent and updater

Error Analysis Protocol

For each evolution update, the researchers manually trace:

1. How the agent interprets the failed trajectory
2. How this interpretation informs failure attribution
3. How the resulting explanation is consolidated into experience

---

Empirical Validation / Results

RQ1: Failure-Only Evolution Repairs but Does Not Generalize

Key finding: Across frameworks, models, and environments, a consistent repair–generalization gap emerges.

• Limited held-out gains: Successive checkpoints on unseen held-out sets show minimal improvement, with some configurations declining as experience accumulates.
• Partial source-task recovery: Retrying failed source tasks shows partial recovery (rates vary across backbones, frameworks, environments).
• Recovery does not predict transfer: E.g., Qwen3.5-27B with SkillOpt recovers more ALFWorld source tasks than Qwen3.5-4B, yet its held-out performance still declines.

Long-horizon scaling (800 episodes): Extending evolution does not close the gap—held-out performance plateaus or declines further despite continued source-task recovery.

RQ2: Causes of Limited Experience Transfer

Errors in Experience Formation

Four failure modes are identified:

Table 1: Percentage of reviewed updates exhibiting each failure mode

Key patterns: Misattribution (M2) is particularly common in τ²-Bench-Verified; False Success (M1) is more prominent in ALFWorld; stronger models exhibit fewer errors; harness evolution shows lower error rates than memory or skill evolution.

Utilization and Transfer of Correct Experience

Even when retaining only manually verified-correct experience:

• Source-task recovery modestly improves in all five configurations
• Final held-out scores remain at or below the no-experience baseline in four configurations

This shows that even correct experience remains overly tied to specific steps and conditions of the source task, indicating a gap in extracting generalizable knowledge from dense failures.

RQ3: Effects of Learning Conditions on Experience Transfer

The paper systematically relaxes CFOE along three dimensions:

Key results:

• Successful interactions produce substantially larger held-out improvements than failure-only learning
• Verifier feedback improves failure-only learning but gains remain limited compared to successes
• Interactive failure diagnosis (Resume) offers further learning opportunities with limited transfer gains
• Additional failure evidence (feedback, interaction) helps learning, but extracting transferable experience remains challenging

---

Theoretical and Practical Implications

Theoretical Implications

1. Diagnosis bottleneck: Agents may turn plausible but unsupported interpretations of failures into persistent experience, which then guides subsequent behavior in ways that conflict with task constraints.

2. Transfer gap beyond diagnosis: Even verified-correct experience fails to transfer, suggesting that correct content alone is insufficient—experience must be abstracted from source-task-specific conditions to be useful.

3. Asymmetric information value: Failures reveal what went wrong, while successful trajectories show what reliably works and how useful actions compose into complete procedures. This asymmetry explains why success provides substantially stronger transfer support.

4. Paradigm differences: Validation-based acceptance (SkillOpt) yields more stable held-out performance than memory or harness evolution, though still with limited gains.

Practical Implications

1. Evaluation design: The repair–generalization gap suggests that evaluating self-evolving agents solely on source-task recovery overestimates their capabilities. Held-out evaluation is essential.

2. System design: Current self-evolving agents are better at repairing observed failures than building generalizable capabilities from failure. Systems deployed in open environments should not assume failure-driven learning will yield transferable competence.

3. Oversight mechanisms: The high rates of error modes (M1–M4) highlight the need for verification and oversight of accumulated experience, as incorrect guidance can be carried forward as persistent action rules.

---

Conclusion

Main Takeaways

1. CFOE reveals a fundamental limitation: Current self-evolving agents can extract useful local corrections from failed interactions but struggle to turn them into transferable capability.

2. Two contributing factors: The repair–generalization gap stems from (a) unreliable failure interpretation during experience formation and (b) limited transferability of even correct experience.

3. Success is informationally valuable: Successful behavioral evidence provides substantially stronger support for transferable evolution than richer failure diagnosis alone—suggesting that positive examples carry information that failure analysis cannot easily recover.

Future Directions

• Broader task distributions: Extending analysis to multi-agent interactions and multimodal inputs
• Longer time horizons: Real-world agents may self-evolve over unbounded horizons; whether transferable gains emerge beyond the studied regime (800 tasks) remains open
• Principled mechanisms: Exploring more general, principled, and verifiable mechanisms to support transferable learning from failures in experience-driven AI systems

Closing Statement

"Current self-evolving agents are better at repairing observed failures than at building generalizable capabilities from failure, highlighting a fundamental challenge for autonomous learning in open environments."