Are Self-Evolving Agents Continual Learners?

Authors: Xiaoyin Chen, Yizheng Huang, Yinglun Xu, Ziyun Zhao, Emmett McQuinn, Julie Liu, Hope Yao (Mila – Quebec AI Institute, Université de Montréal, Google DeepMind)

---

Summary (Overview)

• Core question: Are nonparametric self-evolving LLM agents (which adapt by rewriting prompts, memory, and skills while keeping weights frozen) effective continual learners on drifting task streams?
• Key finding: Self-evolution improves aggregate performance in most settings, but higher plasticity consistently comes at the cost of stability — no method achieves both simultaneously.
• Three representative methods evaluated: ReasoningBank (experience retrieval), GEPA (prompt self-modification), and Meta-Harness (full workspace modification) across four drift scenarios (topic, workflow, domain, compound).
• Two failure mechanisms identified: (1) Self-modifying methods apply global edits to shared prompts/workspaces, causing cross-task interference and overwriting of earlier knowledge; (2) Similarity-based retrieval injects foreign memories and suffers from retrieval collision when similar queries require conflicting workflows.
• Conclusion: Current self-evolving agents remain incomplete continual learners, requiring new mechanisms for isolating task-specific updates and conditioning retrieval beyond surface query similarity.

---

Introduction and Theoretical Foundation

Background and Motivation

LLM agents are increasingly deployed in environments where new tasks, tools, and workflows continuously emerge after model weights are frozen. An agent serving users over extended periods must:

• Learn each new task as it arrives
• Preserve previously acquired knowledge

This resembles the classical continual learning problem, but at a pace that conventional gradient updates cannot sustain — retraining after every session is prohibitively expensive, and fine-tuning on recent interactions erodes general capabilities.

The Nonparametric Alternative

The in-context learning ability of LLMs suggests an alternative: instead of updating weights, the agent updates its harness (system prompt, memory, skill libraries, control code). Two paradigms have emerged:

1. Self-modification: Rewrites the artifacts the agent runs on (system prompt, harness code) — e.g., GEPA, Meta-Harness
2. Experience augmentation: Appends reflections/trajectories to an external store, retrieved during execution — e.g., ReasoningBank

Both paradigms show gains on stationary, single-domain benchmarks, but their continual learning behavior remains unexamined.

Theoretical Formulation

A task family is defined as:

$$\mathcal{F} = (\mathcal{X}, \mathcal{D}, \mathcal{Y}, \mathcal{R})$$

where $\mathcal{X}$ is the input space, $\mathcal{D}$ is the input distribution, $\mathcal{Y}$ is the output space, and $R: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$ is a reward function. In the agentic setting, episodes are trajectories and rewards are binary task success: $R(x,y) \in \{0,1\}$.

A task stream over $K$ families with weight schedule $w = (w_t)_{t=0}^{T-1}$ draws inputs from the mixture:

$$x_t \sim P_t := \sum_{k=1}^{K} w_{t,k} \mathcal{D}_k, \qquad t \in \{0, \dots, T-1\}. \tag{1}$$

The continual learning objective maximizes expected average performance over the stream:

$$\bar{J}_w := \mathbb{E}_{S_w} \bigg[ \frac{1}{T} \sum_{t=1}^{T} J\big(\pi^{(t)}\big) \bigg], \tag{3}$$

where $J(\pi) = \frac{1}{K} \sum_{k=1}^{K} J_k(\pi)$ evaluates performance on all $K$ families — creating the fundamental tension between plasticity and forgetting.

The Evolver

A self-evolving agent updates its policy through an evolver $f$:

$$c_{t+1} = f\big(c_t, \mathcal{H}_t\big), \tag{5}$$

where $c_t$ is the harness at time $t$ and $\mathcal{H}_t$ is the accumulated experience history. The evolver's performance under schedule $w$ is:

$$\bar{J}_w(f) := \mathbb{E}_{S_{w,f}} \bigg[ \frac{1}{T} \sum_{t=1}^{T} J\big(\pi^{(t)}\big) \bigg]. \tag{6}$$

---

Methodology

Task Stream Configurations

Four drifting stream scenarios were constructed, each with one-hot, piecewise-constant weight schedules (contiguous blocks of 50 training tasks per group):

Evaluation Metrics

Seed baseline: Per-family accuracy $B_k := A[0,k]$ from the initial harness before any evolution.

Overall performance (offset against seed, in percentage points):

$$\bar{\Delta} = \frac{1}{MK} \sum_{m=1}^{M} \sum_{k=1}^{K} \big(A[t_m,k] - B_k\big), \quad \text{Term.} = \frac{1}{K} \sum_{k=1}^{K} \big(A[T,k] - B_k\big). \tag{8}$$

Plasticity $P$ — learning during a family's own block:

$$P = \frac{1}{K} \sum_{k=1}^{K} \big(A[\tau_k, k] - B_k\big). \tag{9}$$

Stability $S$ — combined pre-block interference and post-block forgetting:

$$I_k = \frac{1}{M_k} \sum_{0 < t_m \leq \sigma_k} \max\big(0, B_k - A[t_m,k]\big), \quad F_k = \max\big(0, A[\tau_k,k] - A[T,k]\big), \tag{10}$$ $$S = -\frac{1}{K} \sum_{k=1}^{K} (I_k + F_k) = -(\bar{I} + \bar{F}). \tag{11}$$

Experimental Setup

• Base model: Gemini 3.6 Flash in a ReAct-style loop
• Evaluation: Checkpoints at every block boundary (every 50 positions), 8 rollouts per instance
• Methods compared:
  • ReasoningBank: Distills episodes into memory items, retrieves top-1 most similar item
  • GEPA: Rewrites a shared strategy section in the system prompt via reflective mutation
  • Meta-Harness: Proposes edits to full workspace (prompt, memory, skills, controller code) via coding agent with validation gate

---

Empirical Validation / Results

1. Current Methods Trade Stability for Plasticity

Table 1 (abbreviated) | Continual learning performance across drift scenarios (pp. over seed):

Key patterns:

• GEPA has highest plasticity (+6.2 avg) but largest stability loss (−4.3 avg) — rewriting the shared prompt gives broad behavioral change at the cost of cross-task interference
• ReasoningBank achieves best stability (−1.9 avg) with roughly half of GEPA's plasticity — localization via retrieval constrains interference
• Meta-Harness is least stable on domain/compound drift; its apparent stability on topic/workflow comes from its validation gate rejecting half or more candidate updates (freezing the workspace at the cost of low plasticity)

A consistent negative correlation between plasticity and stability emerges across all scenarios and methods ($n=12$).

2. Self-Modification Causes Cross-Task Interference and Overwriting

Pre-block interference: Rules learned on earlier blocks apply unconditionally to unseen tasks:

• Domain drift: ScienceWorld training lowers BFCL-Base by 5.5 (GEPA), 1.5 (Meta-Harness) before BFCL tasks arrive
• Compound drift: BIRD blocks degrade BFCL-Miss to −6.8 (GEPA) before any BFCL task arrives
• Evolved prompts embed SQLite exploration instructions that cause the agent to invoke lookup tools on BFCL tasks instead of asking for missing arguments

Post-block forgetting via compression and overwriting (Figure 3a,b):

• Topic drift: BIRD-F1's prompt share falls from 100% → 75.2% → 40.2% as later blocks write over it; F1 accuracy drops from +7.5 → +3.5 → −2.0
• Compound drift: Seven BIRD SQL sections (3,386 chars) compressed to a single 1,034-char section; BIRD-F1's share drops from 59.5% → 5.8%, erasing its +4.0 gain

Behavioral reversal under conflicting workflows (Figure 3c): With BFCL-Miss and BFCL-Base sharing nearly identical queries but requiring opposite workflows:

• After BFCL-Miss training: clarification rate rises to 29.8% (GEPA) / 60.8% (Meta-Harness)
• After BFCL-Base rewrites instructions: clarification rate drops to 11.2% / 8.5%, causing BFCL-Miss accuracy drops of 13.0 / 10.2 points

3. Similarity-Based Retrieval Cannot Prevent Interference or Collision

Domain separation prevents forgetting: When domains are well-separated, cross-group nearest-match cosine similarity (0.57–0.63) is much lower than within-group (0.80–0.95), yielding $\bar{F} = 0.0$ on domain drift.

However, retrieval still fails:

• Pre-block interference: Before a group's block, 100% of retrievals are foreign memories (zero in-group entries exist), pulling accuracy below seed (e.g., BFCL-Miss −3.2 on compound drift)
• Query similarity is unreliable: Same-task BIRD queries across topics have similarity (0.60–0.64) barely above cross-domain queries (0.56–0.63)
• Retrieval collision: BFCL-Miss and BFCL-Base queries have cross-group nearest-match similarity (0.835–0.840) indistinguishable from within-group (0.834–0.837). At stream end, 52% of BFCL-Miss episodes retrieve BFCL-Base memories (and 46% vice versa), causing accuracy drops of 8.0 points

---

Theoretical and Practical Implications

For Agent Design

1. Harness update scope is a plasticity-stability dial: Expanding the editable surface (memory → prompt → full workspace) increases plasticity but proportionally reduces stability. Designers must consciously choose where on this trade-off their deployment lives.

2. Global edits are fundamentally problematic: Any mechanism that writes task-specific rules to a shared surface applied to all rollouts will cause interference on unseen tasks and overwriting of earlier knowledge.

3. Similarity is not compatibility: Retrieval collision shows that embedding-space similarity captures topical overlap but cannot determine whether a stored trajectory's workflow is compatible with the incoming task.

Required Advancements

The authors' findings call for:

• Mechanisms that isolate task-specific updates in the harness (e.g., modular or scoped prompt sections with explicit task-group identification)
• Conditioned retrieval beyond surface similarity (e.g., workflow-aware or verification-based retrieval that can reject incompatible memories)
• Explicit detection of distribution shift at block boundaries — the agent currently must infer stream structure on its own, which fails when domains look similar but require different behaviors

Broader Significance

This work bridges the continual learning and self-evolving agent literatures, formalizing the first framework for evaluating harness-level adaptation under non-stationary task streams, and revealing that the "free" adaptation of frozen-weight agents still exhibits the same plasticity-stability dilemma as parametric continual learning.

---

Conclusion

Main takeaway: Current nonparametric self-evolving agents are incomplete continual learners.

• Self-modifying methods (GEPA, Meta-Harness) achieve plasticity through global rewrites, but cause pre-block interference and post-block forgetting — most severely when consecutive blocks share a domain but require conflicting workflows
• Experience augmentation (ReasoningBank) prevents overwriting across distant domains but injects foreign memories before a task group is encountered and suffers retrieval collision when distinct workflows share the embedding space
• No current method achieves both high plasticity and high stability on any drift scenario

Future directions: The authors call for harness architectures that support task-isolated updates and workflow-aware retrieval, moving beyond similarity-based matching to condition on whether a stored trajectory's behavioral requirements are compatible with the incoming task.