# Can Coding Agents Sustain Continual Evolution without Forgetting?

> Continual harness evolution for coding agents suffers severe catastrophic forgetting, with verification conflicts causing up to 100% performance drops on prior tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2611.03219)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/WJRKRV
- **Whiteboard:** https://picx.dev/p/WJRKRV/image

## Summary

## Summary

- **Problem Definition**: This paper introduces **continual harness self-evolution**—the successive evolution of coding agent harnesses across different task types (e.g., code repair, test generation, feature generation)—and identifies **catastrophic forgetting** as a critical, previously unexplored challenge in this setting.
- **Empirical Finding**: Both representative harness evolution methods (AHE and Meta-Harness) exhibit severe catastrophic forgetting during continual evolution. For example, Meta-Harness improves SWE-Bench to 94.0% but drops to 32.0% after further evolution on FeatureBench (a 66% relative decrease).
- **Root Cause Analysis**: Through manual analysis of 520+ execution trajectories, the authors classify harness-level conflicts into three major types: **verification conflicts** (87.1% of pass-to-fail cases), **workflow conflicts** (8.2%), and **budget strategy conflicts** (0.6%).
- **Impact Quantification**: Ablation experiments show that artifacts/verification conflicts can completely destroy prior performance (100% drop on SWE-Bench), workflow conflicts cause ~31% degradation, while budget strategies primarily affect execution time rather than task success.
- **Key Insight**: Task relatedness matters—evolution between more similar tasks (SWE-Bench ↔ FeatureBench) causes less forgetting than between dissimilar task types (SWT-Bench vs. other datasets).

---

## Introduction and Theoretical Foundation

### Background

Coding agents are increasingly deployed across diverse software engineering tasks, including issue solving, feature addition, and test generation. A coding agent is formally composed of two parts:

$$A = \langle M, H \rangle$$

where $M$ denotes the foundation model (fixed parameters) and $H$ denotes the **agent harness**—the runtime system that coordinates interaction between the model, tools, and environment. The harness influences how the agent perceives task information, invokes tools, and executes tasks, and can substantially affect performance (e.g., different harnesses on SWE-Bench range from 82.0% to 96.0% with the same foundation model).

### Harness Self-Evolution

Harness self-evolution enables agents to improve their harness by learning from their own execution experience. Given execution trajectories $\tau$ collected from a dataset $\mathcal{D}$, a self-evolution method $\Omega$ updates the harness as:

$$H' = \Omega(H, \tau)$$

Existing methods (AHE, Meta-Harness) demonstrate significant improvements on single tasks—e.g., AHE improves Terminal-Bench 2 from 69.7% to 77.0% over ten rounds.

### Continual Harness Self-Evolution

The paper motivates the **continual setting**: in practice, developers use agents for multiple task types sequentially. Given an ordered sequence of datasets $\mathcal{D} = (D_0, D_1, \ldots, D_K)$, the harness is updated repeatedly:

$$H_{i+1} = \Omega(H_i, \tau_i) \tag{1}$$

where $\tau_i$ denotes trajectories from tasks in $D_i$.

This setting relates to the **stability–plasticity dilemma** in continual learning. The ideal requirements are:

$$\text{Eval}(H_{i+1}, D_i) > \text{Eval}(H_i, D_i) \tag{2} \quad \text{(adaptation)}$$

$$\text{Eval}(H_{i+1}, D_j) \geq \text{Eval}(H_{j+1}, D_j) \tag{3} \quad \text{(retention)}$$

**Catastrophic forgetting** is quantified via absolute and relative performance drops:

$$\Delta_{\text{forget}} = \text{Eval}(H_{j+1}, D_j) - \text{Eval}(H_{i+1}, D_j) \tag{4}$$

$$\rho_{\text{forget}} = \frac{\text{Eval}(H_{j+1}, D_j) - \text{Eval}(H_{i+1}, D_j)}{\text{Eval}(H_{j+1}, D_j)} \tag{5}$$

---

## Methodology

### Datasets

Three software engineering benchmarks covering distinct task types:

| Dataset | Task Type | Description |
|---------|-----------|-------------|
| **SWE-Bench Verified** | Code repair | 500 instances from real GitHub issues; modify repository to fix bugs |
| **SWT-Bench** | Test generation | Generate tests that fail on buggy code but pass after the fix |
| **FeatureBench** | Feature generation | Implement new features; evaluated with F2P (fail-to-pass) and P2P (pass-to-pass) tests |

Due to resource constraints, subsets were sampled: 50 instances from SWE-Bench, 50 from SWT-Bench, and 30 from FeatureBench.

### Initial Harness

The harness is initialized with **Mini-SWE-Agent**, a lightweight agent framework (~100 lines of Python) achieving >74% on SWE-Bench Verified. Evolution is restricted to:
- Modifying the system prompt
- Modifying agent execution logic
- Adding or modifying tools

### Evolution Methods Evaluated

1. **AHE (Agent Harness Evolution)**: Constructs a closed-loop process where an Agent Debugger analyzes execution trajectories (successes/failures) and an Evolve Agent modifies the harness code. Performs 9 rounds of evolution.

2. **Meta-Harness**: Maintains a Pareto frontier of harnesses, proposing new modifications by referencing existing harnesses and accumulated experience. Performs 5 rounds of evolution.

### Experimental Design

- **Foundation model**: DeepSeek-V4-Flash-0731 (fixed for both execution and evolution)
- **Sandbox**: E2B; time limit of 1 hour per task
- **Rollout temperature**: 0
- **Continual evolution setup**: Each experiment forms a two-stage sequence (source → target), with 6 total evolution directions across the 3 datasets
- The best-performing harness from single-dataset evolution is selected as the initial harness for continual evolution

---

## Empirical Validation / Results

### RQ1: Catastrophic Forgetting in Continual Harness Self-Evolution

**Single-dataset evolution works**: Both methods substantially improve performance on individual tasks (Figure 2):
- AHE improves SWE-Bench from 82.0% → 96.0%
- Both methods reach 100.0% on SWT-Bench (50 sampled tasks)
- Both achieve >10 percentage point improvements on FeatureBench

**Catastrophic forgetting is pervasive**: As shown in Figure 3, almost every continual evolution direction causes substantial degradation on previous tasks:

| Evolution Direction | Method | Performance Drop |
|---------------------|--------|-----------------|
| SWE → SWT (5 rounds) | AHE | 100% relative drop on SWE-Bench |
| SWE → SWT (5 rounds) | Meta-Harness | 98% relative drop on SWE-Bench |
| SWE → Feature | AHE | 29% relative drop |
| SWE → Feature | Meta-Harness | 66% relative drop |

**Task relatedness matters**: Evolution between SWE-Bench and FeatureBench (both require source-code modification) causes relatively smaller degradation, whereas evolution between SWT-Bench (test generation) and the other datasets causes substantially larger forgetting.

### RQ2: Root Causes of Catastrophic Forgetting

Manual analysis of 520 execution trajectories with pass-to-fail transitions revealed **three major harness conflict types**:

#### Conflict Distribution (among pass-to-fail cases)

| Conflict Type | Proportion |
|---------------|------------|
| **Verification conflicts** | 87.1% |
| **Workflow conflicts** | 8.2% |
| **Budget strategy** | 0.6% |

1. **Verification conflicts** (most prominent: **artifact conflicts**): The harness introduces checks/constraints tailored to the current dataset (e.g., requiring test-only patches or rejecting source-code modifications), which conflicts with previous task requirements. This can effectively reverse the submission logic.

2. **Workflow conflicts**: Differences in problem-solving procedures between task types. The harness may enforce a task-specific workflow (e.g., write test → verify fail-to-pass → revert source changes) that interferes with workflows appropriate for previous tasks.

3. **Budget strategy conflicts**: Modifications to how the harness manages time/computation (e.g., timeout-aware convergence strategies that prompt agents to stop retrying commands).

#### Impact of Individual Conflict Types (Table 1)

*Impact of Individual Harness Conflict Types on SWE-Bench Performance and Agent Execution Time*

| Harness Configuration | SWE Performance (%) | Agent Time (s) |
|----------------------|---------------------|----------------|
| Original Best Harness | 96.0 | 245.5 |
| + Verification Conflict | **0.0 (↓ 100%)** | 415.7 (↑ 69%) |
| + Workflow Conflict | 66.0 (↓ 31%) | 821.5 (↑ 235%) |
| + Budget Strategy | 96.0 (↓ 0%) | 439.7 (↑ 79%) |

Key findings:
- **Verification/artifact conflicts** are most destructive: enforcing test-only submissions on SWE-Bench (which requires source fixes) reduces performance to 0.0%.
- **Workflow conflicts** cause substantial degradation (31%) and dramatically increase execution time (235%) due to unnecessary exploratory operations (e.g., blocking `pip`, `curl`, `git` commands).
- **Budget strategies** have negligible impact on performance but alter efficiency (79% execution time increase).

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **New problem formulation**: The paper formalizes continual harness self-evolution and its ideal requirements (adaptation + retention conditions), connecting the harness evolution literature to continual learning's stability–plasticity dilemma.

2. **Empirical characterization of forgetting**: Demonstrates that catastrophic forgetting is not unique to parameter-based learning but also afflicts the *structural/harness-level* component of agents, providing evidence that forgetting can occur even when foundation model parameters remain fixed.

3. **Taxonomy of conflicts**: Introduces a systematic categorization of harness-level conflicts (verification, workflow, budget), providing an analytical framework for understanding *why* harness evolution causes forgetting—a contribution that can guide future method design.

### Practical Implications

- **Deployment risk**: Agents evolved on a single task type may be dramatically degraded on other tasks. A user deploying an agent evolved on test generation may find it completely unable to perform code repair (100% degradation in some settings).
- **Need for conflict-aware evolution**: Future self-evolution methods should explicitly track and preserve task-specific constraints, workflows, and strategies across evolution steps.
- **Task scheduling considerations**: The finding that task relatedness affects forgetting severity suggests that practitioners should consider the sequence of task types when planning continual evolution.

---

## Conclusion

This paper studies **continual harness self-evolution** for coding agents, identifying catastrophic forgetting as a central and previously overlooked challenge. Key contributions:

1. **Problem proposal**: Introduces continual harness self-evolution and formalizes the adaptation/retention requirements and forgetting metrics.

2. **Empirical evaluation**: Evaluates two representative harness evolution methods (AHE, Meta-Harness) across three benchmarks and six evolution directions, revealing severe catastrophic forgetting (up to 100% relative performance drops).

3. **Root cause analysis**: Manually analyzes 520+ trajectories and categorizes harness modifications into **verification conflicts** (87.1%), **workflow conflicts** (8.2%), and **budget strategy** (0.6%), with ablation experiments quantifying their differential impacts.

The authors conclude that evolving coding agents reliably across diverse software engineering tasks remains a significant challenge, and call for more attention to **conflict-aware continual self-evolution** methods. Future directions include designing evolution algorithms that explicitly preserve prior capabilities while adapting to new task types, potentially by maintaining task-specific harness components or identifying and protecting task-invariant strategies.

---

_Markdown view of https://picx.dev/p/WJRKRV, served by PicX — AI-generated visual whiteboard summaries of research papers._