Summary

• Problem Definition: This paper introduces continual harness self-evolution—the successive evolution of coding agent harnesses across different task types (e.g., code repair, test generation, feature generation)—and identifies catastrophic forgetting as a critical, previously unexplored challenge in this setting.
• Empirical Finding: Both representative harness evolution methods (AHE and Meta-Harness) exhibit severe catastrophic forgetting during continual evolution. For example, Meta-Harness improves SWE-Bench to 94.0% but drops to 32.0% after further evolution on FeatureBench (a 66% relative decrease).
• Root Cause Analysis: Through manual analysis of 520+ execution trajectories, the authors classify harness-level conflicts into three major types: verification conflicts (87.1% of pass-to-fail cases), workflow conflicts (8.2%), and budget strategy conflicts (0.6%).
• Impact Quantification: Ablation experiments show that artifacts/verification conflicts can completely destroy prior performance (100% drop on SWE-Bench), workflow conflicts cause ~31% degradation, while budget strategies primarily affect execution time rather than task success.
• Key Insight: Task relatedness matters—evolution between more similar tasks (SWE-Bench ↔ FeatureBench) causes less forgetting than between dissimilar task types (SWT-Bench vs. other datasets).

---

Introduction and Theoretical Foundation

Background

Coding agents are increasingly deployed across diverse software engineering tasks, including issue solving, feature addition, and test generation. A coding agent is formally composed of two parts:

$$A = \langle M, H \rangle$$

where $M$ denotes the foundation model (fixed parameters) and $H$ denotes the agent harness—the runtime system that coordinates interaction between the model, tools, and environment. The harness influences how the agent perceives task information, invokes tools, and executes tasks, and can substantially affect performance (e.g., different harnesses on SWE-Bench range from 82.0% to 96.0% with the same foundation model).

Harness Self-Evolution

Harness self-evolution enables agents to improve their harness by learning from their own execution experience. Given execution trajectories $\tau$ collected from a dataset $\mathcal{D}$, a self-evolution method $\Omega$ updates the harness as:

$$H' = \Omega(H, \tau)$$

Existing methods (AHE, Meta-Harness) demonstrate significant improvements on single tasks—e.g., AHE improves Terminal-Bench 2 from 69.7% to 77.0% over ten rounds.

Continual Harness Self-Evolution

The paper motivates the continual setting: in practice, developers use agents for multiple task types sequentially. Given an ordered sequence of datasets $\mathcal{D} = (D_0, D_1, \ldots, D_K)$, the harness is updated repeatedly:

$$H_{i+1} = \Omega(H_i, \tau_i) \tag{1}$$

where $\tau_i$ denotes trajectories from tasks in $D_i$.

This setting relates to the stability–plasticity dilemma in continual learning. The ideal requirements are:

$$\text{Eval}(H_{i+1}, D_i) > \text{Eval}(H_i, D_i) \tag{2} \quad \text{(adaptation)}$$ $$\text{Eval}(H_{i+1}, D_j) \geq \text{Eval}(H_{j+1}, D_j) \tag{3} \quad \text{(retention)}$$

Catastrophic forgetting is quantified via absolute and relative performance drops:

$$\Delta_{\text{forget}} = \text{Eval}(H_{j+1}, D_j) - \text{Eval}(H_{i+1}, D_j) \tag{4}$$ $$\rho_{\text{forget}} = \frac{\text{Eval}(H_{j+1}, D_j) - \text{Eval}(H_{i+1}, D_j)}{\text{Eval}(H_{j+1}, D_j)} \tag{5}$$

---

Methodology

Datasets

Three software engineering benchmarks covering distinct task types:

Due to resource constraints, subsets were sampled: 50 instances from SWE-Bench, 50 from SWT-Bench, and 30 from FeatureBench.

Initial Harness

The harness is initialized with Mini-SWE-Agent, a lightweight agent framework (~100 lines of Python) achieving >74% on SWE-Bench Verified. Evolution is restricted to:

• Modifying the system prompt
• Modifying agent execution logic
• Adding or modifying tools

Evolution Methods Evaluated

1. AHE (Agent Harness Evolution): Constructs a closed-loop process where an Agent Debugger analyzes execution trajectories (successes/failures) and an Evolve Agent modifies the harness code. Performs 9 rounds of evolution.

2. Meta-Harness: Maintains a Pareto frontier of harnesses, proposing new modifications by referencing existing harnesses and accumulated experience. Performs 5 rounds of evolution.

Experimental Design

• Foundation model: DeepSeek-V4-Flash-0731 (fixed for both execution and evolution)
• Sandbox: E2B; time limit of 1 hour per task
• Rollout temperature: 0
• Continual evolution setup: Each experiment forms a two-stage sequence (source → target), with 6 total evolution directions across the 3 datasets
• The best-performing harness from single-dataset evolution is selected as the initial harness for continual evolution

---

Empirical Validation / Results

RQ1: Catastrophic Forgetting in Continual Harness Self-Evolution

Single-dataset evolution works: Both methods substantially improve performance on individual tasks (Figure 2):

• AHE improves SWE-Bench from 82.0% → 96.0%
• Both methods reach 100.0% on SWT-Bench (50 sampled tasks)
• Both achieve >10 percentage point improvements on FeatureBench

Catastrophic forgetting is pervasive: As shown in Figure 3, almost every continual evolution direction causes substantial degradation on previous tasks:

Task relatedness matters: Evolution between SWE-Bench and FeatureBench (both require source-code modification) causes relatively smaller degradation, whereas evolution between SWT-Bench (test generation) and the other datasets causes substantially larger forgetting.

RQ2: Root Causes of Catastrophic Forgetting

Manual analysis of 520 execution trajectories with pass-to-fail transitions revealed three major harness conflict types:

Conflict Distribution (among pass-to-fail cases)

1. Verification conflicts (most prominent: artifact conflicts): The harness introduces checks/constraints tailored to the current dataset (e.g., requiring test-only patches or rejecting source-code modifications), which conflicts with previous task requirements. This can effectively reverse the submission logic.

2. Workflow conflicts: Differences in problem-solving procedures between task types. The harness may enforce a task-specific workflow (e.g., write test → verify fail-to-pass → revert source changes) that interferes with workflows appropriate for previous tasks.

3. Budget strategy conflicts: Modifications to how the harness manages time/computation (e.g., timeout-aware convergence strategies that prompt agents to stop retrying commands).

Impact of Individual Conflict Types (Table 1)

Impact of Individual Harness Conflict Types on SWE-Bench Performance and Agent Execution Time

Key findings:

• Verification/artifact conflicts are most destructive: enforcing test-only submissions on SWE-Bench (which requires source fixes) reduces performance to 0.0%.
• Workflow conflicts cause substantial degradation (31%) and dramatically increase execution time (235%) due to unnecessary exploratory operations (e.g., blocking pip, curl, git commands).
• Budget strategies have negligible impact on performance but alter efficiency (79% execution time increase).

---

Theoretical and Practical Implications

Theoretical Contributions

1. New problem formulation: The paper formalizes continual harness self-evolution and its ideal requirements (adaptation + retention conditions), connecting the harness evolution literature to continual learning's stability–plasticity dilemma.

2. Empirical characterization of forgetting: Demonstrates that catastrophic forgetting is not unique to parameter-based learning but also afflicts the structural/harness-level component of agents, providing evidence that forgetting can occur even when foundation model parameters remain fixed.

3. Taxonomy of conflicts: Introduces a systematic categorization of harness-level conflicts (verification, workflow, budget), providing an analytical framework for understanding why harness evolution causes forgetting—a contribution that can guide future method design.

Practical Implications

• Deployment risk: Agents evolved on a single task type may be dramatically degraded on other tasks. A user deploying an agent evolved on test generation may find it completely unable to perform code repair (100% degradation in some settings).
• Need for conflict-aware evolution: Future self-evolution methods should explicitly track and preserve task-specific constraints, workflows, and strategies across evolution steps.
• Task scheduling considerations: The finding that task relatedness affects forgetting severity suggests that practitioners should consider the sequence of task types when planning continual evolution.

---

Conclusion

This paper studies continual harness self-evolution for coding agents, identifying catastrophic forgetting as a central and previously overlooked challenge. Key contributions:

1. Problem proposal: Introduces continual harness self-evolution and formalizes the adaptation/retention requirements and forgetting metrics.

2. Empirical evaluation: Evaluates two representative harness evolution methods (AHE, Meta-Harness) across three benchmarks and six evolution directions, revealing severe catastrophic forgetting (up to 100% relative performance drops).

3. Root cause analysis: Manually analyzes 520+ trajectories and categorizes harness modifications into verification conflicts (87.1%), workflow conflicts (8.2%), and budget strategy (0.6%), with ablation experiments quantifying their differential impacts.

The authors conclude that evolving coding agents reliably across diverse software engineering tasks remains a significant challenge, and call for more attention to conflict-aware continual self-evolution methods. Future directions include designing evolution algorithms that explicitly preserve prior capabilities while adapting to new task types, potentially by maintaining task-specific harness components or identifying and protecting task-invariant strategies.