# How Do Coding-Agent Harnesses Self-Evolve? An Empirical Study of Retrospective Adaptation

> Retrospective harness self-evolution rarely reduces repeated work; only adaptations covering encountered instances succeed, while incomplete ones can add work.

- **Source:** [arXiv](https://arxiv.org/abs/2610.34938)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/YfKPC0
- **Whiteboard:** https://picx.dev/p/YfKPC0/image

## Summary

## Summary (Overview)

- This paper presents an empirical study of **/retro**, a skill that asks coding agents to review completed sessions and propose harness changes (software, configuration, and instructions used to run the agent), examining whether these self-evolution mechanisms actually improve behavior on later tasks.
- The study ran **Claude Code on 30 SWE-bench Verified issues**, collected **125 proposals** from /retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories.
- **Key finding (RQ1):** Most proposals (101 of 125) were experience-driven (based on session events), but 18 of 43 lint-related proposals recommended practices regardless of what happened in the session, revealing that /retro encodes normative practices beyond observed problems.
- **Key finding (RQ2):** Among 7 informative transfer events, only **2 adaptations eliminated the targeted action**; 1 reduced it, and 4 showed no reduction. Some changes even **added work** on unrelated issues (e.g., baseline comparisons increased from 0/5 to 4/5 sessions on a control task).
- The central conclusion: **harness self-evolution requires more than generating and implementing changes**—it needs mechanisms to establish whether changes help on later tasks and to revise or remove ineffective ones.

---

## Introduction and Theoretical Foundation

Coding agents such as Claude Code run inside a **harness**: the software, configuration, and instructions used to run the agent, including context files like `CLAUDE.md` that the agent reads at the start of each session. The concept of **harness self-evolution** refers to retaining changes from completed sessions for use on later tasks, without retraining the model.

The /retro skill is one mechanism for producing such changes. It instructs a coding agent to conduct a **retrospective**—a review of a completed session that generates numbered proposals for harness changes requiring human approval.

The paper operationalizes **repeated work** as a later session performing a specific action that an adaptation was intended to prevent, such as:
- Running tests with `pytest` when it is not installed
- Rerunning tests on unmodified code to check whether an unrelated failure was already present

This is a **behavioral measure**, not a judgment that the action was unnecessary or that preventing it improved the task outcome.

### Research Questions
- **RQ1:** What harness changes does /retro propose, and to what extent are they based on events in the session?
- **RQ2:** Under what conditions do implemented adaptations prevent targeted actions on later issues from the same repository?

### Related Work
| Work | Focus |
|------|-------|
| Galster et al. | Found context files in 91% of 2,853 engineered GitHub projects with configuration artifacts |
| Lulla et al. | Median wall-clock time fell from 99 to 70 seconds with `AGENTS.md` files (124 PRs, 10 repos) |
| Zhang et al. | Same pass rate for random community rules vs. curated rules (58 SWE-bench tasks) |
| Agentic Harness Engineering | Uses session traces to propose harness edits, checks predicted effects in later rounds |
| Wu et al. | Traced harness changes into later attempts at the same task (multiple changes bundled together) |
| Lin et al. | Compared task success with original vs. updated harnesses |
| BenchTrace | All six memory-update methods achieved **less** task progress than baseline |

This study complements prior work by evaluating **individual adaptations, one at a time**, on other issues from the same repository.

---

## Methodology

### Study Design Overview
1. **30 coding-agent sessions** were run, collecting 125 proposals from /retro
2. Proposals were **coded qualitatively** using a fixed codebook
3. **24 proposals were implemented**, yielding 12 evaluated adaptations (11 experience-driven + 1 author-written prior-driven comparison)
4. Adaptations were **transferred** to other issues from the same repositories

### Tasks and Sessions
- **Exploratory sample:** 10 issues (2 each from Django, SymPy, Matplotlib, Astropy, Sphinx), randomly sampled from SWE-bench Verified's "15 min–1 hour" category
- **Second sample:** 20 issues from Astropy, pytest, Sphinx, SymPy (non-random, chosen for environment problems)
- Each issue ran in a **fresh Claude Code session** (Claude Code 2.1.295 with Opus 5.5) in a Docker container
- After task completion, the session was resumed with /retro, and output was saved unchanged

### Proposal Coding
Each numbered entry counted as one proposal. Two codebooks were used:

**Trigger codes** (relationship between rationale and session events):
- **Experience-driven:** based on an event observed in the session trace
- **Prior-driven:** justified without relying on session events (e.g., adding a lint hook solely because the repository lacked one)
- **Weak:** indirect connection to the session

**Mechanism codes** (type of change recommended):

| Mechanism | Count | Sessions | Example |
|-----------|-------|----------|---------|
| Executable check | 38 | 29 | Pre-commit hook running flake8 on staged files |
| Navigation pointer | 24 | 22 | CLAUDE.md line naming bin/test as test runner |
| Instruction | 21 | 16 | Coding-standards rule requiring a failing test before fix |
| Information access | 20 | 18 | Known-failures file listing tests failing under installed docutils |
| Tool or configuration | 12 | 10 | Script printing one line per failing test |
| None | 10 | 10 | Use Edit tool instead of Python replace script |

Two LLM coders (Claude Code with `claude-fable-5-1` and GPT with `gpt-6.1-sol`) coded independently. Agreement: trigger code κ = 0.67, mechanism code κ = 0.84 (109/125 for both).

### Eligibility and Implementation
A proposal was eligible if it was:
- Coded **experience-driven**
- Proposed a **persistent harness change** in permitted files/settings
- **Safe to apply**
- Did not merely fix the original issue
- Admitted a **written criterion** for identifying repeated work in later sessions

### Transfer Events
For each adaptation, another SWE-bench Verified issue from the same repository was chosen, and its environment was verified to reproduce the targeted problem. Each transfer event ran:
- **5 sessions under the original harness**
- **5 sessions under the adapted harness**

For each session, the researchers recorded:
- **Recurrence:** whether Claude Code performed the action the adaptation was intended to prevent
- **Coverage:** whether the adaptation addressed the particular instances encountered
- **Loaded:** whether the adaptation entered Claude Code's context
- **Consulted:** whether Claude Code read a referenced file or ran a recommended command

A transfer event was **informative** when at least 3 of 5 original-harness sessions encountered the problem.

---

## Empirical Validation / Results

### RQ1: What does /retro propose?

- **101 of 125** proposals were coded experience-driven, 19 prior-driven, 5 weak
- **17 of 19 prior-driven proposals were executable checks** (lint-related)
- Only **9 of 38 executable checks** could report failure in the task environment as written; 28 needed missing tools or were commit hooks that never fired (sessions made no commits)
- **43 lint-related proposals** appeared across 29 of 30 sessions; 23 experience-driven, 18 prior-driven, 2 weak
- Environment problems recurred: 5/10 exploratory and 19/20 second-sample sessions contained failed shell commands, repeated commands, or multiple test runners—mostly due to pre-existing test failures from incompatible docutils/Python versions, outdated leap-second data, or missing lint tools

> **Answer to RQ1:** Executable checks were the most common change (38 of 125), and 101 proposals were based on events in the session, but 18 of the 19 prior-driven proposals recommended lint checks regardless of what had happened.

### RQ2: Do adaptations reduce repeated work?

Of 13 transfer events, **7 were informative**. Results are shown in Table II:

| Repository | Adaptation | Original | Adapted | Consulted |
|------------|-----------|----------|---------|-----------|
| **Adaptation covered encountered instances** | | | | |
| SymPy | CLAUDE.md line naming bin/test | 5/5 | **0/5** | 5/5 |
| Astropy | Known-failures file: 3 tests | 3/5 | **0/5** | 5/5 |
| — control task | same file, unrelated issue | 0/5 | 0/5 | 0/5 |
| **Adaptation did not cover encountered instances** | | | | |
| Sphinx | Known-failures file + AGENTS.md | 5/5 | 5/5 | 5/5 |
| — control task | same file, unrelated issue | 0/5 | 1/5 | 5/5 |
| Sphinx | Known-failures file + CLAUDE.md | 5/5 | 5/5 | 5/5 |
| — control task | same file, unrelated issue | 0/5 | 4/5 | 5/5 |
| SymPy | 2-package known-failures file + script | 4/5 | 4/4 | 4/4 |
| pytest | Known-failures file: 3 tests | 5/5 | 5/5 | 5/5 |
| Astropy | Cache repair plugin + script | 5/5 | 2/5 | 5/5 |

### Key Findings from Transfer Events

**Two adaptations eliminated repeated work:**
1. **SymPy CLAUDE.md line** naming `bin/test` prevented the initial attempt to run unavailable `pytest` in every adapted session (5/5 → 0/5)
2. **Astropy known-failures file** listing 3 failing tests (stale leap-second data + NumPy deprecation) reduced baseline comparisons from 3/5 to 0/5

**Four events showed no decrease:**
- Sessions encountered failures **absent from** the transferred known-failures files
- The adaptations were consulted but did not cover the instances encountered

**One adaptation reduced but did not eliminate recurrence:**
- Astropy cache repair plugin + script: recurrence fell from 5/5 to 2/5 (but no session encountered the leap-second failure; NumPy failures outside the script's stored list were reported as new)

**Control tasks showed adaptations can add work:**
- One Sphinx known-failures file told Claude Code to treat any unlisted failure as caused by its own edits. On an unrelated control issue, **4/5 adapted sessions ran baseline comparisons vs. 0/5 original-harness sessions**

**Secondary outcomes:**
- Task success differed by at most one session between harnesses in every event
- Lower recurrence did not consistently mean shorter sessions: the SymPy CLAUDE.md file removed the failed pytest call but bundled a CI lint-check instruction, raising median elapsed time from 88 to 156 seconds

> **Answer to RQ2:** The two adaptations that covered the instances later sessions encountered eliminated the targeted action; of the five that did not, one reduced it and four left it unchanged, one of which added work on an unrelated issue.

---

## Theoretical and Practical Implications

### Theoretical Implications
1. **Scope of retained experience matters:** A repository's test command can remain useful across issues (stable repository-level knowledge), whereas a list of specific tests that failed in one environment may not cover those encountered later (instances are environment- and version-specific).
2. **Consultation ≠ compliance:** The study found that adaptations were loaded and consulted in nearly all adapted sessions, yet recurrence often did not decrease. Reading a file does not establish that the agent acts on its contents.
3. **Retrospectives encode norms, not just events:** The /retro skill's instructions treat a repository without lint hooks as "a finding," explaining why 18 of 19 prior-driven proposals recommended lint checks regardless of session events.
4. **Self-evolution is not autonomous improvement:** The process studied was "self-evolution with human selection and evaluation," and even then, most adaptations failed to reduce repeated work.

### Practical Implications
1. **Evaluation is essential:** An evaluation of an adaptation needs to establish whether the targeted problem arose, whether the agent consulted the adaptation, and whether its actions changed.
2. **Continuous monitoring required:** Retained changes need continued evaluation because outdated or incomplete instructions can remain ineffective or add work.
3. **Incomplete failure lists are dangerous:** An incomplete known-failures file can actively cause the verification work it was meant to prevent (e.g., the Sphinx control task where baseline comparisons rose from 0/5 to 4/5).
4. **Bundled instructions can have hidden costs:** Adding instructions alongside useful adaptations (e.g., the CI lint check in the SymPy CLAUDE.md) can increase session time even when the primary adaptation works.

---

## Conclusion

### Main Takeaways
- Retrospective harness adaptation must account for the **scope of retained experience**: stable repository knowledge (test commands) transfers well, while specific failure lists may not cover later instances.
- Implementing an adaptation does **not guarantee** it reduces repeated work—coverage of encountered instances is the critical factor.
- Harness self-evolution requires mechanisms to **establish whether a change helps** on later tasks and to **revise or remove changes that do not**.

### Limitations
- Evidence covers one agent (Claude Code), one version of /retro, and environment problems only
- Selection of evaluation issues that reproduced the targeted problem may overestimate how often adaptations would help on unselected tasks
- The first author's labeling was not fully blind (traces could reveal transferred files)
- With five sessions per harness and seven informative events, counts are descriptive, not statistically established

### Future Work
- Evaluate other agents and adaptation types (code navigation, coding conventions)
- Evaluate bundled adaptation components one at a time
- Test on later issues **not** selected for the targeted problem
- Study how to automatically detect ineffective or harmful adaptations for revision or removal

### Data Availability
An artifact with all scripts, prompts, session traces, codebook, labels, and harness changes is available at: https://doi.org/10.5281/zenodo.23653700

---

_Markdown view of https://picx.dev/p/YfKPC0, served by PicX — AI-generated visual whiteboard summaries of research papers._