Summary (Overview)

• This paper presents an empirical study of /retro, a skill that asks coding agents to review completed sessions and propose harness changes (software, configuration, and instructions used to run the agent), examining whether these self-evolution mechanisms actually improve behavior on later tasks.
• The study ran Claude Code on 30 SWE-bench Verified issues, collected 125 proposals from /retro, implemented 24, and evaluated 12 harness changes on later issues from the same repositories.
• Key finding (RQ1): Most proposals (101 of 125) were experience-driven (based on session events), but 18 of 43 lint-related proposals recommended practices regardless of what happened in the session, revealing that /retro encodes normative practices beyond observed problems.
• Key finding (RQ2): Among 7 informative transfer events, only 2 adaptations eliminated the targeted action; 1 reduced it, and 4 showed no reduction. Some changes even added work on unrelated issues (e.g., baseline comparisons increased from 0/5 to 4/5 sessions on a control task).
• The central conclusion: harness self-evolution requires more than generating and implementing changes—it needs mechanisms to establish whether changes help on later tasks and to revise or remove ineffective ones.

---

Introduction and Theoretical Foundation

Coding agents such as Claude Code run inside a harness: the software, configuration, and instructions used to run the agent, including context files like CLAUDE.md that the agent reads at the start of each session. The concept of harness self-evolution refers to retaining changes from completed sessions for use on later tasks, without retraining the model.

The /retro skill is one mechanism for producing such changes. It instructs a coding agent to conduct a retrospective—a review of a completed session that generates numbered proposals for harness changes requiring human approval.

The paper operationalizes repeated work as a later session performing a specific action that an adaptation was intended to prevent, such as:

• Running tests with pytest when it is not installed
• Rerunning tests on unmodified code to check whether an unrelated failure was already present

This is a behavioral measure, not a judgment that the action was unnecessary or that preventing it improved the task outcome.

Research Questions

• RQ1: What harness changes does /retro propose, and to what extent are they based on events in the session?
• RQ2: Under what conditions do implemented adaptations prevent targeted actions on later issues from the same repository?

Related Work

This study complements prior work by evaluating individual adaptations, one at a time, on other issues from the same repository.

---

Methodology

Study Design Overview

1. 30 coding-agent sessions were run, collecting 125 proposals from /retro
2. Proposals were coded qualitatively using a fixed codebook
3. 24 proposals were implemented, yielding 12 evaluated adaptations (11 experience-driven + 1 author-written prior-driven comparison)
4. Adaptations were transferred to other issues from the same repositories

Tasks and Sessions

• Exploratory sample: 10 issues (2 each from Django, SymPy, Matplotlib, Astropy, Sphinx), randomly sampled from SWE-bench Verified's "15 min–1 hour" category
• Second sample: 20 issues from Astropy, pytest, Sphinx, SymPy (non-random, chosen for environment problems)
• Each issue ran in a fresh Claude Code session (Claude Code 2.1.295 with Opus 5.5) in a Docker container
• After task completion, the session was resumed with /retro, and output was saved unchanged

Proposal Coding

Each numbered entry counted as one proposal. Two codebooks were used:

Trigger codes (relationship between rationale and session events):

• Experience-driven: based on an event observed in the session trace
• Prior-driven: justified without relying on session events (e.g., adding a lint hook solely because the repository lacked one)
• Weak: indirect connection to the session

Mechanism codes (type of change recommended):

Two LLM coders (Claude Code with claude-fable-5-1 and GPT with gpt-6.1-sol) coded independently. Agreement: trigger code κ = 0.67, mechanism code κ = 0.84 (109/125 for both).

Eligibility and Implementation

A proposal was eligible if it was:

• Coded experience-driven
• Proposed a persistent harness change in permitted files/settings
• Safe to apply
• Did not merely fix the original issue
• Admitted a written criterion for identifying repeated work in later sessions

Transfer Events

For each adaptation, another SWE-bench Verified issue from the same repository was chosen, and its environment was verified to reproduce the targeted problem. Each transfer event ran:

• 5 sessions under the original harness
• 5 sessions under the adapted harness

For each session, the researchers recorded:

• Recurrence: whether Claude Code performed the action the adaptation was intended to prevent
• Coverage: whether the adaptation addressed the particular instances encountered
• Loaded: whether the adaptation entered Claude Code's context
• Consulted: whether Claude Code read a referenced file or ran a recommended command

A transfer event was informative when at least 3 of 5 original-harness sessions encountered the problem.

---

Empirical Validation / Results

RQ1: What does /retro propose?

• 101 of 125 proposals were coded experience-driven, 19 prior-driven, 5 weak
• 17 of 19 prior-driven proposals were executable checks (lint-related)
• Only 9 of 38 executable checks could report failure in the task environment as written; 28 needed missing tools or were commit hooks that never fired (sessions made no commits)
• 43 lint-related proposals appeared across 29 of 30 sessions; 23 experience-driven, 18 prior-driven, 2 weak
• Environment problems recurred: 5/10 exploratory and 19/20 second-sample sessions contained failed shell commands, repeated commands, or multiple test runners—mostly due to pre-existing test failures from incompatible docutils/Python versions, outdated leap-second data, or missing lint tools

Answer to RQ1: Executable checks were the most common change (38 of 125), and 101 proposals were based on events in the session, but 18 of the 19 prior-driven proposals recommended lint checks regardless of what had happened.

RQ2: Do adaptations reduce repeated work?

Of 13 transfer events, 7 were informative. Results are shown in Table II:

Key Findings from Transfer Events

Two adaptations eliminated repeated work:

1. SymPy CLAUDE.md line naming bin/test prevented the initial attempt to run unavailable pytest in every adapted session (5/5 → 0/5)
2. Astropy known-failures file listing 3 failing tests (stale leap-second data + NumPy deprecation) reduced baseline comparisons from 3/5 to 0/5

Four events showed no decrease:

• Sessions encountered failures absent from the transferred known-failures files
• The adaptations were consulted but did not cover the instances encountered

One adaptation reduced but did not eliminate recurrence:

• Astropy cache repair plugin + script: recurrence fell from 5/5 to 2/5 (but no session encountered the leap-second failure; NumPy failures outside the script's stored list were reported as new)

Control tasks showed adaptations can add work:

• One Sphinx known-failures file told Claude Code to treat any unlisted failure as caused by its own edits. On an unrelated control issue, 4/5 adapted sessions ran baseline comparisons vs. 0/5 original-harness sessions

Secondary outcomes:

• Task success differed by at most one session between harnesses in every event
• Lower recurrence did not consistently mean shorter sessions: the SymPy CLAUDE.md file removed the failed pytest call but bundled a CI lint-check instruction, raising median elapsed time from 88 to 156 seconds

Answer to RQ2: The two adaptations that covered the instances later sessions encountered eliminated the targeted action; of the five that did not, one reduced it and four left it unchanged, one of which added work on an unrelated issue.

---

Theoretical and Practical Implications

Theoretical Implications

1. Scope of retained experience matters: A repository's test command can remain useful across issues (stable repository-level knowledge), whereas a list of specific tests that failed in one environment may not cover those encountered later (instances are environment- and version-specific).
2. Consultation ≠ compliance: The study found that adaptations were loaded and consulted in nearly all adapted sessions, yet recurrence often did not decrease. Reading a file does not establish that the agent acts on its contents.
3. Retrospectives encode norms, not just events: The /retro skill's instructions treat a repository without lint hooks as "a finding," explaining why 18 of 19 prior-driven proposals recommended lint checks regardless of session events.
4. Self-evolution is not autonomous improvement: The process studied was "self-evolution with human selection and evaluation," and even then, most adaptations failed to reduce repeated work.

Practical Implications

1. Evaluation is essential: An evaluation of an adaptation needs to establish whether the targeted problem arose, whether the agent consulted the adaptation, and whether its actions changed.
2. Continuous monitoring required: Retained changes need continued evaluation because outdated or incomplete instructions can remain ineffective or add work.
3. Incomplete failure lists are dangerous: An incomplete known-failures file can actively cause the verification work it was meant to prevent (e.g., the Sphinx control task where baseline comparisons rose from 0/5 to 4/5).
4. Bundled instructions can have hidden costs: Adding instructions alongside useful adaptations (e.g., the CI lint check in the SymPy CLAUDE.md) can increase session time even when the primary adaptation works.

---

Conclusion

Main Takeaways

• Retrospective harness adaptation must account for the scope of retained experience: stable repository knowledge (test commands) transfers well, while specific failure lists may not cover later instances.
• Implementing an adaptation does not guarantee it reduces repeated work—coverage of encountered instances is the critical factor.
• Harness self-evolution requires mechanisms to establish whether a change helps on later tasks and to revise or remove changes that do not.

Limitations

• Evidence covers one agent (Claude Code), one version of /retro, and environment problems only
• Selection of evaluation issues that reproduced the targeted problem may overestimate how often adaptations would help on unselected tasks
• The first author's labeling was not fully blind (traces could reveal transferred files)
• With five sessions per harness and seven informative events, counts are descriptive, not statistically established

Future Work

• Evaluate other agents and adaptation types (code navigation, coding conventions)
• Evaluate bundled adaptation components one at a time
• Test on later issues not selected for the targeted problem
• Study how to automatically detect ineffective or harmful adaptations for revision or removal

Data Availability

An artifact with all scripts, prompts, session traces, codebook, labels, and harness changes is available at: https://doi.org/10.5281/zenodo.23653700