RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

Summary (Overview)

• Problem Identified: Recursive self-improvement (RSI) of LLM agent harnesses—the prompts, control flow, tooling, memory, and context management surrounding a frozen backbone model—suffers from adaptive overfitting: evolve-set performance improves without corresponding gains on unseen tasks.
• Key Contribution: The paper introduces RRSI (Regularized Recursive Self-Improvement), a framework that regularizes both the proposal and selection sides of harness evolution while keeping the full harness edit space open.
• Results: Across eight benchmarks in three domains (coding, agentic workspace, engineering design), RRSI gains up to 14.1 points on the evolve split and up to 4.7 points on out-of-distribution benchmarks, while using 30% fewer policy tokens than unregularized evolution.
• Generalization: RRSI outperforms four prior harness evolution baselines (Meta-Harness, AHE, TTHE, HarnessX) on all held-out datasets, with the only out-of-distribution average that clears the base harness by more than a point (43.6 vs. 39.7).
• Robustness: Gains transfer across different policy families (Claude Opus 4.8 and Gemini 3.5 Flash) and even to weaker backbones never used in the search (30.4% relative gain on Terminal-Bench with Gemini 3.1 Flash Lite).

Introduction and Theoretical Foundation

Background: Agents as Systems

Modern LLM agents are systems rather than standalone models. A frozen backbone policy $\pi$ is wrapped in a harness $H$ comprising:

• System and task prompts
• Control flow (when to plan, act, reflect, or stop)
• Tool interfaces and descriptions
• Memory and skill files
• Context management

The harness determines whether the same model reads the right file before editing it, recovers from failed commands, manages working context efficiently, and writes findings into deliverables. Recent progress in agent products has largely come from harness engineering rather than new model weights—but this engineering is manual and limited by how many trajectories an engineer can read.

The Overfitting Problem in Harness Evolution

Recent methods automate harness optimization using LLMs to propose and select component-wise edits from task feedback. This creates a practical form of recursive self-improvement (RSI) at the agent-system level. However, as shown in Figure 1(a), this creates an adaptive overfitting risk:

"Evolve-set performance may improve without corresponding gains on unseen tasks."

The paper identifies three coupled behaviors that cause overfitting:

1. Benchmark-specific fitting: Encoding task-specific patterns into the harness
2. Noise chasing: Promoting candidates favored by evaluation noise
3. Complexity accumulation: Adding complexity that improves evolve-set scores without improving the underlying mechanism

Theoretical Formulation

An agent $A = (\pi, H)$ produces trajectories $\tau \sim A(\cdot \mid x)$ scored by a verifier $r(x, \tau) \in [0, 1]$. Task performance and policy-token cost are:

$$S(H; \mathcal{D}) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{\tau \sim A(\cdot | x)} [r(x, \tau)], \quad C(H; \mathcal{D}) = \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{\tau \sim A(\cdot | x)} [c(\tau)], \tag{1}$$

Harness evolution follows a generic loop at round $t$:

$$\mathcal{H}_t = \{H_t^{(1)}, \ldots, H_t^{(m_t)}\} \sim P_0(\cdot \mid H_t, \mathcal{F}_t), \qquad H_{t+1} = \arg\max_{H' \in \mathcal{H}_t \cup \{H_t\}} \hat{S}(H'; \mathcal{D}_{\text{evolve}}), \tag{2}$$

The key insight is that this reuse of $\mathcal{D}_{\text{evolve}}$ is adaptive: candidates proposed at round $t$ depend on measurements from the same tasks in earlier rounds. Harness evolution is thus "adaptive empirical optimization over an unusually expressive search space."

Methodology

RRSI keeps the harness edit space fully open but regularizes the search trajectory through it. The framework draws analogies to three classical regularization approaches:

1. Regularizing the Proposal Distribution

$L_0$-Style Annealed Update Sparsity: Caps the number of independently attributable edits per proposal. At round $t$ of a $T$-round run:

$$b_t = \left[ b_{\min} + (b_{\max} - b_{\min}) \cdot \frac{1}{2}(1 + \cos(\pi t / T)) \right]. \tag{4}$$

Early rounds may combine several coordinated changes; later rounds become increasingly sparse and attributable.

Evidence-Aware Credit Assignment: Records for every evaluated candidate its component, hypothesis, source diff, score/cost changes, and acceptance. Rejected mechanisms remain negative evidence; successful mechanisms retain explicit credit.

Structured Exploration: Detects when the search has stalled (progress over previous $w$ rounds within noise band $\delta$) and reserves proposal budget for unexplored components.

2. Regularizing Candidate Selection

Leakage Screening: A critic reads each candidate diff and rejects edits that explicitly encode task names, entity names, task-specific values, answers, or inert machinery—before full evaluation.

Stability-Aware Acceptance: A candidate must satisfy a noise-adjusted floor:

$$\hat{S}(H') \geq S^{\star} - \delta, \tag{5}$$

where $S^{\star}$ is the best evolve-set score observed so far and $\delta$ is an empirically estimated noise band.

Ridge/$L_2$-Style Complexity-Aware Acceptance: For a candidate with gain exceeding noise ($\Delta S > \delta$):

$$\Delta C \leq \beta_0 + \beta_1 \Delta S, \tag{7}$$

where $\Delta S = \hat{S}(H') - \hat{S}(H_t)$ and $\Delta C = \frac{\hat{C}(H') - \hat{C}(H_t)}{\hat{C}(H_t)}$. Additional inference cost must be justified by measurable performance improvement.

Lasso/$L_1$-Style Structural Pruning: Tracks whether recently exercised components produced strictly positive measured gain over a pruning window. Unproductive components become deletion targets.

Theoretical Analogies Summary

Empirical Validation / Results

Experimental Setup

• Coding: Terminal-Bench 2.1 (89 containerized terminal tasks); OOD: SWE-bench Verified
• Agentic workspace: Harvey LAB (legal work, 120 evolve + 40 held-out tasks); OOD: JobBench, GDPval, APEX-Agents
• Engineering design: EngDesign (61 tasks, deterministic simulators); OOD: Frontier-Eng
• Policy: Claude Opus 4.8 (frozen) across all domains; Gemini 3.5 Flash for robustness tests
• Baselines: Meta-Harness, AHE, TTHE, HarnessX (all from same $H_0$)

Main Results

Evolve-set gains: 6.0 points on Terminal-Bench 2.1, 4.9 on EngDesign, 1.1 on Harvey LAB.

Out-of-distribution transfer (Figure 3):

• SWE-bench Verified: +1.8 points (never scored during evolution)
• Harvey LAB ID held-out: +2.3 points
• JobBench: +3.5 to +4.7 points (7.2% to 13.1% relative)
• Frontier-Eng: +4.3 Medal points (24.3% relative improvement)
• No held-out split regressed anywhere

Comparison with Baselines (Table 1)

"RRSI posts the smallest evolve-set gain of any evolved harness and the only out-of-distribution average that clears $H_0$ by more than a point, 43.6 against 39.7, which is the trade the regularizers are designed to make."

Ablation Study (Table 2)

Removing both regularizer groups yields the highest evolve-set score (92.8) but an OOD average within a point of the unevolved harness—at 3.80 million tokens per trial vs. RRSI's 2.42.

Policy Robustness (Tables 3 & 4)

Different policy families (Table 3):

Cross-model transfer (Table 4)—harness evolved with Gemini 3.5 Flash evaluated with unseen Gemini 3.1 Flash Lite:

Efficiency

RRSI produces the lightest harness of any evolved harness:

• 2.42M tokens/trial vs. 3.80M for unregularized evolution (37% reduction)
• 26.3 steps/trial vs. 27.3–34.6 for prior methods
• AHE spends 58% more tokens for 4.4 points less OOD performance

Theoretical and Practical Implications

Theoretical Implications

1. Recursive self-improvement requires regularization: The paper demonstrates that RSI at the agent-system level suffers from the same overfitting risks as parametric learning, and that regularization principles (sparsity, shrinkage, pruning) translate naturally to discrete harness search.

2. The evolve-to-transfer gap is measurable and controllable: By separating evolve-set gains from held-out performance, the work provides a framework for evaluating whether "self-improvement" reflects genuine mechanism learning or benchmark-specific fitting.

3. Classical regularization analogies hold for discrete search: The mapping of $L_0$ constraints to edit budgets, Lasso to structural pruning, and Ridge to complexity-aware acceptance provides a principled vocabulary for designing regularized search algorithms.

Practical Implications

1. Automated harness engineering becomes viable: RRSI reduces the manual burden of harness design while producing harnesses that transfer across benchmarks, domains, and even policy models.

2. Cost-aware evolution: The complexity-aware acceptance criterion directly controls inference costs, making evolution practical for production systems where token budgets matter.

3. Cross-model generality: The finding that harnesses transfer across policy families (including to weaker, unseen backbones) suggests that harness improvements capture model-agnostic mechanisms rather than policy-specific artifacts.

4. Benchmark design: The results highlight that evolve-set performance alone is insufficient to validate harness evolution methods; held-out and OOD evaluation is essential.

Conclusion

The paper establishes that iterative harness evolution is a practical form of recursive self-improvement that itself requires regularization. Because a finite evolve set is reused adaptively across rounds, apparent self-improvement can reflect benchmark-specific fitting, evaluation noise, or unnecessary complexity rather than transferable progress.

RRSI addresses this by regularizing both proposal and selection while leaving the harness edit space open. Across coding, agentic workspace, and engineering design tasks, the resulting harnesses improve held-out and cross-benchmark performance while using less inference cost than unregularized evolution.

Key takeaway: "Making agent systems increasingly capable through recursive self-improvement requires controlling not only what can change, but also how repeated feedback is converted into persistent changes."

Limitations

• Focuses on harness-level RSI with frozen backbone models; does not address weight updates during evolution
• Relies on a finite evolve set and several regularization hyperparameters ($\delta$, $\beta_0$, $\beta_1$, $b_{\min}$, $b_{\max}$, pruning window)
• Broader validation needed for substantially different agent architectures, tool ecosystems, and longer-running self-improvement processes