SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

Summary (Overview)

• SoL-Pi is an RSI-inspired (Recursive Self-Improvement) system that automatically discovers token-efficient harness mechanisms for coding agents through a broad-to-deep auto-research funnel, yielding four reusable mechanisms that transfer across models and tasks.
• The four retained mechanisms—Action Fusion, Online Context Compact, ObservationPack, and Evidence-Preserving Reducer—target action execution, context compaction, observation handling, and delegated reading, respectively.
• On the 51-task EdgeBench benchmark, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by 44.7–49.0% and API cost by ~33% (about one third).
• Estimated hourly savings are $8.75–$13.50 relative to native Codex and Claude Code harnesses, and $4.36–$5.71 relative to Pi.
• The search process scaled to ~150 proposed directions, ~500 executable environments, >3,000 runs, and >60,000 agent–environment interactions, demonstrating that RSI at the harness layer can discover improvements that generalize beyond development settings.

---

Introduction and Theoretical Foundation

Background

As coding agents evolve from supervised code completion to unattended, around-the-clock exploration, their work expands from isolated predictions into long trajectories of reasoning, tool use, and feedback. This shift makes task-level token efficiency a first-order systems concern.

Existing efficiency work focuses on:

• Lowering cost per token via faster attention kernels and serving infrastructure
• Model compression (quantization)
• Using cheaper models

SoL-Pi explores an orthogonal direction: improving token use through the agent harness—the layer that mediates interactions between the model and its environment.

Motivation

Optimizing a harness is difficult because tool use, context management, verification, delegation, recovery, and termination are tightly coupled. A locally beneficial change may cause downstream failures or shift token costs later in execution. Traditional harness development requires substantial human effort inspecting long execution traces—a process that is costly and difficult to scale.

Theoretical Foundation

The work draws on:

• Recursive Self-Improvement (RSI) — agents improving their own infrastructure
• Gödel Machine — theoretical roots of self-modification
• Meta-Harness — searching over executable harness programs
• Recursive Harness Self-Improvement (RHI) — iterative refinement of agent loop specifications

Prior work (Wang et al.) found that evolved harnesses can overfit search tasks, motivating a clear separation between search feedback and final evaluation.

Three Guiding Principles

1. Breadth and depth — Breadth expands hypothesis coverage; depth repeatedly implements, reviews, and hardens promising candidates.
2. Independent validation — Held-out evidence is evaluated only after a candidate is frozen; validation failures never feed back into search.
3. Scalable orchestration — Isolated, disposable lineages let the funnel expand without coupling failures across candidates.

---

Methodology

2.1 Harness Auto-Research for Token Efficiency

The pipeline frames harness improvement as an RSI-inspired search:

1. Research agent analyzes execution trajectories from the base harness
2. Identifies recurring sources of overhead
3. Builds an idea pool of candidate harness changes
4. Tests effects in development environments

Acceptance criteria (fixed before experimentation):

• Capability gate: every capability metric must remain within predeclared tolerance
• Efficiency gate: candidate must improve at least one declared efficiency metric
• Among passing candidates, retain nondominated results under declared metrics

2.2 Broad-to-Deep Harness Search

Outer stage: Explores a broad pool of 152 proposed directions across six proposal families:

• Context, progress, tools, delegation, prompt/policy, improvement/evaluation

Inner stage: Develops selected hypotheses independently using an iterative implementation loop (Ralph Loop-based):

• Propose → Implement → Independent Review → Development Validation → Revise or Retain

Each independent search runs as a disposable instance of a shared skill template, with modified orchestration code discarded after completion.

2.3 Search Environments

EdgeBench (51 tasks) is reserved for held-out validation, completely isolated from search.

2.4 Discovered Harness Mechanisms

(a) Action Fusion

Combines file mutation with follow-up command (test/build/run) into one tool request, eliminating an intermediate model round trip:

$$\text{API calls: } 3 \rightarrow 2 \quad (\text{1 call saved})$$

(b) Online Context Compact

Evaluates compaction at plan-step completion using a cost gate. The gate compares projected input savings with estimated cache-rewrite cost:

$$\text{Compact if: } \text{projected input savings} > \text{estimated cache rewrite cost}$$

Later compactions account for unrecovered rewrite costs and require a larger savings margin.

(c) ObservationPack

For results exceeding 10 KiB:

• First 2 provider requests: full result sent
• From 3rd request onward: stable handle + original size + 1 KiB excerpt (head/tail lines)
• Exact pages retrievable on demand via handle

(d) Evidence-Preserving Reducer

Compresses build/test logs ≥ 4 KiB using a lower-cost model (GPT-5.6 Luna):

• Archives exact output
• Extracts key evidence into compact receipt
• Deterministic verifier checks: schema, source hash, exit status, exact quotes, size
• Falls back to original log on verification failure

2.5 Implementation Details

• All mechanisms implemented as extensions to Pi
• Action Fusion adds optional follow-up command to file-mutation tools
• Evidence-Preserving Reducer uses GPT-5.6 Luna (high setting)
• Before held-out evaluation: mechanism source, configuration, metrics, tolerances, and acceptance rule are frozen
• 11 EdgeBench tasks used for one-way acceptance; 40 reserved for final evaluation

---

Empirical Validation / Results

3.1 Overall Comparison (EdgeBench)

Table 1 | Harness comparison on EdgeBench (GPT-5.6 Sol backend):

Key findings:

• SoL-Pi [Efficiency]: 49.0% fewer tokens than Pi, 33.2% lower cost, retains 93.7% of Pi's score (42.0 vs 44.8)
• SoL-Pi [Performance]: +5.3% score gain (47.2 vs 44.8), 6.1% less token traffic, 9.8% better token efficiency

Cross-model transfer (Table 2) — SoL-Pi developed on GPT-5.6 Sol applied to Opus 5 without adaptation:

On Opus 5: retains 94.3% of Pi's score, reduces token traffic by 44.7%, API cost by 33.5%.

3.2 Terminal-Bench 4 and IMO 2026

Table 3 | Terminal-Bench 4 (63 tasks) and IMO 2026 (6 problems):

• TB4: SoL-Pi reduces total cost by 26.3% and cost/solved by 11.6% vs Pi
• IMO: SoL-Pi achieves lowest cost per passed problem ($20.90)

3.3 Efficient Agent Swarms

Kernel-optimization experiment (20 workers + coordinator):

• SoL-Pi swarm: 26.8% lower API cost than Pi swarm
• SoL-Pi swarm and single agent pass all 8 speed thresholds; Pi swarm misses final threshold

3.4 Single-Mechanism Evaluation

Table 4 | Add-one evaluation (GPT-5.6 Sol):

Cache trade-off: Complete stack reduces cache-read from 2.1326B → 1.0605B tokens, while cache-write increases from 0.0141B → 0.0316B. Despite additional writes, total cost falls from $1,339 →$894.

3.5 Action Fusion Case Study

• Oracle analysis: identified 12.3% direct headroom from missed adjacent actions; projected 11.5% token reduction
• Baseline build: prompt-only triggering unreliable → extended tool schema for direct fused action
• Prompt optimization: 18 iterations refining prompt/schema; introduced trigger rate as intermediate metric
• Final validation: 100% trigger rate, 87.0 task score
• Result: saved 149 model turns (10.8%) and 3.74M tokens (11.5%)

---

Theoretical and Practical Implications

Theoretical Contributions

1. RSI at the harness layer: Demonstrates that RSI-inspired search can discover reusable improvements that transfer beyond development environments—addressing the overfitting concern raised by Wang et al.

2. "Pretraining the Harness": Analogous to model pretraining, exposing the harness to many tasks and updating from trajectories suggests a scalable paradigm for continuous harness improvement.

3. Cache-cost trade-off insight: Shortening context can reduce prompt-cache reuse, but cached input still costs money. The study shows that total task cost should be evaluated rather than cache reuse alone.

4. Complementarity of mechanisms: Independent mechanisms show larger token-efficiency gains when combined, suggesting synergies across action execution, context management, and observation handling.

Practical Implications

• Cost savings: Estimated hourly savings of $8.75–$13.50 vs native Codex/Claude Code harnesses; $4.36–$5.71 vs Pi
• Cross-model transfer: Harness improvements developed on one LLM transfer to another without adaptation
• Agent swarms: More efficient harnesses enable better collective exploration under fixed budgets
• Production readiness: The frozen SoL-Pi harness achieves production-level outcomes with substantial cost reduction

---

Conclusion

SoL-Pi demonstrates that scaling auto-research loops across diverse environments can discover token-efficient harness mechanisms that:

• Reduce token traffic by 44.7–49.0%
• Cut API cost by ~33% at comparable performance
• Transfer across models (GPT-5.6 Sol → Opus 5) and benchmarks (EdgeBench → Terminal-Bench 4 → IMO 2026)

Future Directions

1. Pretraining the Harness: Scaling executable environments and research idea diversity for sustained gains
2. Multi-Backend Training: Training/validating harness across multiple LLM backends for robustness
3. Recursive Efficient Improvement: Using SoL-Pi as the starting harness for the next research cycle—efficiency as both outcome and resource for further search
4. Search Coverage and Cost: Systematic investigation of scaling laws for search breadth and depth under fixed budgets

The lasting value of RSI may lie in a search process that scales across public environments to discover reusable improvements—a perspective that positions SoL-Pi as a preliminary step toward scalable recursive self-improvement systems.