# SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

> SoL-Pi's auto-research loop discovers four reusable harness mechanisms that cut coding agent token traffic by up to 49% and API cost by ~33% while maintaining performance.

- **Source:** [arXiv](https://arxiv.org/abs/2609.20519)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/GumQ6D
- **Whiteboard:** https://picx.dev/p/GumQ6D/image

## Summary

# SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

## Summary (Overview)

- **SoL-Pi** is an RSI-inspired (Recursive Self-Improvement) system that automatically discovers token-efficient harness mechanisms for coding agents through a broad-to-deep auto-research funnel, yielding four reusable mechanisms that transfer across models and tasks.
- The four retained mechanisms—**Action Fusion**, **Online Context Compact**, **ObservationPack**, and **Evidence-Preserving Reducer**—target action execution, context compaction, observation handling, and delegated reading, respectively.
- On the 51-task EdgeBench benchmark, SoL-Pi achieves performance comparable to Pi across GPT-5.6 Sol and Opus 5 while reducing recorded token traffic by **44.7–49.0%** and API cost by **~33%** (about one third).
- Estimated hourly savings are **$8.75–$13.50** relative to native Codex and Claude Code harnesses, and **$4.36–$5.71** relative to Pi.
- The search process scaled to ~150 proposed directions, ~500 executable environments, >3,000 runs, and >60,000 agent–environment interactions, demonstrating that RSI at the harness layer can discover improvements that generalize beyond development settings.

---

## Introduction and Theoretical Foundation

### Background

As coding agents evolve from supervised code completion to unattended, around-the-clock exploration, their work expands from isolated predictions into **long trajectories of reasoning, tool use, and feedback**. This shift makes **task-level token efficiency** a first-order systems concern.

Existing efficiency work focuses on:
- Lowering cost per token via faster attention kernels and serving infrastructure
- Model compression (quantization)
- Using cheaper models

**SoL-Pi explores an orthogonal direction**: improving token use through the *agent harness*—the layer that mediates interactions between the model and its environment.

### Motivation

Optimizing a harness is difficult because tool use, context management, verification, delegation, recovery, and termination are tightly coupled. A locally beneficial change may cause downstream failures or shift token costs later in execution. Traditional harness development requires substantial human effort inspecting long execution traces—a process that is costly and difficult to scale.

### Theoretical Foundation

The work draws on:
- **Recursive Self-Improvement (RSI)** — agents improving their own infrastructure
- **Gödel Machine** — theoretical roots of self-modification
- **Meta-Harness** — searching over executable harness programs
- **Recursive Harness Self-Improvement (RHI)** — iterative refinement of agent loop specifications

Prior work (Wang et al.) found that evolved harnesses can overfit search tasks, motivating a **clear separation between search feedback and final evaluation**.

### Three Guiding Principles

1. **Breadth and depth** — Breadth expands hypothesis coverage; depth repeatedly implements, reviews, and hardens promising candidates.
2. **Independent validation** — Held-out evidence is evaluated only after a candidate is frozen; validation failures never feed back into search.
3. **Scalable orchestration** — Isolated, disposable lineages let the funnel expand without coupling failures across candidates.

---

## Methodology

### 2.1 Harness Auto-Research for Token Efficiency

The pipeline frames harness improvement as an RSI-inspired search:

1. **Research agent** analyzes execution trajectories from the base harness
2. Identifies recurring sources of overhead
3. Builds an **idea pool** of candidate harness changes
4. Tests effects in development environments

**Acceptance criteria** (fixed before experimentation):
- **Capability gate**: every capability metric must remain within predeclared tolerance
- **Efficiency gate**: candidate must improve at least one declared efficiency metric
- Among passing candidates, retain **nondominated results** under declared metrics

### 2.2 Broad-to-Deep Harness Search

**Outer stage**: Explores a broad pool of 152 proposed directions across six proposal families:
- Context, progress, tools, delegation, prompt/policy, improvement/evaluation

**Inner stage**: Develops selected hypotheses independently using an iterative implementation loop (Ralph Loop-based):
- Propose → Implement → Independent Review → Development Validation → Revise or Retain

Each independent search runs as a **disposable instance** of a shared skill template, with modified orchestration code discarded after completion.

### 2.3 Search Environments

| Environment Type | Count | Description |
|---|---|---|
| **Repository-derived** | 495 | GitHub issue–PR pairs with pre-fix repo state, hidden regression tests |
| **Verifier-driven** | 40 | Executable verifiers with multiple valid solution paths |

**EdgeBench** (51 tasks) is reserved for held-out validation, completely isolated from search.

### 2.4 Discovered Harness Mechanisms

#### (a) Action Fusion
Combines file mutation with follow-up command (test/build/run) into **one tool request**, eliminating an intermediate model round trip:

$$\text{API calls: } 3 \rightarrow 2 \quad (\text{1 call saved})$$

#### (b) Online Context Compact
Evaluates compaction at **plan-step completion** using a cost gate. The gate compares projected input savings with estimated cache-rewrite cost:

$$\text{Compact if: } \text{projected input savings} > \text{estimated cache rewrite cost}$$

Later compactions account for unrecovered rewrite costs and require a larger savings margin.

#### (c) ObservationPack
For results exceeding **10 KiB**:
- First 2 provider requests: full result sent
- From 3rd request onward: stable handle + original size + 1 KiB excerpt (head/tail lines)
- Exact pages retrievable on demand via handle

#### (d) Evidence-Preserving Reducer
Compresses build/test logs ≥ 4 KiB using a **lower-cost model** (GPT-5.6 Luna):
- Archives exact output
- Extracts key evidence into compact receipt
- **Deterministic verifier** checks: schema, source hash, exit status, exact quotes, size
- Falls back to original log on verification failure

### 2.5 Implementation Details

- All mechanisms implemented as extensions to **Pi**
- Action Fusion adds optional follow-up command to file-mutation tools
- Evidence-Preserving Reducer uses GPT-5.6 Luna (high setting)
- Before held-out evaluation: mechanism source, configuration, metrics, tolerances, and acceptance rule are **frozen**
- 11 EdgeBench tasks used for one-way acceptance; 40 reserved for final evaluation

---

## Empirical Validation / Results

### 3.1 Overall Comparison (EdgeBench)

**Table 1 | Harness comparison on EdgeBench** (GPT-5.6 Sol backend):

| Harness | Total Token Traffic (B) | Token Cost ($) ↓ | Avg. Score ↑ | Token Eff. ($/score) ↓ |
|---|---|---|---|---|
| Codex | 3.0537 | 1,787 | 34.738 | 1.0086 |
| OpenSquilla | 1.3353 | 1,243 | 24.506 | 0.9945 |
| Oh-My-Pi | 2.2235 | 1,832 | 26.921 | 1.3347 |
| OpenCode | 2.5668 | 3,422 | 29.552 | 2.2704 |
| Oh-My-Opencode | 2.5825 | 2,678 | 38.523 | 1.3633 |
| **Pi** | 2.1538 | 1,339 | 44.833 | 0.5855 |
| **SoL-Pi [Efficiency]** | **1.0990** | **894** | 42.003 | **0.4174** |
| **SoL-Pi [Performance]** | 2.0224 | 1,271 | **47.208** | 0.5280 |

**Key findings**:
- SoL-Pi [Efficiency]: **49.0% fewer tokens** than Pi, **33.2% lower cost**, retains **93.7% of Pi's score** (42.0 vs 44.8)
- SoL-Pi [Performance]: **+5.3% score gain** (47.2 vs 44.8), **6.1% less token traffic**, **9.8% better token efficiency**

**Cross-model transfer (Table 2)** — SoL-Pi developed on GPT-5.6 Sol applied to Opus 5 without adaptation:

| Configuration | Token Traffic (B) | Token Cost ($) | Avg. Score | Token Eff. ($/score) |
|---|---|---|---|---|
| **GPT-5.6 Sol** | | | | |
| Pi | 2.1538 | 1,339 | 44.833 | 0.5855 |
| SoL-Pi [Efficiency] | 1.0990 | 894 | 42.003 | 0.4174 |
| **Opus 5** | | | | |
| Pi | 2.3697 | 1,741 | 44.756 | 0.7625 |
| SoL-Pi [Efficiency] | 1.3101 | 1,158 | 42.224 | 0.5376 |

On Opus 5: retains **94.3% of Pi's score**, reduces token traffic by **44.7%**, API cost by **33.5%**.

### 3.2 Terminal-Bench 4 and IMO 2026

**Table 3 | Terminal-Bench 4 (63 tasks) and IMO 2026 (6 problems)**:

| Harness | TB4 Solved | TB4 Cost ($) | TB4 Cost/Solved ($) | IMO Pass | IMO Cost ($) | IMO Cost/Passed ($) |
|---|---|---|---|---|---|---|
| Codex | 18 | 272.35 | 15.13 | 5 | 114.47 | 22.89 |
| Pi | 18 | 286.45 | 15.91 | 3 | 75.95 | 25.32 |
| **SoL-Pi** | 15 | **211.12** | **14.07** | 3 | **62.69** | **20.90** |

- TB4: SoL-Pi reduces total cost by **26.3%** and cost/solved by **11.6%** vs Pi
- IMO: SoL-Pi achieves **lowest cost per passed problem** ($20.90)

### 3.3 Efficient Agent Swarms

Kernel-optimization experiment (20 workers + coordinator):

| Configuration | Final Cycles ↓ | Model Cost ($) |
|---|---|---|
| Single Codex agent | 1,333 | $39.20 |
| 20 Pi-baseline workers | 1,366 | $82.12 |
| **20 SoL-Pi workers** | **1,127** | **$60.11** |

- SoL-Pi swarm: **26.8% lower API cost** than Pi swarm
- SoL-Pi swarm and single agent pass all 8 speed thresholds; Pi swarm misses final threshold

### 3.4 Single-Mechanism Evaluation

**Table 4 | Add-one evaluation** (GPT-5.6 Sol):

| Configuration | Total Tokens (B) | Token Cost ($) | Avg. Score | Token Eff. ($/score) |
|---|---|---|---|---|
| Pi Baseline | 2.1538 | 1,339 | 44.833 | 0.5855 |
| + Action Fusion | 1.8968 | 1,235 | 46.664 | 0.5190 |
| + Online Context Compact | 1.2881 | 935 | 41.993 | 0.4365 |
| + Evidence-Preserving Reducer | 1.9375 | 1,200 | 44.630 | 0.5274 |
| + ObservationPack | 2.0224 | 1,271 | **47.208** | 0.5280 |
| **SoL-Pi [Efficiency]** | **1.0990** | **894** | 42.003 | **0.4174** |

**Cache trade-off**: Complete stack reduces cache-read from 2.1326B → 1.0605B tokens, while cache-write increases from 0.0141B → 0.0316B. Despite additional writes, total cost falls from $1,339 → $894.

### 3.5 Action Fusion Case Study

- **Oracle analysis**: identified 12.3% direct headroom from missed adjacent actions; projected 11.5% token reduction
- **Baseline build**: prompt-only triggering unreliable → extended tool schema for direct fused action
- **Prompt optimization**: 18 iterations refining prompt/schema; introduced **trigger rate** as intermediate metric
- **Final validation**: 100% trigger rate, 87.0 task score
- Result: saved **149 model turns (10.8%)** and **3.74M tokens (11.5%)**

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **RSI at the harness layer**: Demonstrates that RSI-inspired search can discover *reusable* improvements that transfer beyond development environments—addressing the overfitting concern raised by Wang et al.

2. **"Pretraining the Harness"**: Analogous to model pretraining, exposing the harness to many tasks and updating from trajectories suggests a scalable paradigm for continuous harness improvement.

3. **Cache-cost trade-off insight**: Shortening context can reduce prompt-cache reuse, but cached input still costs money. The study shows that **total task cost** should be evaluated rather than cache reuse alone.

4. **Complementarity of mechanisms**: Independent mechanisms show larger token-efficiency gains when combined, suggesting synergies across action execution, context management, and observation handling.

### Practical Implications

- **Cost savings**: Estimated hourly savings of **$8.75–$13.50** vs native Codex/Claude Code harnesses; **$4.36–$5.71** vs Pi
- **Cross-model transfer**: Harness improvements developed on one LLM transfer to another without adaptation
- **Agent swarms**: More efficient harnesses enable better collective exploration under fixed budgets
- **Production readiness**: The frozen SoL-Pi harness achieves production-level outcomes with substantial cost reduction

---

## Conclusion

SoL-Pi demonstrates that **scaling auto-research loops** across diverse environments can discover token-efficient harness mechanisms that:
- Reduce token traffic by **44.7–49.0%**
- Cut API cost by **~33%** at comparable performance
- Transfer across models (GPT-5.6 Sol → Opus 5) and benchmarks (EdgeBench → Terminal-Bench 4 → IMO 2026)

### Future Directions

1. **Pretraining the Harness**: Scaling executable environments and research idea diversity for sustained gains
2. **Multi-Backend Training**: Training/validating harness across multiple LLM backends for robustness
3. **Recursive Efficient Improvement**: Using SoL-Pi as the starting harness for the next research cycle—efficiency as both outcome and resource for further search
4. **Search Coverage and Cost**: Systematic investigation of scaling laws for search breadth and depth under fixed budgets

The lasting value of RSI may lie in a **search process that scales across public environments to discover reusable improvements**—a perspective that positions SoL-Pi as a preliminary step toward scalable recursive self-improvement systems.

---

_Markdown view of https://picx.dev/p/GumQ6D, served by PicX — AI-generated visual whiteboard summaries of research papers._