# DuoDiscover: Scaling Autonomous Discovery Beyond the Evaluation Bottleneck

> DuoDiscover co-discovers optimization targets and cheap proxy evaluators with fidelity testing, achieving up to 15x speedups and beating human experts across software and chip design.

- **Source:** [arXiv](https://arxiv.org/abs/2610.17184)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/vFeeK0
- **Whiteboard:** https://picx.dev/p/vFeeK0/image

## Summary

## Summary (Overview)

- **DuoDiscover** is a novel search system for autonomous code optimization that addresses the evaluation bottleneck—where a single true evaluation (e.g., full benchmark or serving trace) can take hours, limiting search to only a handful of evaluations.
- The system introduces **co-discovery**: each worker agent simultaneously identifies an optimization target (a code region) and builds a cheap, tailored proxy evaluator called a **yardstick**, then hill-climbs on the yardstick rather than the expensive true objective.
- **Fidelity testing** continuously validates yardsticks against the true objective using Kendall's τ rank correlation, invalidating any that drift due to reward hacking, while all gains are confirmed by true evaluations.
- A **bottom-up fleet** of agents is coordinated by a lightweight orchestrator that terminates stalled agents, spawns new ones, briefs running agents, and combines compatible gains—without central problem decomposition.
- Results span the computing stack: 2.16× on FormulaCode, 15.02× on SWE-ficiency (doubling human expert speedup), first place on 121/235 SOL-Exec kernels, 1.92× on vLLM, 1.38× on SQLite, and a 6.31% gain on CPU microarchitecture design—surpassing both AlphaEvolve-based ArchAgent v2 and the best human design.

## Introduction and Theoretical Foundation

The paper addresses a fundamental bottleneck in scaling autonomous discovery: **evaluation cost**. While prior work on LLM-driven search (e.g., FunSearch, AlphaEvolve) focuses on refining search harnesses, they largely treat the evaluator as given and fixed. In real engineering systems, this assumption breaks down—e.g., optimizing vLLM for multi-objective serving requires 85 minutes per true evaluation, limiting even frontier coding agents to a dozen evaluations within a 20-hour budget.

The key insight is that **large speedups often come from focused edits to a few small regions of code (targets)**, whose progress can be measured much more cheaply than the entire true evaluation. However, proxy evaluators are target-specific, making hand-design impractical. Moreover, effective proxy-guided search requires:
1. Jointly identifying promising targets **and** cheap, faithful proxies (optimizing the wrong target yields little benefit; an inaccurate proxy misguides search).
2. Maintaining fidelity—even faithful proxies can drift as search exploits discrepancies between proxy and true objective.

The theoretical foundation draws on **Amdahl's Law** to bound potential gains from any single target region, and **Kendall's τ rank correlation** for nonparametric assessment of yardstick fidelity.

## Methodology

### Problem Formulation

Let $X$ be the space of program states, $x_0 \in X$ the initial state, and the expensive true objective $f: X \to \mathbb{R} \cup \{\bot\}$ mapping states to scores (or failure $\bot$). The cost of one call to $f$ is $c_f$ (ranging from minutes to hours). Within a run, $N$ agents work under shared wall-clock and evaluation budgets.

### Co-Discovery of Targets and Yardsticks

- **Target** $T$: a region of code (function, module). The set $\mathcal{X}_T \subseteq \mathcal{X}$ denotes program states reachable by changing code only inside $T$.
- The agent estimates $w_T \in [0,1]$, the share of true score attributable to $T$. By Amdahl's law, even infinite speedup of $T$ yields at most $1/(1-w_T)$ overall improvement.
- **Yardstick** $g_T: \mathcal{X}_T \to \mathbb{R} \cup \{\bot\}$: a cheap scorer ($c_g \ll c_f$) written as a script. The absolute value need not track $f$—only the **ordering** of edits matters:

$$\text{sign}(g_T(x) - g_T(x')) = \text{sign}(f(x) - f(x'))$$

### Two-Loop Search Architecture

Each agent runs:
- **Fast loop**: edits target, scores with yardstick (seconds per edit), keeps improvements, reverts otherwise.
- **Slow loop**: true evaluation confirms gains and runs fidelity tests (hours each).

### Orchestrator Actions

The lightweight orchestrator coordinates via three actions:
1. **Terminate** agents not making progress.
2. **Spawn** new agents—fresh, seeded with confirmed state (transfer), or re-implementing several agents' changes (combine).
3. **Brief** running agents with fleet-wide findings.

### Fidelity Testing

The orchestrator computes Kendall's τ on pairs of states whose true scores differ by more than evaluation noise. A yardstick is **validated** when $\tau \geq 0.5$; it is **invalidated** if correlation later drops, forcing the agent to rebuild or move on.

## Empirical Validation / Results

### Benchmark Results (8-hour, 4-agent budget)

| Method | FormulaCode (×) | SWE-ficiency (×) | SOL-Exec (% SOL) | FA4 (%) | JAXBench (×) | TPU-Inference (×) |
|---|---|---|---|---|---|---|
| Claude Code | 1.778 | 12.81 | 67.31 | 7.05 | 3.82 | 2.38 |
| OpenEvolve | 1.161 | 3.76 | 52.36 | 1.48 | 2.66 | 1.14 |
| PerfAgent | 1.774 | 12.27 | 64.10 | 6.86 | 4.08 | 2.14 |
| KDA | - | - | 70.22 | 8.93 | - | - |
| **DuoDiscover** | **2.156** | **15.02** | **79.74** | **11.97** | **4.62** | **2.68** |

DuoDiscover outperforms all baselines on every benchmark, with a 21% gain over the strongest agentic baseline on FormulaCode.

### Real-World Case Studies

- **vLLM**: 1.92× geometric mean across four SLOs (p99 decode latency, first-token latency, GPU power, throughput) vs. Claude Code's 1.07×.
- **FreeToken**: 1.44× geometric mean (1.17× at batch 1, 1.76× at batch 8).
- **SQLite**: 1.38× vs. 1.12× for Claude Code, using 17 vs. 45 true evaluations.
- **zstd**: 1.16× instruction ratio vs. 1.05×, with 7 vs. 21 true evaluations.

### CPU Microarchitecture Design (DPC4)

| Method | FullBW | LimitBW | Multi-Core | Overall |
|---|---|---|---|---|
| BertiGO (best human) | +5.16% | +2.57% | +2.85% | +3.52% |
| ArchAgent v2 | +5.61% | +4.56% | +1.33% | +3.82% |
| **DuoDiscover** | **+9.08%** | **+5.93%** | **+3.93%** | **+6.31%** |

DuoDiscover achieves this with ~8,000 candidates over 16.7 days vs. ArchAgent v2's 12,000+ candidates over 40+ days—the first autonomous system to outperform the human best on multi-core cases.

### Yardstick Quality and Search Efficiency

| | True eval | Yardstick | Faster by | Validated | Invalidated | τ |
|---|---|---|---|---|---|---|
| SWE-ficiency | 15 min | 40 s | 22× | 5.3/9.2 targets | 41% | 0.66 |
| SOL-Exec | 14 min | 16 s | 53× | 11/15 targets | 45% | 0.76 |

DuoDiscover equals Claude Code's final SOL score in 2.8 hours using 74% fewer tokens and 83% fewer true evaluations, cutting run cost from $309 to $79.60.

### Component Ablations

| Configuration | SWE-ficiency (×) | SOL-Exec (% SOL) |
|---|---|---|
| Claude Code (N=4) | 12.85 | 69.12 |
| + target | 12.71 | 69.68 |
| + yardstick (no fidelity test) | 10.33 | 66.91 |
| + fidelity test | 13.25 | 72.57 |
| + co-discovery | 13.76 | 74.43 |
| **+ orchestrator (DuoDiscover)** | **14.93** | **80.80** |

Untested yardsticks hurt performance (reward hacking), while the fidelity test recovers the loss and the orchestrator provides the largest single gain on kernels (+6.37 SOL points).

## Theoretical and Practical Implications

- **Evaluation as a discoverable artifact**: DuoDiscover reframes evaluation from a fixed input to a searchable component, showing that cheap, faithful proxies can be synthesized per-target by LLM agents.
- **Amdahl's law as a search guide**: The target-weight estimate $w_T$ provides an upper bound on achievable gains, helping agents avoid unpromising targets.
- **Bottom-up coordination**: The orchestrator's lightweight actions (spawn, terminate, brief, transfer, combine) collectively outperform any single mechanism, with ablation losses summing to more than the orchestrator's net gain—demonstrating synergy.
- **Backbone generalization**: DuoDiscover works across different coding agents (Claude Code/Opus 5 and Antigravity/Gemini 3.8 Flash), largely closing the gap between them and making cheaper models viable (5.1× cost reduction).
- **Practical impact**: Demonstrates autonomous optimization from software (vLLM, SQLite, zstd) to chip architecture, with the first autonomous system to beat human best on multi-core prefetcher design.

## Conclusion

DuoDiscover establishes that scaling autonomous discovery requires **discovering both what to optimize and how to evaluate progress efficiently**. The co-discovery of targets and yardsticks, combined with fidelity testing and bottom-up fleet coordination, enables autonomous optimization across the computing stack—from repositories and kernels to heavily optimized production systems and chip architecture.

**Limitations**:
1. When no faithful yardstick can be constructed, DuoDiscover falls back to standard search over the expensive objective.
2. Yardstick construction and initial fidelity testing incur upfront warmup overhead, making the approach less advantageous under extremely short evaluation budgets.

**Future directions**: Extending co-discovery to other scientific and engineering domains, improving yardstick robustness, and scaling fleet coordination to larger, more heterogeneous search spaces.

---

_Markdown view of https://picx.dev/p/vFeeK0, served by PicX — AI-generated visual whiteboard summaries of research papers._