# The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

> Agent evolution fails because noisy evaluations cannot identify helpful improvements, not because proposals are lacking, making selection nearly random.

- **Source:** [arXiv](https://arxiv.org/abs/2611.23068)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/wE54GM
- **Whiteboard:** https://picx.dev/p/wE54GM/image

## Summary

# The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

**Authors:** Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan, Kan Ren (ShanghaiTech University, ByteDance, BIGAI, SUSTech)

---

## Summary (Overview)

- **Core hypothesis**: Agent evolution stalls not because methods fail to *propose* better artifacts (prompts, skills, memories), but because noisy evaluations cannot reliably *identify* them—the authors term this the **Resolution Hypothesis**.
- **Massive scale**: The study spans 7 evolution methods, 6 models, 4 agent benchmarks, 1,030 independent evolution runs, and over 10M LLM calls / 2.1M test trajectories—the largest reproducibility study of agent evolution to date.
- **Method-level finding**: Typical method margins (~3.4 pp median) are smaller than the difference between two independent runs of the same method; single-run rankings often rest on luck (33% accuracy at best on Terminal-Bench vs. 25% random).
- **Candidate-level finding**: Helpful candidate artifacts exist (Quality@K rises from 49.2% to 61.8% as K grows to 100), but all standard selection mechanisms—execution scores, native operators, LLM judges—perform at chance level (~49-50% pairwise accuracy).
- **Key mechanism**: Large reported gains (10-40 pp) are traced to *evaluation shortcuts*—single sentences that exploit grader format rigidity or benchmark version drift—and vanish once evaluations are repaired (median gain drops from 17.6 to 0.8 pp).

---

## Introduction and Theoretical Foundation

### Background and Motivation

Agent evolution improves frozen LLM agents by refining persistent external artifacts (prompts, skills, memories, harnesses) through a **proposer–selector loop**:

> **Proposer** writes candidate artifacts → **Selector** evaluates them on a task set → **Selector** accepts the candidate that scores higher.

While promising, the field suffers from **instability**: methods flip between effective and ineffective across benchmarks, rankings trade places across papers, and evolution repeatedly stalls on coding benchmarks like Terminal-Bench.

### The Resolution Hypothesis

> **The Resolution Hypothesis.** The bottleneck of agent evolution is *identifying* which artifacts bring improvement, not *proposing* them.

**Why identification is hard**: Model weights are frozen, so an artifact helps only by changing inference-time behavior, while LLM trajectories vary each attempt. The authors define **resolution** as the smallest true improvement an evaluation pipeline can reliably distinguish from noise. Once true improvement falls below resolution, selection degrades to chance.

### Theoretical Foundation: Variance Decomposition

The paper formalizes the problem with a variance decomposition. Let artifact $A$ be the persistent external state produced by a method. Observed score decomposes as:

$$\hat{Y}(A) = Y(A) + \epsilon$$

where $Y(A) = \mathbb{E}[\hat{Y} | A]$ is true quality and $\epsilon$ is zero-mean evaluation noise. By the law of total variance:

$$\text{Var}(\hat{Y}) = \underbrace{\mathbb{E}_A\left[\text{Var}(\hat{Y} | A)\right]}_{\sigma^2} + \underbrace{\text{Var}_A\left(\mathbb{E}[\hat{Y} | A]\right)}_{\tau^2}$$

where:
- $\sigma^2$ = average evaluation noise within each artifact
- $\tau^2$ = run-to-run variance in true artifact quality

**Key insight**: Identifying the gain of a fixed artifact is limited by $\sigma^2$ alone, while comparing two methods is limited by $(\tau^2 + \sigma^2)$, since each run draws a new artifact.

---

## Methodology

### Experimental Design (Four Groups)

| Group | Purpose | Methods/Benchmarks |
|-------|---------|-------------------|
| **Variance measurement** | Estimate $\tau$ and $\sigma$ | All 7 methods on ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1 |
| **Gain audit** | Re-examine largest reported gains | 12 benchmarks, 3 models, repaired evaluations |
| **Candidate pools** | Test proposal quality & selection reliability | 2 pools of $n=100$ skills on Terminal-Bench |
| **Trajectory re-evaluation** | Measure cumulative selection errors | 8 SkillOpt-Lite + 8 GEPA runs, re-evaluated 8× per task |

### Methods and Models

**Seven evolution methods** in two families:
- **Text-artifact methods**: GEPA (prompts), SkillOpt-Lite (skills), Trace2Skill (skills), ReasoningBank (memory playbooks)
- **Harness methods**: HarnessLens, Meta-Harness, AHE (agent scaffolding)

**Six models**: GPT-5.4 nano, GPT-5.6 Luna/Terra, DeepSeek-V4-Flash, Kimi-K3, GLM-5.2

### Key Protocols

- Each method run **8 times** per model/benchmark (16 for text-artifact methods on GPT-5.4 nano)
- Each artifact evaluated **16 times** on held-out test splits (8 on Terminal-Bench)
- **Proposal Pool**: 100 skills generated from 10 training tasks, evaluated on 10 disjoint test tasks with 16 rollouts per pair (16,000 total)
- **Selection Pool**: 100 skills from Terminal-Bench + DeepSWE trajectories across 11 difficulty-stratified tasks

---

## Empirical Validation / Results

### Method-Level Results (Section 4)

**Single reports cannot rank methods.** The evaluation noise $\sigma$ ranges from 1.0–3.7 pp (SpreadsheetBench, OfficeQA, ALFWorld) to 4.5 pp (Terminal-Bench); run-to-run variance $\tau$ ranges from 0 to 12.3 pp.

For **over 60% of configuration pairs**, the gap between all-run means falls below one standard deviation of a single-run comparison. Surveying 171 benchmark–model results from 9 papers, the median reported margin is **3.4 pp** (IQR: 1.2–6.6 pp).

**Required runs for statistical power**. Identifying a true margin $\Delta$ at 80% power with $\alpha = 0.05$ requires:

$$N \geq 2 \cdot \left(\frac{z_{1-\alpha/2} + z_{1-\beta}}{\Delta}\right)^2 \cdot (\tau^2 + \sigma^2)$$

At the median 3.4 pp margin, this means ~5 runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and **23 on Terminal-Bench** (181 at the lower-quartile margin).

**Evaluation shortcuts explain large gains**. The gain audit found that:
- A **single sentence** appended to the baseline recovers a median of **90% of reported gains** across 27 model–benchmark pairs
- **Repairing evaluations** (restoring official grading) drops median gain from **17.6 to 0.8 pp**
- Shortcuts exploit grader format rigidity or benchmark version drift

### Candidate-Level Results (Section 5)

**P1: Helpful artifacts exist.** Quality@K (expected success of best among $K$ skills) climbs steadily:

| Budget | Q@K=1 | Q@K=100 | Gain |
|--------|-------|---------|------|
| 2-hour | 49.2% | 61.8% | +12.6 pp |
| Original timeout | 39.7% | 51.6% | +11.9 pp |

The observed curve pulls above the equal-quality null by $K=5$, maintaining a **+3.3 to +3.4 pp lead** at $K=100$, isolating genuine skill differences from sampling luck.

**P2: Selection operates near chance.** All selectors fail to identify helpful artifacts:

| Selection Mechanism | Pairwise Acc. (%) | False Accept. (%) |
|--------------------|-------------------|-------------------|
| Random | 50.0 ± 0.0 | 30.5 ± 14.0 |
| Execution (1 rollout) | 49.7 ± 1.9 | 62.1 ± 42.9 |
| SkillOpt-Lite | 48.6 ± 2.6 | 59.4 ± 46.2 |
| GEPA | 48.6 ± 2.6 | 31.6 ± 14.5 |
| LLM judge (pointwise) | 45.4 ± 4.5 | 39.9 ± 17.5 |
| LLM judge (pairwise) | 49.0 ± 2.2 | 57.6 ± 23.3 |
| LLM judge (ACE) | 46.9 ± 3.5 | 39.0 ± 15.6 |

The rank correlation between single execution score and posterior quality is **0.054** (essentially zero). No selector benefits from more candidates—all remain flat around random baseline even at $K=100$.

**P3: Errors compound across rounds.** In-loop validation gains grow to 6.9–9.0 pp, but independently re-evaluated gains remain between **+0.4 and −3.2 pp**. Run-to-run variance $\hat{\tau}^2_t$ grows from $\tau^2_0 = 0$ to **4.2–24.5 squared pp** by the final round, matching the margins that separate methods in the literature.

---

## Theoretical and Practical Implications

### Theoretical Significance

1. **Paradigm shift**: The field should move from designing ever more candidate improvement systems to making true improvements *detectable and accumulable*. The proposer determines what improvements are *available*; evaluation resolution determines which improvements can be *retained*.

2. **Unified explanation**: The same variance that prevents single-step selection from identifying improvement also destabilizes method rankings, compounded over rounds. Both failures trace to a single constraint: evaluation resolution.

3. **Shortcut exploitation**: Large reported gains are consistent with the hypothesis "from the other side"—they exceed resolution by exploiting evaluation shortcuts that lift nearly every task at once.

### Practical Recommendations

**Candidate level — finer-grained feedback**: Supplement terminal pass/fail with programmatic checks of intermediate behavior (e.g., unmasked failure diagnostics on verification commands, whether agent runs tests after editing). Preliminary analysis shows these checks exhibit **13×–57× lower evaluation noise** than terminal verdicts.

**Method level — component-wise evaluation**:
- **Proposers**: Quality@K curves measure best-candidate quality among $K$ proposals
- **Selectors**: Selected-artifact quality curves measure quality recovered from a fixed pool as budget grows
- **Reporting**: Publish both $\hat{\sigma}$ (evaluation noise) and $\hat{\tau}$ (run-to-run variance) alongside point scores so readers can judge whether a claimed margin exceeds resolution

---

## Conclusion

The paper proposes and validates the **Resolution Hypothesis**: agent evolution is bottlenecked by *identifying* helpful artifacts, not proposing them. Evidence spans both levels where evaluation scores are used:

- **Method level**: Published margins (~3.4 pp) are comparable to single-run fluctuation (1.7–4.1 pp SD), making rankings noise-dominated
- **Candidate level**: Helpful artifacts exist, but all selectors perform at chance; per-round errors compound into run-to-run variance
- **Shortcut gains**: The largest reported gains exploit evaluation shortcuts and vanish after repair

**Future directions**:
1. **Resolution-aware evaluations**: Less noisy intermediate process metrics
2. **Component-wise diagnostics**: Separate proposer and selector evaluation
3. **Variance reporting**: Standard practice of reporting $\hat{\sigma}$ and $\hat{\tau}$
4. **Program evolution**: Deterministic code evolution (GPU kernels, ARC-AGI-2) may structurally reduce noise through cheap parallel re-evaluation—an open question the framework can address

### Limitations

- **Frozen weights only**: RL-based weight updates may face different resolution constraints
- **Candidate pool scope**: Terminal-Bench v2.1 only; larger pools and other domains untested
- **Model strength**: Stronger future models may exhibit lower noise, though Terminal-Bench median stays at 3.5–4.0 pp even for the strongest models tested
- **Grid coverage**: Seven methods and six models is broad but not exhaustive

---

_Markdown view of https://picx.dev/p/wE54GM, served by PicX — AI-generated visual whiteboard summaries of research papers._
