Full text not available for this paper

The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements

Authors: Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan, Kan Ren (ShanghaiTech University, ByteDance, BIGAI, SUSTech)


Summary (Overview)

  • Core hypothesis: Agent evolution stalls not because methods fail to propose better artifacts (prompts, skills, memories), but because noisy evaluations cannot reliably identify them—the authors term this the Resolution Hypothesis.
  • Massive scale: The study spans 7 evolution methods, 6 models, 4 agent benchmarks, 1,030 independent evolution runs, and over 10M LLM calls / 2.1M test trajectories—the largest reproducibility study of agent evolution to date.
  • Method-level finding: Typical method margins (~3.4 pp median) are smaller than the difference between two independent runs of the same method; single-run rankings often rest on luck (33% accuracy at best on Terminal-Bench vs. 25% random).
  • Candidate-level finding: Helpful candidate artifacts exist (Quality@K rises from 49.2% to 61.8% as K grows to 100), but all standard selection mechanisms—execution scores, native operators, LLM judges—perform at chance level (~49-50% pairwise accuracy).
  • Key mechanism: Large reported gains (10-40 pp) are traced to evaluation shortcuts—single sentences that exploit grader format rigidity or benchmark version drift—and vanish once evaluations are repaired (median gain drops from 17.6 to 0.8 pp).

Introduction and Theoretical Foundation

Background and Motivation

Agent evolution improves frozen LLM agents by refining persistent external artifacts (prompts, skills, memories, harnesses) through a proposer–selector loop:

Proposer writes candidate artifacts → Selector evaluates them on a task set → Selector accepts the candidate that scores higher.

While promising, the field suffers from instability: methods flip between effective and ineffective across benchmarks, rankings trade places across papers, and evolution repeatedly stalls on coding benchmarks like Terminal-Bench.

The Resolution Hypothesis

The Resolution Hypothesis. The bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.

Why identification is hard: Model weights are frozen, so an artifact helps only by changing inference-time behavior, while LLM trajectories vary each attempt. The authors define resolution as the smallest true improvement an evaluation pipeline can reliably distinguish from noise. Once true improvement falls below resolution, selection degrades to chance.

Theoretical Foundation: Variance Decomposition

The paper formalizes the problem with a variance decomposition. Let artifact AA be the persistent external state produced by a method. Observed score decomposes as:

Y^(A)=Y(A)+ϵ\hat{Y}(A) = Y(A) + \epsilon

where Y(A)=E[Y^∣A]Y(A) = \mathbb{E}[\hat{Y} | A] is true quality and ϵ\epsilon is zero-mean evaluation noise. By the law of total variance:

Var(Y^)=EA[Var(Y^∣A)]⏟σ2+VarA(E[Y^∣A])⏟τ2\text{Var}(\hat{Y}) = \underbrace{\mathbb{E}_A\left[\text{Var}(\hat{Y} | A)\right]}_{\sigma^2} + \underbrace{\text{Var}_A\left(\mathbb{E}[\hat{Y} | A]\right)}_{\tau^2}

where:

  • σ2\sigma^2 = average evaluation noise within each artifact
  • τ2\tau^2 = run-to-run variance in true artifact quality

Key insight: Identifying the gain of a fixed artifact is limited by σ2\sigma^2 alone, while comparing two methods is limited by (τ2+σ2)(\tau^2 + \sigma^2), since each run draws a new artifact.


Methodology

Experimental Design (Four Groups)

GroupPurposeMethods/Benchmarks
Variance measurementEstimate τ\tau and σ\sigmaAll 7 methods on ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1
Gain auditRe-examine largest reported gains12 benchmarks, 3 models, repaired evaluations
Candidate poolsTest proposal quality & selection reliability2 pools of n=100n=100 skills on Terminal-Bench
Trajectory re-evaluationMeasure cumulative selection errors8 SkillOpt-Lite + 8 GEPA runs, re-evaluated 8× per task

Methods and Models

Seven evolution methods in two families:

  • Text-artifact methods: GEPA (prompts), SkillOpt-Lite (skills), Trace2Skill (skills), ReasoningBank (memory playbooks)
  • Harness methods: HarnessLens, Meta-Harness, AHE (agent scaffolding)

Six models: GPT-5.4 nano, GPT-5.6 Luna/Terra, DeepSeek-V4-Flash, Kimi-K3, GLM-5.2

Key Protocols

  • Each method run 8 times per model/benchmark (16 for text-artifact methods on GPT-5.4 nano)
  • Each artifact evaluated 16 times on held-out test splits (8 on Terminal-Bench)
  • Proposal Pool: 100 skills generated from 10 training tasks, evaluated on 10 disjoint test tasks with 16 rollouts per pair (16,000 total)
  • Selection Pool: 100 skills from Terminal-Bench + DeepSWE trajectories across 11 difficulty-stratified tasks

Empirical Validation / Results

Method-Level Results (Section 4)

Single reports cannot rank methods. The evaluation noise σ\sigma ranges from 1.0–3.7 pp (SpreadsheetBench, OfficeQA, ALFWorld) to 4.5 pp (Terminal-Bench); run-to-run variance τ\tau ranges from 0 to 12.3 pp.

For over 60% of configuration pairs, the gap between all-run means falls below one standard deviation of a single-run comparison. Surveying 171 benchmark–model results from 9 papers, the median reported margin is 3.4 pp (IQR: 1.2–6.6 pp).

Required runs for statistical power. Identifying a true margin Δ\Delta at 80% power with α=0.05\alpha = 0.05 requires:

N≥2⋅(z1−α/2+z1−βΔ)2⋅(τ2+σ2)N \geq 2 \cdot \left(\frac{z_{1-\alpha/2} + z_{1-\beta}}{\Delta}\right)^2 \cdot (\tau^2 + \sigma^2)

At the median 3.4 pp margin, this means ~5 runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and 23 on Terminal-Bench (181 at the lower-quartile margin).

Evaluation shortcuts explain large gains. The gain audit found that:

  • A single sentence appended to the baseline recovers a median of 90% of reported gains across 27 model–benchmark pairs
  • Repairing evaluations (restoring official grading) drops median gain from 17.6 to 0.8 pp
  • Shortcuts exploit grader format rigidity or benchmark version drift

Candidate-Level Results (Section 5)

P1: Helpful artifacts exist. Quality@K (expected success of best among KK skills) climbs steadily:

BudgetQ@K=1Q@K=100Gain
2-hour49.2%61.8%+12.6 pp
Original timeout39.7%51.6%+11.9 pp

The observed curve pulls above the equal-quality null by K=5K=5, maintaining a +3.3 to +3.4 pp lead at K=100K=100, isolating genuine skill differences from sampling luck.

P2: Selection operates near chance. All selectors fail to identify helpful artifacts:

Selection MechanismPairwise Acc. (%)False Accept. (%)
Random50.0 ± 0.030.5 ± 14.0
Execution (1 rollout)49.7 ± 1.962.1 ± 42.9
SkillOpt-Lite48.6 ± 2.659.4 ± 46.2
GEPA48.6 ± 2.631.6 ± 14.5
LLM judge (pointwise)45.4 ± 4.539.9 ± 17.5
LLM judge (pairwise)49.0 ± 2.257.6 ± 23.3
LLM judge (ACE)46.9 ± 3.539.0 ± 15.6

The rank correlation between single execution score and posterior quality is 0.054 (essentially zero). No selector benefits from more candidates—all remain flat around random baseline even at K=100K=100.

P3: Errors compound across rounds. In-loop validation gains grow to 6.9–9.0 pp, but independently re-evaluated gains remain between +0.4 and −3.2 pp. Run-to-run variance τ^t2\hat{\tau}^2_t grows from τ02=0\tau^2_0 = 0 to 4.2–24.5 squared pp by the final round, matching the margins that separate methods in the literature.


Theoretical and Practical Implications

Theoretical Significance

  1. Paradigm shift: The field should move from designing ever more candidate improvement systems to making true improvements detectable and accumulable. The proposer determines what improvements are available; evaluation resolution determines which improvements can be retained.

  2. Unified explanation: The same variance that prevents single-step selection from identifying improvement also destabilizes method rankings, compounded over rounds. Both failures trace to a single constraint: evaluation resolution.

  3. Shortcut exploitation: Large reported gains are consistent with the hypothesis "from the other side"—they exceed resolution by exploiting evaluation shortcuts that lift nearly every task at once.

Practical Recommendations

Candidate level — finer-grained feedback: Supplement terminal pass/fail with programmatic checks of intermediate behavior (e.g., unmasked failure diagnostics on verification commands, whether agent runs tests after editing). Preliminary analysis shows these checks exhibit 13×–57× lower evaluation noise than terminal verdicts.

Method level — component-wise evaluation:

  • Proposers: Quality@K curves measure best-candidate quality among KK proposals
  • Selectors: Selected-artifact quality curves measure quality recovered from a fixed pool as budget grows
  • Reporting: Publish both σ^\hat{\sigma} (evaluation noise) and τ^\hat{\tau} (run-to-run variance) alongside point scores so readers can judge whether a claimed margin exceeds resolution

Conclusion

The paper proposes and validates the Resolution Hypothesis: agent evolution is bottlenecked by identifying helpful artifacts, not proposing them. Evidence spans both levels where evaluation scores are used:

  • Method level: Published margins (~3.4 pp) are comparable to single-run fluctuation (1.7–4.1 pp SD), making rankings noise-dominated
  • Candidate level: Helpful artifacts exist, but all selectors perform at chance; per-round errors compound into run-to-run variance
  • Shortcut gains: The largest reported gains exploit evaluation shortcuts and vanish after repair

Future directions:

  1. Resolution-aware evaluations: Less noisy intermediate process metrics
  2. Component-wise diagnostics: Separate proposer and selector evaluation
  3. Variance reporting: Standard practice of reporting σ^\hat{\sigma} and τ^\hat{\tau}
  4. Program evolution: Deterministic code evolution (GPU kernels, ARC-AGI-2) may structurally reduce noise through cheap parallel re-evaluation—an open question the framework can address

Limitations

  • Frozen weights only: RL-based weight updates may face different resolution constraints
  • Candidate pool scope: Terminal-Bench v2.1 only; larger pools and other domains untested
  • Model strength: Stronger future models may exhibit lower noise, though Terminal-Bench median stays at 3.5–4.0 pp even for the strongest models tested
  • Grid coverage: Seven methods and six models is broad but not exhaustive

Related papers