Full text not available for this paper
The Resolution Hypothesis: Agent Evolution's Key Is Identifying Improvements
Authors: Yuanhao Zeng, Shaocheng Lan, Ao Lu, Kuang Yan, Heng Lin, Xinang Xu, Yuan Luo, Yueting Wang, Yexin Li, Ruo Shan, Kan Ren (ShanghaiTech University, ByteDance, BIGAI, SUSTech)
Summary (Overview)
- Core hypothesis: Agent evolution stalls not because methods fail to propose better artifacts (prompts, skills, memories), but because noisy evaluations cannot reliably identify them—the authors term this the Resolution Hypothesis.
- Massive scale: The study spans 7 evolution methods, 6 models, 4 agent benchmarks, 1,030 independent evolution runs, and over 10M LLM calls / 2.1M test trajectories—the largest reproducibility study of agent evolution to date.
- Method-level finding: Typical method margins (~3.4 pp median) are smaller than the difference between two independent runs of the same method; single-run rankings often rest on luck (33% accuracy at best on Terminal-Bench vs. 25% random).
- Candidate-level finding: Helpful candidate artifacts exist (Quality@K rises from 49.2% to 61.8% as K grows to 100), but all standard selection mechanisms—execution scores, native operators, LLM judges—perform at chance level (~49-50% pairwise accuracy).
- Key mechanism: Large reported gains (10-40 pp) are traced to evaluation shortcuts—single sentences that exploit grader format rigidity or benchmark version drift—and vanish once evaluations are repaired (median gain drops from 17.6 to 0.8 pp).
Introduction and Theoretical Foundation
Background and Motivation
Agent evolution improves frozen LLM agents by refining persistent external artifacts (prompts, skills, memories, harnesses) through a proposer–selector loop:
Proposer writes candidate artifacts → Selector evaluates them on a task set → Selector accepts the candidate that scores higher.
While promising, the field suffers from instability: methods flip between effective and ineffective across benchmarks, rankings trade places across papers, and evolution repeatedly stalls on coding benchmarks like Terminal-Bench.
The Resolution Hypothesis
The Resolution Hypothesis. The bottleneck of agent evolution is identifying which artifacts bring improvement, not proposing them.
Why identification is hard: Model weights are frozen, so an artifact helps only by changing inference-time behavior, while LLM trajectories vary each attempt. The authors define resolution as the smallest true improvement an evaluation pipeline can reliably distinguish from noise. Once true improvement falls below resolution, selection degrades to chance.
Theoretical Foundation: Variance Decomposition
The paper formalizes the problem with a variance decomposition. Let artifact be the persistent external state produced by a method. Observed score decomposes as:
where is true quality and is zero-mean evaluation noise. By the law of total variance:
where:
- = average evaluation noise within each artifact
- = run-to-run variance in true artifact quality
Key insight: Identifying the gain of a fixed artifact is limited by alone, while comparing two methods is limited by , since each run draws a new artifact.
Methodology
Experimental Design (Four Groups)
| Group | Purpose | Methods/Benchmarks |
|---|---|---|
| Variance measurement | Estimate and | All 7 methods on ALFWorld, SpreadsheetBench, OfficeQA, Terminal-Bench v2.1 |
| Gain audit | Re-examine largest reported gains | 12 benchmarks, 3 models, repaired evaluations |
| Candidate pools | Test proposal quality & selection reliability | 2 pools of skills on Terminal-Bench |
| Trajectory re-evaluation | Measure cumulative selection errors | 8 SkillOpt-Lite + 8 GEPA runs, re-evaluated 8× per task |
Methods and Models
Seven evolution methods in two families:
- Text-artifact methods: GEPA (prompts), SkillOpt-Lite (skills), Trace2Skill (skills), ReasoningBank (memory playbooks)
- Harness methods: HarnessLens, Meta-Harness, AHE (agent scaffolding)
Six models: GPT-5.4 nano, GPT-5.6 Luna/Terra, DeepSeek-V4-Flash, Kimi-K3, GLM-5.2
Key Protocols
- Each method run 8 times per model/benchmark (16 for text-artifact methods on GPT-5.4 nano)
- Each artifact evaluated 16 times on held-out test splits (8 on Terminal-Bench)
- Proposal Pool: 100 skills generated from 10 training tasks, evaluated on 10 disjoint test tasks with 16 rollouts per pair (16,000 total)
- Selection Pool: 100 skills from Terminal-Bench + DeepSWE trajectories across 11 difficulty-stratified tasks
Empirical Validation / Results
Method-Level Results (Section 4)
Single reports cannot rank methods. The evaluation noise ranges from 1.0–3.7 pp (SpreadsheetBench, OfficeQA, ALFWorld) to 4.5 pp (Terminal-Bench); run-to-run variance ranges from 0 to 12.3 pp.
For over 60% of configuration pairs, the gap between all-run means falls below one standard deviation of a single-run comparison. Surveying 171 benchmark–model results from 9 papers, the median reported margin is 3.4 pp (IQR: 1.2–6.6 pp).
Required runs for statistical power. Identifying a true margin at 80% power with requires:
At the median 3.4 pp margin, this means ~5 runs on SpreadsheetBench, 10 on OfficeQA, 19 on ALFWorld, and 23 on Terminal-Bench (181 at the lower-quartile margin).
Evaluation shortcuts explain large gains. The gain audit found that:
- A single sentence appended to the baseline recovers a median of 90% of reported gains across 27 model–benchmark pairs
- Repairing evaluations (restoring official grading) drops median gain from 17.6 to 0.8 pp
- Shortcuts exploit grader format rigidity or benchmark version drift
Candidate-Level Results (Section 5)
P1: Helpful artifacts exist. Quality@K (expected success of best among skills) climbs steadily:
| Budget | Q@K=1 | Q@K=100 | Gain |
|---|---|---|---|
| 2-hour | 49.2% | 61.8% | +12.6 pp |
| Original timeout | 39.7% | 51.6% | +11.9 pp |
The observed curve pulls above the equal-quality null by , maintaining a +3.3 to +3.4 pp lead at , isolating genuine skill differences from sampling luck.
P2: Selection operates near chance. All selectors fail to identify helpful artifacts:
| Selection Mechanism | Pairwise Acc. (%) | False Accept. (%) |
|---|---|---|
| Random | 50.0 ± 0.0 | 30.5 ± 14.0 |
| Execution (1 rollout) | 49.7 ± 1.9 | 62.1 ± 42.9 |
| SkillOpt-Lite | 48.6 ± 2.6 | 59.4 ± 46.2 |
| GEPA | 48.6 ± 2.6 | 31.6 ± 14.5 |
| LLM judge (pointwise) | 45.4 ± 4.5 | 39.9 ± 17.5 |
| LLM judge (pairwise) | 49.0 ± 2.2 | 57.6 ± 23.3 |
| LLM judge (ACE) | 46.9 ± 3.5 | 39.0 ± 15.6 |
The rank correlation between single execution score and posterior quality is 0.054 (essentially zero). No selector benefits from more candidates—all remain flat around random baseline even at .
P3: Errors compound across rounds. In-loop validation gains grow to 6.9–9.0 pp, but independently re-evaluated gains remain between +0.4 and −3.2 pp. Run-to-run variance grows from to 4.2–24.5 squared pp by the final round, matching the margins that separate methods in the literature.
Theoretical and Practical Implications
Theoretical Significance
-
Paradigm shift: The field should move from designing ever more candidate improvement systems to making true improvements detectable and accumulable. The proposer determines what improvements are available; evaluation resolution determines which improvements can be retained.
-
Unified explanation: The same variance that prevents single-step selection from identifying improvement also destabilizes method rankings, compounded over rounds. Both failures trace to a single constraint: evaluation resolution.
-
Shortcut exploitation: Large reported gains are consistent with the hypothesis "from the other side"—they exceed resolution by exploiting evaluation shortcuts that lift nearly every task at once.
Practical Recommendations
Candidate level — finer-grained feedback: Supplement terminal pass/fail with programmatic checks of intermediate behavior (e.g., unmasked failure diagnostics on verification commands, whether agent runs tests after editing). Preliminary analysis shows these checks exhibit 13×–57× lower evaluation noise than terminal verdicts.
Method level — component-wise evaluation:
- Proposers: Quality@K curves measure best-candidate quality among proposals
- Selectors: Selected-artifact quality curves measure quality recovered from a fixed pool as budget grows
- Reporting: Publish both (evaluation noise) and (run-to-run variance) alongside point scores so readers can judge whether a claimed margin exceeds resolution
Conclusion
The paper proposes and validates the Resolution Hypothesis: agent evolution is bottlenecked by identifying helpful artifacts, not proposing them. Evidence spans both levels where evaluation scores are used:
- Method level: Published margins (~3.4 pp) are comparable to single-run fluctuation (1.7–4.1 pp SD), making rankings noise-dominated
- Candidate level: Helpful artifacts exist, but all selectors perform at chance; per-round errors compound into run-to-run variance
- Shortcut gains: The largest reported gains exploit evaluation shortcuts and vanish after repair
Future directions:
- Resolution-aware evaluations: Less noisy intermediate process metrics
- Component-wise diagnostics: Separate proposer and selector evaluation
- Variance reporting: Standard practice of reporting and
- Program evolution: Deterministic code evolution (GPU kernels, ARC-AGI-2) may structurally reduce noise through cheap parallel re-evaluation—an open question the framework can address
Limitations
- Frozen weights only: RL-based weight updates may face different resolution constraints
- Candidate pool scope: Terminal-Bench v2.1 only; larger pools and other domains untested
- Model strength: Stronger future models may exhibit lower noise, though Terminal-Bench median stays at 3.5–4.0 pp even for the strongest models tested
- Grid coverage: Seven methods and six models is broad but not exhaustive
Related papers
- RIPPLE: Recovery Internalization through Process-Level Policy Learning and Effect-Aware Credit for Long-Horizon Agentic Coding
RIPPLE internalizes prefix-only recovery guidance into a single policy via stage-specific credit, improving coding agents by 4.8 points across four benchmarks without deployment-time critics.
- Shared Weights, Selected Computations: How Looped Transformers Route What Each Loop Does
Looped Transformers perform state-dependent computation, not fixed one-loop-one-step algorithms, and a learned linear layer can steer frozen shared weights via attention routing.
- Inside the Loop: A Mechanistic Study of Looped Transformers on Permutation Traversal
Looped transformers solve sequential algorithmic tasks untied models of equal effective depth cannot, by executing one algorithm step per iteration without intermediate supervision.