# Agent Benchmarks Stop Ranking Before They Run Out of Tasks

> Agent benchmarks saturate in the measurement, not the tasks, because the single pass/fail bit is too coarse and capped at the reference, so a clipped log-ratio score with dated anchors and separate cost restores ranking.

- **Source:** [arXiv](https://arxiv.org/abs/2611.13541)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/oxjN6X
- **Whiteboard:** https://picx.dev/p/oxjN6X/image

## Summary

## Summary (Overview)

- **Core finding**: Agent benchmarks saturate in the measurement, not the tasks. The single pass/fail bit used to grade most agent benchmarks is simultaneously too coarse to separate leading agents and capped at the reference, preventing further ranking.
- **Two failures identified**: Failure A (the bit is too coarse) and Failure B (the bit is capped at the reference) both stem from the design choice where an answer key encodes an answer rather than a preference.
- **Key contribution**: Derivation of a clipped log-ratio score with dated anchors and cost as a separate axis, which resolves top-tier agents that pass/fail grading ties together.
- **Empirical evidence**: Across eight benchmarks including SWE-bench Verified, Terminal-Bench, and ALE-Bench, the authors demonstrate that neighboring top-ten agents differ by less than the tests' errors and that graded credit restores ranking where pass/fail fails.
- **Design proposal**: A layered evaluation methodology separating acceptance (gate), quality (objective), and cost (meter), with frozen anchors ensuring reproducibility and comparability across versions.

## Introduction and Theoretical Foundation

The paper addresses a critical problem in AI agent evaluation: **benchmark saturation**. Data-science and ML-engineering agents are evaluated on benchmarks like MLE-bench, ALE-Bench, SWE-ficiency, and AlgoTune, where tasks yield a quantity (Kaggle metric, contest score, or speedup). As models improve, leaderboards stop separating the best models, and the usual remedy is to create harder versions.

The authors argue this saturation lies largely in the **measurement design**, not the tasks themselves. Almost every agent benchmark grades a run against an answer key (reference solution, expected end state, or tests), which returns one bit. This key is:
- The costliest and least verifiable part of a benchmark
- Often wrong (OSWorld reports twice as much effort on the evaluator as on the task)
- Vulnerable to LLM judge failures (a single token can fool them)

**Theoretical foundation**: The key insight is that an answer key encodes an answer rather than a preference. An answer must be written for every task and stops informing once agents reach it. A preference ("faster is better") can be scored once per domain, with room above any reference. This mirrors the transition made in reinforcement learning when agents passed humans, and in optimization benchmarking.

The paper identifies two distinct failure modes:

**Failure A (too coarse)**: The bit cannot distinguish neighboring top agents because:
- Statistical error from finite task sets
- Verifier error from keys that accept wrong solutions or reject right ones

**Failure B (capped)**: A run that beats the reference scores the same as one that just passes it, discarding information above the reference.

## Methodology

### Tier Analysis

The authors group agents into tiers using a **step-down max-T test** on paired t-statistics with a null from permuting agent labels within tasks. This bounds the chance of any false split at 5%. They analyzed:
- SWE-bench Verified: per-instance verdicts for official top 30, with SWE-ABS re-checking passing patches
- Terminal-Bench 2.0: per-trial results of 61 submissions (after removing 3 flagged for cheating)
- OSWorld-Verified: top ten agents, seven with per-task trajectories

### Scoring Comparisons

For benchmarks with released per-run, per-task continuous results, the authors score the same runs three ways:
1. **Answer key**: official binary verdict, or whether the run reaches the reference
2. **Capped score**: partial credit up to the reference, none beyond
3. **Objective**: 0 for runs failing validity checks, 1 at the reference, above 1 for runs beating it (clipped at 3)

### Saturation Index

They use a paired variant of the saturation index from prior work:

$$S = \exp(-R^2)$$

where $R$ is the mean gap between the first and fifth model over the standard deviation of their per-task differences, times $n^{1/4}$ for $n$ tasks. Near 1: top five are within noise; near 0: clearly separated.

### Controlled Re-run

The authors re-ran 13 Terminal-Bench tasks with tests modified only to print the quantity they compute (e.g., runtime), running four models (gpt-5.3-codex, claude-opus-5, gpt-6-astra, gemini-3.1-pro) with three trials per task in the Terminus-2 agent (156 runs, ~$115 at list prices).

## Empirical Validation / Results

### Failure A: The Bit Is Too Coarse

**Statistical ties at the top**: In September 2024, SWE-bench Verified's top ten spanned 19.2 points and three tiers; since March 2025 they form one or two tiers (today 3.6 points, one tier). Terminal-Bench 2.0's top ten form one tier; OSWorld-Verified's top ten form two.

None of the 45 pairs within the top ten on SWE-bench Verified and Terminal-Bench 2.0 differ significantly. The test is not weak: it detects about 6.3 points with 80% power, but the largest gap between SWE-bench neighbors is 1.4 points.

**Verifier errors exceed gaps**: Strengthened tests reject 17.7% of the median agent's passing patches on SWE-bench (range 10.8–23.2%). Correcting verdicts shrinks the top tier from 23 agents to 5, and reverses 8 of 18 neighboring orders (Kendall τ = 0.51 between the two leaderboards). Removing the same share of passes at random reverses about as many (8; 95% band 4–11), showing the order of neighbors is essentially noise.

### Failure B: The Bit Is Capped at the Reference

**Table 1** shows that scoring the same runs with room above the cap restores ranking where it binds:

| Benchmark | Models | Tasks | Beat ref. | Official | Answer key | Capped | Objective |
|-----------|--------|-------|-----------|----------|------------|--------|-----------|
| ALE-Bench | 145 | 40 | 47.8% | 3 (0.07) | 1 (1.00) | 1 (0.93) | 3 (0.03) |
| SWE-efficiency | 7 | 430 | 36.3% | 1 (0.95) | 4 (0.10) | 5 (0.01) | 4 (0.02) |
| AlgoTune | 18 | 154 | 70.6% | 2 (0.51) | 2 (0.63) | 2 (0.88) | 3 (0.58) |
| METR† | 20 | 210 | 3.7% | = key | 3 (0.51) | 3 (0.55) | n/a* |
| MLE-bench† | 27 | 74 | 10.8% | = key | 4 (0.10) | 3 (0.52) | 3 (0.73) |

ALE-Bench's official score separates its top ten into three tiers, but scored as an answer key against the median human, the same runs give one tier (S = 1.00). The objective restores three tiers (S = 0.03).

### Why Graded Credit, Not a Higher Bar, Fixes It

On ALE-Bench (141 models split into 46 released by end of 2025 and 95 later), the authors test the "harder bar" remedy:

- **Median-human bar** (the key): ties the later top ten
- **Contest-winner bar**: separates later models into two tiers but scores 89% of earlier models zero
- **Bar tuned on earlier models**: puts later top ten in one tier

Each bar resolves one band of ability; raising it moves the band instead of widening it. Graded scores (clipped log-ratio, percentiles) resolve both groups.

### Derivation of the Clipped Log-Ratio Score

For a positive quantity $J$, the score is derived rather than chosen: a score should not depend on the unit of $J$, so it is a function $f$ of $x = J / J_{base}$, and credit should add under composition, so $f(xy) = f(x) + f(y)$. The only continuous solutions of this Cauchy equation are $f(x) = k \log x$. Dividing by $\log(J_{ref}/J_{base})$ sets $k$ per task so the reference scores 1.

### Terminal-Bench Re-run Results

- **Pass/fail ties** opus, astra, and gemini; separates codex
- **Tied models differ 10.9×** on the quantity: on portfolio-optimization, opus's median speedup is 10.9× while others achieve 1.3× (the reference's level)
- **69% of passing runs** beat the reference solution
- **Cost varies 7.3×** in median tokens per run among tied top-ten agents

## Theoretical and Practical Implications

### The Clipped Log-Ratio Score

For positive quantities (runtime, contest score):

$$s(x) = G(x) \cdot \min\left(3, \max\left(0, \frac{\log(J(x) / J_{\text{base}})}{\log(J_{\text{ref}} / J_{\text{base}})}\right)\right)$$

The clip at 3 bounds any single task's share of the score to $3/n$ for $n$ tasks, preventing weak references from dominating. The design requires:
- A strong reference near the best known solution
- Exclusion of tasks whose reference barely beats the baseline
- A clip that binds often signals a weak reference to be replaced

### Layered Benchmark Design

The paper proposes separating roles that answer keys merge (Figure 4):

1. **Intent**: What the human wants (hard constraints + direction of preference), never a procedure
2. **Gate** $G(x) \in \{0,1\}$: Deterministic, hardened acceptance check (behavioral, physics, safety, anti-tamper)
3. **Objective** $s(x)$: Normalizes raw quantity $J(x)$ to dated anchors (0 at baseline, 1 at reference)
4. **Meter** $c(h)$: Records dollars at dated price table, tokens, steps, time — reported separately, never blended into the score
5. **Anchors**: Frozen with a date so benchmark versions share one scale; reference artifacts re-measured on the run machine as ratios

### Practical Implications

- **LLM judges** may draft gate and objective code at authoring time with human audit, but do not score at run time
- **Results reported lexicographically**: gate rate with statistical tiers, then mean of $s$
- **Cost as separate axis**: reported jointly with quality and its Pareto front; any claimed lead states the interval of exchange rates λ over which it holds
- **A composite would rank cost, not quality**: on ALE-Bench its τ with the human rating is −0.35 in the later cohort

## Conclusion

Agent benchmarks often saturate in the score, not the tasks. The single pass/fail bit is too coarse to separate leading agents and capped at its author. The authors recommend benchmarks should build in:
- An **anchored objective** with room above the reference
- A **separate cost meter**
- **Frozen anchors** that keep versions reproducible and comparable

A saturated key should end a version, not the measurement. The design was instantiated in a small re-run showing that pass/fail-tied models differ over tenfold on the quantity the tests discard.

**Limitations**: Per-agent verifier error exists only for SWE-bench Verified; the sliding-window result rests on ALE-Bench with partly reconstructed dates; the re-run's aggregate separation is borderline (three trials); rubric-graded deliverables like GDPval have neither a gate nor a scalar objective.

**Future directions**: Hardening gates against gaming, extending the methodology to rubric-graded deliverables, and broader validation of the anchored objective across more benchmark domains.

---

_Markdown view of https://picx.dev/p/oxjN6X, served by PicX — AI-generated visual whiteboard summaries of research papers._