# Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

> SWE-bench Verified's top entries are statistically indistinguishable, with nested solution sets and zero separable adjacent pairs, so leaderboard ranks no longer reflect meaningful ordering.

- **Source:** [arXiv](https://arxiv.org/abs/2609.17394)
- **Published:** 2026-09-19
- **Permalink:** https://picx.dev/p/zfk4fl
- **Whiteboard:** https://picx.dev/p/zfk4fl/image

## Summary

# Coding Agents Have Converged: Why the SWE-bench Leaderboard Can No Longer Order Its Top Entries, and What to Measure Instead

## Summary (Overview)

- **Core finding**: The SWE-bench Verified leaderboard can no longer meaningfully order its top entries—the top 10 systems solve the same 285 of 500 instances and fail the same 51, leaving only 164 instances (33%) that distinguish them.
- **Convergence mechanism**: Frontier solution sets exhibit median nesting of 0.935 against a score-implied baseline of 0.774, meaning weaker systems' successes are almost entirely subsets of stronger systems' successes.
- **Model-scaffold entanglement**: Observed within-model scaffold ranges reach 29.8 percentage points—more than triple the 8.8-point spread of the top thirty submissions—and scaffold orderings reverse across models.
- **Statistical non-separability**: Exact paired McNemar tests separate **none** of the 29 adjacent top-thirty pairs on Verified at α = 0.05, yielding only descriptive tiers (3 tiers uncorrected, 2 after Holm correction).
- **Key contribution**: A five-step audit protocol (Protocol 1) that mines per-instance verdict matrices to profile degeneracy, test paired differences, and estimate instance budgets—released as an open-source reproducibility package.

---

## Introduction and Theoretical Foundation

### Motivation

The paper addresses a fundamental question in AI evaluation: **when can leaderboard ranks be read as meaningful orderings?** SWE-bench Verified evaluates coding agents on 500 real GitHub issues, and the leaderboard displays ranks 1–8 that are "often read as an ordering for model selection, procurement, and research reporting." However, the top two entries each resolve 396 instances, and six more sit within 14 of them.

### Distinguishing from Prior Work

The authors explicitly differentiate their contribution from existing critiques:

> "One line of work asks whether the tasks are sound... We ask the other question: even where the tasks are sound, can the ranks be read?"

Prior work on task quality (automated auditing finding quality problems in over 25% of tasks, shifting rankings by up to 9.9%) is **complementary**—the authors focus on whether the comparison structure itself supports rank ordering.

### Theoretical Foundations

The work builds on established statistical machinery:

- **Test collection reliability** in information retrieval [14, 15, 12]
- **Paired adjacent-rank testing** and required-sample-size inversion [5]
- **Rank intervals** [7]
- **Benchmark power analysis** [2]
- **Reduced-size construction** [11, 10]
- **Item-level psychometrics** [19]

### Novel Constructs

Two constructs are new to this work:

1. **Comparison-set-relative effective size** $n_{eff}(S)$
2. **Nesting coefficient against its score-implied baseline**

---

## Methodology

### Data Pool

The analysis uses **254 SWE-bench submissions across four splits**, requiring no model runs—all verdicts come from the published leaderboard (retrieved 30 July 2026):

| Split | Submissions | Instances | Verified overlap | Role |
|-------|-------------|-----------|------------------|------|
| Verified | 134 | 500 | — | main analysis |
| Lite | 84 | 299 | 93 (31%) | indep. replication |
| Test | 24 | 2,294 | — | counter-test |
| Multimodal | 12 of 22 | 301 | — | descriptive |

### Key Definitions

**Effective size** (Eq. 1)—the number of instances that can contribute to comparisons within a set $S$:

$$n_{eff}(S) = \left| \left\{i: 0 < \operatorname{resolved}_{S}(i) < |S| \right\} \right|$$

**Nesting coefficient** (Eq. 2)—fraction of the weaker system's solutions that the stronger one also produces:

$$\operatorname{cov}(A, B) = \frac{|A \cap B|}{|B|}, \qquad \mathbb{E}[\operatorname{cov}] = \frac{|A|}{n}$$

### Statistical Tests

- **Exact paired McNemar tests** for adjacent rank pairs
- **Bootstrap paired difference-in-differences** (100,000 resamples) for model×scaffold interactions
- **Holm-Bonferroni correction** for multiple testing
- **Two-parameter logistic IRT model** fitted to the 134×468 non-constant response matrix

### The Five-Step Audit Protocol (Protocol 1)

1. **Profile**: Compute $n_{eff}(S)$ over nested comparison sets
2. **Explain**: Nesting coefficient against score-implied null
3. **Test**: Exact paired McNemar for every pair
4. **Partition**: Walk sorted ranking, start new tier at $p < 0.05$
5. **Budget**: Invert paired condition to price new instances

---

## Empirical Validation / Results

### 1. Degeneracy Collapse

As the comparison set narrows, the effective size collapses dramatically:

| K | solved by all | solved by none | n_eff/n | KR-20 |
|---|---------------|----------------|---------|-------|
| 2 | 378 | 86 | **0.07** | — |
| 10 | 285 | 51 | **0.33** | 0.475 |
| 20 | 250 | 41 | 0.42 | 0.722 |
| 134 | — | — | 0.94 | 0.994 |

The top two entries are separated by only **36 informative instances** (7% of the benchmark).

### 2. Nesting, Not Complementarity

| band (resolved instances) | systems | median cov | baseline | excess |
|---------------------------|---------|------------|----------|--------|
| frontier (≥ 370) | 16 | **0.935** | 0.774 | +0.161 |
| strong (300–369) | 38 | 0.913 | 0.708 | +0.205 |
| mid (200–299) | 48 | 0.828 | 0.530 | +0.298 |
| weak (100–199) | 23 | 0.712 | 0.370 | +0.342 |

Nesting exceeds baseline in every band, ruling out high scores as the explanation.

### 3. Model–Scaffold Entanglement

- **Within-model scaffold range**: median 78 instances (15.6pp); for claude-3-5-sonnet: **149 instances (29.8pp)** across 9 scaffolds
- **Top-thirty spread**: only 8.8pp
- **Replication floor**: same scaffold+model at different dates varies by median 27 instances (5.4pp)
- **Additive fit** $y = \mu + \alpha_{scaffold} + \beta_{model}$: $R^2 = 0.989$, scaffold-effect range 37.6pp vs. model range 47.9pp (ratio 0.78)

**Interactions are significant**: Six of nine cell-mean interaction tests remain significant after Holm correction. Scaffold orderings **reverse** across models (e.g., agentless vs. epam-ai-run changes sign between claude-3-5-sonnet and gpt-4o, interaction −75.5 instances, adjusted $p < 0.001$).

### 4. Paired Separability Across Splits

| split | n | systems | n_eff/n | spread | adj. pairs | separable |
|-------|---|---------|---------|--------|------------|-----------|
| Verified | 500 | 134 | 0.33 | 8.8 pp | 29 | **0** |
| Lite | 299 | 84 | 0.54 | 21.1 pp | 29 | **0** |
| Multimodal | 301 | 12 | 0.44 | 18.3 pp | 11 | **0** |
| Test | 2294 | 24 | 0.52 | 52.4 pp | 23 | **14** |

The Test split (where ability still spans a wide range) shows the same test **can** separate pairs—confirming irresolvability is a property of the converged comparison set, not the benchmark family.

### 5. Descriptive Tiers

Using uncorrected leader comparisons at α = 0.05:

| Tier | resolved instances (% of 500) | systems |
|------|-------------------------------|---------|
| 1 | 382–396 (76.4–79.2%) | 8 |
| 2 | 362–378 (72.4–75.6%) | 12 |
| 3 | 352–359 (70.4–71.8%) | 10 |

With Holm correction: **19/11 groups**. The smallest adjacent p-value is 0.545—far from any threshold.

### 6. Instance Budget

For the ten leading adjacent pairs (excluding two zero-gap pairs), the upper median multiplier is **52×**—roughly **26,000 instances** of the same character needed for separation. However, instances that disagree and lean one way change the picture dramatically: at an imbalance rate of 0.2, only ~900 instances are needed (vs. 26,000).

### 7. Item Response Theory Analysis

- Test information peaks at $\theta = -0.62$ (below median system)
- Instrument carries 2.3× more information about a median system ($I = 278$, SE = 0.060) than a top-ten system ($I = 118$, SE = 0.092)
- Counterintuitively, degenerate instances have **higher** mean discrimination (2.52 vs 1.82)—they separate across the full 2023–2025 range but not within the 2025 frontier

### 8. Ruling Out Alternative Explanations

- **Task defects**: Human effort estimates show degenerate instances are rated "easy" (49% of shared successes rated "< 15 min"), not broken—the opposite of what a defect account predicts
- **Contamination**: PR year vs. top-ten solve rate correlation is −0.069 (95% CI [−0.148, +0.012])—no detectable age effect

---

## Theoretical and Practical Implications

### For Benchmark Designers

1. **Report $n_{eff}$**—but don't expect retirement of degenerate instances to buy resolution (rescaling to 209 non-degenerate instances doubles visible spread but keeps separable pairs at 0/29)
2. **Add instances that break nesting**, not instances that add count. The curation criterion is the discordance/imbalance ratio, not a target task count
3. **Record the (model, scaffold, version) pair in machine-readable form**—54% of Verified submissions cannot currently be placed in a factorial design
4. **Publish tiers with the linkage rule**, not strict ranks

### For Organizations Building Internal Benchmarks

1. **Evaluate the deployed pair** (model + your harness), not a transferred external model ranking
2. **Size the benchmark for the decision**: at $n = 100$, independent-sample detectable difference is ~17pp; state the detectable gap before testing
3. **Track candidate-specific $n_{eff}$** when curating new tasks
4. **Grade from machine-readable artefacts**, never from the agent's own report
5. **Pre-register the scoring convention** for missing verdicts and report both
6. **Report tiers to decision-makers**, not ranked tables
7. **Prefer internal tasks** (free of pretraining overlap) and **re-mine periodically**

### Key Design Equation

The paired comparison budget (Eq. 3):

$$|b - c| \gtrsim 1.96\sqrt{b + c}, \quad k \geq 1.96^2 \frac{b + c}{(b - c)^2}$$

where $b$ and $c$ are the discordant instance counts, and $k$ is the scale factor needed at fixed discordance and imbalance rates.

---

## Conclusion

### Main Takeaways

> "Coding agents at the top of SWE-bench Verified have converged: they solve the same 285 of 500 instances, fail the same 51, and their solution sets are nested at 0.935 against a score-implied 0.774."

The number that separates top entries is **not a property of the model alone**. The leaderboard is "being read at a resolution it does not have."

### Limitations

- **Observational design**: Teams choose scaffold and model together; scaffold effects absorb co-optimization and are not causal
- **One run per submission**: Run-to-run variance is not captured
- **Contamination**: Verified overlaps pretraining data (76% buggy-file identification accuracy vs. 53% on unseen repos)
- **Scope**: One benchmark family, one harness, voluntarily submitted submissions

### Future Directions

- Controlled factorial experiments (same scaffolds × same models under one harness)
- Applying the protocol to multi-domain suites and internal benchmarks
- Validation on unrelated leaderboards

### Reproducibility

All inputs are publicly available per-instance verdicts; no model access or API keys required. The complete pipeline is released at: https://github.com/Adkid-Zephyr/resolution-audit

---

_Markdown view of https://picx.dev/p/zfk4fl, served by PicX — AI-generated visual whiteboard summaries of research papers._