# Higher-Scoring Coding Models Did Not Get More Work Done

> Benchmark gains on coding tasks did not translate into measurable real-world productivity gains, as developers using higher-scoring models showed no significant improvement in pull request merge rates.

- **Source:** [arXiv](https://arxiv.org/abs/2611.23216)
- **Published:** 2026-11-21
- **Permalink:** https://picx.dev/p/Q3tYNd
- **Whiteboard:** https://picx.dev/p/Q3tYNd/image

## Summary

## Summary (Overview)

- **Natural experiment design**: The study leverages a 19-day U.S. export control ban on "Claude Fable 5" (June 12–July 1, 2026) as a natural experiment, isolating the causal effect of model access on developer productivity while holding users and projects constant.
- **No significant productivity gains**: Having access to Fable 5, which scored 10.8 points higher on SWE-Bench Pro than the next-best model, increased the 7-day pull request (PR) merge rate by only 1.4 percentage points (95% CI: −1.4 to +4.3), which was not statistically significant.
- **Consistent null results across upgrades**: Across five model upgrades in 2026 (1,208 developers, 15,442 PRs), the pooled change in merge rate was −0.9 percentage points (95% CI: −1.9 to +0.2), with all five point estimates negative and none statistically significant.
- **No change in volume**: Developers did not merge more PRs per week after switching to newer models (+2.4%, 95% CI: −4.9% to +10.2%).
- **Merge behavior insight**: 95% of sampled PRs were merged by the author themselves, and 90% had no approval from any other account, suggesting merges reflect self-assessment of completion rather than external validation.

## Introduction and Theoretical Foundation

The paper addresses a fundamental gap in the evaluation of coding models: **benchmark scores are widely used to advertise and select models, but no evidence exists that higher scores translate into more real work completed**.

**Key theoretical concerns with benchmarks:**
- Benchmark tasks (e.g., SWE-Bench Pro) are derived from issues already solved in public repositories and are checked only against existing tests.
- Real work involves vague requirements, project-specific style conventions, and human review/acceptance.
- Maintainers do not use benchmark tests as standards for merging PRs.
- Benchmark scores lack confidence intervals, making it unclear if reported gains exceed chance.

**The identification problem**: Observational comparisons of productivity across models are confounded by:
- **Selection effects**: Early adopters of new models may be more experienced.
- **Task assignment**: Users may give harder tasks to stronger models, making them look less productive.
- **Repository composition**: Merging habits vary across repositories, and most agent PRs come from a few repositories.

The study exploits the export control ban as an exogenous intervention, where the people and projects stayed the same and only the model changed.

## Methodology

### Study Design

**Three research questions:**
1. Causal effect of Fable 5 access on PR merge rate (suspension analysis)
2. Association between model upgrades and merge rate/weekly merged PRs (upgrade analysis)
3. Descriptive: what does a merge mean?

**Suspension analysis**: PRs from the same Fable 5 users split into four phases:
- **P0**: Before release (May 26–Jun 9)
- **P1**: Fable 5 available (Jun 9–12)
- **P2**: Suspended (reference phase, Jun 12–Jul 1)
- **P3**: After restoration (Jul 2–15)

**Upgrade analysis**: Eight model upgrades within Opus and Sonnet families (Feb–Sep 2026), comparing PRs from developers who switched models in the week before vs. after each upgrade.

### Data Collection

- Used GitHub Search API to collect public PRs created with Claude Code (identified via "Generated with Claude Code" description and co-author lines naming the model).
- Inclusion criteria: (1) description names exactly one Claude model, (2) author is a user account (not a bot), (3) created within study windows.
- Model name verification: 96–99% match between PR description and commit co-author lines.

### Study Population

| Group | N (Developers) | Description |
|-------|----------------|-------------|
| Treated (suspension) | 297 | Opened ≥1 PR naming Fable 5 during P1 |
| Control (suspension) | 8,908 | Used Opus 4.8, never Fable 5 |
| Switched (upgrades) | 1,208 | Used old model before, new model after upgrade |
| Did not switch (upgrades) | 944 | Used another Claude model in both weeks |

### Outcomes

**Main outcomes:**
1. **7-day merge rate**: Share of PRs merged within 7 days of creation
2. **Weekly merged PRs**: Number of PRs a developer opened in a week that were merged within 7 days

**Secondary outcomes:** Merged within 1 hour, closed without merge, comments per PR, reverted within 30 days, failed automated checks.

### Statistical Models

**Suspension analysis** — difference-in-differences at PR level:

$$Y_{it} = \alpha_i + \gamma_t + \beta_0 P0_{it} + \beta_1 P1_{it} + \beta_3 P3_{it} + \varepsilon_{it}$$

Where:
- $Y_{it}$ = 1 if PR by developer $i$ on day $t$ was merged within 7 days
- $\alpha_i$ = developer fixed effect (adjusts for time-invariant developer differences)
- $\gamma_t$ = calendar-day fixed effect (adjusts for shared daily changes)
- $P0, P1, P3$ = indicators for treated group's PRs in each phase (P2 = suspension is reference)
- Main estimate: reversal contrast $(\beta_1 + \beta_3)/2$
- $\beta_0$ = placebo (pre-release vs. suspension)
- Standard errors clustered by developer

**Upgrade analysis**:

$$Y_{ip} = \alpha_i + \delta \cdot \text{New}_{ip} + \varepsilon_{ip}$$

Where $\text{New}_{ip}$ = 1 if PR $p$ was opened with the new model in the week after upgrade; $\delta$ = change in merge rate; five upgrades pooled with inverse-variance weighting.

## Empirical Validation / Results

### Baseline Characteristics

Pre-Fable 5 release, treated and control groups were highly similar (all standardized differences between −0.18 and +0.15). In the upgrade analysis, switchers opened more PRs in others' repositories than non-switchers (23.5% vs. 11.7%).

### Compliance with the Ban

- During P1, 82.3% of treated PRs named Fable 5
- During P2 (suspension), only 0.9% named Fable 5; 74.6% named Opus 4.8
- After restoration (P3), only 26.9% named Fable 5 again
- Type of work did not change meaningfully across phases

### Main Results

**Table 2. Merge outcomes and check failures did not change with access to Fable 5 or after switching to a newer model.**

| Outcome | Suspension analysis | Upgrade analysis |
|---------|---------------------|------------------|
| **Main outcomes** | | |
| Merged within 7 days (pp) | +1.4 (−1.4, +4.3) | −0.9 (−1.9, +0.2) |
| Weekly merged PRs (%) | not estimated | +2.4 (−4.9, +10.2) |
| **Secondary outcomes** | | |
| Merged within 1 hour (pp) | 0.0 (−5.8, +5.8) | +0.8 (−2.1, +3.8) |
| Closed without merge (pp) | −0.5 (−3.0, +2.1) | +0.5 (−1.3, +2.3) |
| Comments per PR | not estimated | +0.03 (−0.06, +0.12) |
| Reverted within 30 days (pp) | 0.0 (−0.4, +0.4) | too few reverts |
| Failed automated check (pp) | – | +0.3 (−1.4, +2.0) |
| Failed test or build check (pp) | – | +0.2 (−1.6, +1.9) |
| **Checks on the merge rate** | | |
| Placebo: before release vs. suspension | −2.6 (−6.2, +1.1) | – |
| Within developer and repository | +1.4 (−1.4, +4.3) | −1.2 (−2.3, −0.02) |
| Refit on all PRs | +0.5 (−2.3, +3.3) | – |
| Time to merge among merged PRs (%) | +8.4 (−9.0, +29.2) | +2.4 (−2.1, +7.2) |

### Key Findings

1. **Suspension analysis**: The reversal contrast was +1.4 percentage points (95% CI: −1.4 to +4.3), not statistically significant. The placebo was −2.6 (−6.2 to +1.1). Only one of many sensitivity settings (equal weight per developer on all PRs) gave a significant result (+3.6, CI: +0.1 to +7.1).

2. **Upgrade analysis**: Pooled change in merge rate was −0.9 percentage points (95% CI: −1.9 to +0.2). Within developer and repository, the change was −1.2 (−2.3 to −0.02), which was statistically significant (i.e., merge rate *fell*). All five point estimates were negative.

3. **Benchmark vs. real-work gap**: Newer models scored 5–18 points higher on SWE-Bench Pro, but none of the merge rate changes were significant, and five of six point estimates were negative.

4. **Merge behavior**: 95% of sampled PRs were merged by the author; 90% had no approval from others. Median time to merge was under 30 minutes. Before upgrades, 94.5% of PRs by switchers were merged within 7 days, leaving little room for improvement.

## Theoretical and Practical Implications

### Why Benchmark Gains Did Not Translate

The authors hypothesize three main reasons:

1. **Task selection**: Developers give agents tasks that are already within the older model's capability. A survey of 132 Anthropic engineers found they mostly give Claude tasks that are easy to check and cheap to get wrong; more than half could fully hand off only 0–20% of their work.

2. **Benchmark limitations**: SWE-Bench Pro counts a task as solved if the change passes tests from the original fix, but these tests have poor quality and only test a small portion of an engineer's work (e.g., not clarifying vague requests or revising after review).

3. **Ceiling effects**: Since 94.5% of PRs were already merged before upgrades, there was little room for improvement in merge rate.

### Implications for Model Selection and Regulation

- Benchmark scores alone provide little evidence for model selection in real work contexts.
- Regulators and buyers who rely on benchmark differences (e.g., for export control decisions) may be overestimating the productivity impact of these differences.
- The findings align with prior work warning that benchmark scores mean little without evidence they measure the work they describe.

### Caveats and Limitations

- Merge rate is an imperfect proxy for productivity (almost all merges are self-merges).
- The suspension lasted only 19 days, yielding wide confidence intervals (though the upgrade analysis corroborates with narrower intervals).
- Only ~1% of Claude Code PRs name the model in the description; however, these PRs were merged about as often as others.
- The upgrade analysis is observational; developers who switch models may be better at using new tools (which would bias estimates upward, yet no increase was found).

## Conclusion

The study provides the first causal estimate of whether higher-scoring coding models increase real-work productivity. Using a natural experiment (the 19-day Fable 5 suspension), the authors found that:

- **Access to Fable 5** (11 points higher on SWE-Bench Pro) increased the 7-day merge rate by only 1.4 percentage points (not statistically significant).
- **Model upgrades** (5–18 SWE-Bench Pro points) did not increase merge rates or weekly merged PRs; if anything, within developer-repository comparisons showed a slight *decrease*.
- **Merges are largely self-validations**: 95% of PRs were merged by authors without external review, suggesting merges reflect the developer's own judgment of completion.

**Future directions**: The authors suggest that measuring time saved would require randomized trials, though such trials face challenges (e.g., participants knowing they're studied, not using assigned tools, working on study-assigned rather than real tasks). The results caution against using benchmark scores as the primary evidence for model superiority in real-world coding productivity.

---

_Markdown view of https://picx.dev/p/Q3tYNd, served by PicX — AI-generated visual whiteboard summaries of research papers._
