Full text not available for this paper

Summary (Overview)

  • Natural experiment design: The study leverages a 19-day U.S. export control ban on "Claude Fable 5" (June 12–July 1, 2026) as a natural experiment, isolating the causal effect of model access on developer productivity while holding users and projects constant.
  • No significant productivity gains: Having access to Fable 5, which scored 10.8 points higher on SWE-Bench Pro than the next-best model, increased the 7-day pull request (PR) merge rate by only 1.4 percentage points (95% CI: −1.4 to +4.3), which was not statistically significant.
  • Consistent null results across upgrades: Across five model upgrades in 2026 (1,208 developers, 15,442 PRs), the pooled change in merge rate was −0.9 percentage points (95% CI: −1.9 to +0.2), with all five point estimates negative and none statistically significant.
  • No change in volume: Developers did not merge more PRs per week after switching to newer models (+2.4%, 95% CI: −4.9% to +10.2%).
  • Merge behavior insight: 95% of sampled PRs were merged by the author themselves, and 90% had no approval from any other account, suggesting merges reflect self-assessment of completion rather than external validation.

Introduction and Theoretical Foundation

The paper addresses a fundamental gap in the evaluation of coding models: benchmark scores are widely used to advertise and select models, but no evidence exists that higher scores translate into more real work completed.

Key theoretical concerns with benchmarks:

  • Benchmark tasks (e.g., SWE-Bench Pro) are derived from issues already solved in public repositories and are checked only against existing tests.
  • Real work involves vague requirements, project-specific style conventions, and human review/acceptance.
  • Maintainers do not use benchmark tests as standards for merging PRs.
  • Benchmark scores lack confidence intervals, making it unclear if reported gains exceed chance.

The identification problem: Observational comparisons of productivity across models are confounded by:

  • Selection effects: Early adopters of new models may be more experienced.
  • Task assignment: Users may give harder tasks to stronger models, making them look less productive.
  • Repository composition: Merging habits vary across repositories, and most agent PRs come from a few repositories.

The study exploits the export control ban as an exogenous intervention, where the people and projects stayed the same and only the model changed.

Methodology

Study Design

Three research questions:

  1. Causal effect of Fable 5 access on PR merge rate (suspension analysis)
  2. Association between model upgrades and merge rate/weekly merged PRs (upgrade analysis)
  3. Descriptive: what does a merge mean?

Suspension analysis: PRs from the same Fable 5 users split into four phases:

  • P0: Before release (May 26–Jun 9)
  • P1: Fable 5 available (Jun 9–12)
  • P2: Suspended (reference phase, Jun 12–Jul 1)
  • P3: After restoration (Jul 2–15)

Upgrade analysis: Eight model upgrades within Opus and Sonnet families (Feb–Sep 2026), comparing PRs from developers who switched models in the week before vs. after each upgrade.

Data Collection

  • Used GitHub Search API to collect public PRs created with Claude Code (identified via "Generated with Claude Code" description and co-author lines naming the model).
  • Inclusion criteria: (1) description names exactly one Claude model, (2) author is a user account (not a bot), (3) created within study windows.
  • Model name verification: 96–99% match between PR description and commit co-author lines.

Study Population

GroupN (Developers)Description
Treated (suspension)297Opened ≥1 PR naming Fable 5 during P1
Control (suspension)8,908Used Opus 4.8, never Fable 5
Switched (upgrades)1,208Used old model before, new model after upgrade
Did not switch (upgrades)944Used another Claude model in both weeks

Outcomes

Main outcomes:

  1. 7-day merge rate: Share of PRs merged within 7 days of creation
  2. Weekly merged PRs: Number of PRs a developer opened in a week that were merged within 7 days

Secondary outcomes: Merged within 1 hour, closed without merge, comments per PR, reverted within 30 days, failed automated checks.

Statistical Models

Suspension analysis — difference-in-differences at PR level:

Yit=αi+γt+β0P0it+β1P1it+β3P3it+εitY_{it} = \alpha_i + \gamma_t + \beta_0 P0_{it} + \beta_1 P1_{it} + \beta_3 P3_{it} + \varepsilon_{it}

Where:

  • YitY_{it} = 1 if PR by developer ii on day tt was merged within 7 days
  • αi\alpha_i = developer fixed effect (adjusts for time-invariant developer differences)
  • γt\gamma_t = calendar-day fixed effect (adjusts for shared daily changes)
  • P0,P1,P3P0, P1, P3 = indicators for treated group's PRs in each phase (P2 = suspension is reference)
  • Main estimate: reversal contrast (β1+β3)/2(\beta_1 + \beta_3)/2
  • β0\beta_0 = placebo (pre-release vs. suspension)
  • Standard errors clustered by developer

Upgrade analysis:

Yip=αi+δ⋅Newip+εipY_{ip} = \alpha_i + \delta \cdot \text{New}_{ip} + \varepsilon_{ip}

Where Newip\text{New}_{ip} = 1 if PR pp was opened with the new model in the week after upgrade; δ\delta = change in merge rate; five upgrades pooled with inverse-variance weighting.

Empirical Validation / Results

Baseline Characteristics

Pre-Fable 5 release, treated and control groups were highly similar (all standardized differences between −0.18 and +0.15). In the upgrade analysis, switchers opened more PRs in others' repositories than non-switchers (23.5% vs. 11.7%).

Compliance with the Ban

  • During P1, 82.3% of treated PRs named Fable 5
  • During P2 (suspension), only 0.9% named Fable 5; 74.6% named Opus 4.8
  • After restoration (P3), only 26.9% named Fable 5 again
  • Type of work did not change meaningfully across phases

Main Results

Table 2. Merge outcomes and check failures did not change with access to Fable 5 or after switching to a newer model.

OutcomeSuspension analysisUpgrade analysis
Main outcomes
Merged within 7 days (pp)+1.4 (−1.4, +4.3)−0.9 (−1.9, +0.2)
Weekly merged PRs (%)not estimated+2.4 (−4.9, +10.2)
Secondary outcomes
Merged within 1 hour (pp)0.0 (−5.8, +5.8)+0.8 (−2.1, +3.8)
Closed without merge (pp)−0.5 (−3.0, +2.1)+0.5 (−1.3, +2.3)
Comments per PRnot estimated+0.03 (−0.06, +0.12)
Reverted within 30 days (pp)0.0 (−0.4, +0.4)too few reverts
Failed automated check (pp)–+0.3 (−1.4, +2.0)
Failed test or build check (pp)–+0.2 (−1.6, +1.9)
Checks on the merge rate
Placebo: before release vs. suspension−2.6 (−6.2, +1.1)–
Within developer and repository+1.4 (−1.4, +4.3)−1.2 (−2.3, −0.02)
Refit on all PRs+0.5 (−2.3, +3.3)–
Time to merge among merged PRs (%)+8.4 (−9.0, +29.2)+2.4 (−2.1, +7.2)

Key Findings

  1. Suspension analysis: The reversal contrast was +1.4 percentage points (95% CI: −1.4 to +4.3), not statistically significant. The placebo was −2.6 (−6.2 to +1.1). Only one of many sensitivity settings (equal weight per developer on all PRs) gave a significant result (+3.6, CI: +0.1 to +7.1).

  2. Upgrade analysis: Pooled change in merge rate was −0.9 percentage points (95% CI: −1.9 to +0.2). Within developer and repository, the change was −1.2 (−2.3 to −0.02), which was statistically significant (i.e., merge rate fell). All five point estimates were negative.

  3. Benchmark vs. real-work gap: Newer models scored 5–18 points higher on SWE-Bench Pro, but none of the merge rate changes were significant, and five of six point estimates were negative.

  4. Merge behavior: 95% of sampled PRs were merged by the author; 90% had no approval from others. Median time to merge was under 30 minutes. Before upgrades, 94.5% of PRs by switchers were merged within 7 days, leaving little room for improvement.

Theoretical and Practical Implications

Why Benchmark Gains Did Not Translate

The authors hypothesize three main reasons:

  1. Task selection: Developers give agents tasks that are already within the older model's capability. A survey of 132 Anthropic engineers found they mostly give Claude tasks that are easy to check and cheap to get wrong; more than half could fully hand off only 0–20% of their work.

  2. Benchmark limitations: SWE-Bench Pro counts a task as solved if the change passes tests from the original fix, but these tests have poor quality and only test a small portion of an engineer's work (e.g., not clarifying vague requests or revising after review).

  3. Ceiling effects: Since 94.5% of PRs were already merged before upgrades, there was little room for improvement in merge rate.

Implications for Model Selection and Regulation

  • Benchmark scores alone provide little evidence for model selection in real work contexts.
  • Regulators and buyers who rely on benchmark differences (e.g., for export control decisions) may be overestimating the productivity impact of these differences.
  • The findings align with prior work warning that benchmark scores mean little without evidence they measure the work they describe.

Caveats and Limitations

  • Merge rate is an imperfect proxy for productivity (almost all merges are self-merges).
  • The suspension lasted only 19 days, yielding wide confidence intervals (though the upgrade analysis corroborates with narrower intervals).
  • Only ~1% of Claude Code PRs name the model in the description; however, these PRs were merged about as often as others.
  • The upgrade analysis is observational; developers who switch models may be better at using new tools (which would bias estimates upward, yet no increase was found).

Conclusion

The study provides the first causal estimate of whether higher-scoring coding models increase real-work productivity. Using a natural experiment (the 19-day Fable 5 suspension), the authors found that:

  • Access to Fable 5 (11 points higher on SWE-Bench Pro) increased the 7-day merge rate by only 1.4 percentage points (not statistically significant).
  • Model upgrades (5–18 SWE-Bench Pro points) did not increase merge rates or weekly merged PRs; if anything, within developer-repository comparisons showed a slight decrease.
  • Merges are largely self-validations: 95% of PRs were merged by authors without external review, suggesting merges reflect the developer's own judgment of completion.

Future directions: The authors suggest that measuring time saved would require randomized trials, though such trials face challenges (e.g., participants knowing they're studied, not using assigned tools, working on study-assigned rather than real tasks). The results caution against using benchmark scores as the primary evidence for model superiority in real-world coding productivity.

Related papers