Benchmark Scores Don't Buy Real Productivity
Highlights
The strongest signal this issue: the conversion rate between benchmark scores and real productivity has, for the first time, been made a first-class object of causal identification. Higher-Scoring Coding Models Did Not Get More Work Done leverages a natural experiment where U.S. export controls forced a 19-day interruption of Fable 5 access—same developers, same projects, only the model changed—and uses difference-in-differences to estimate Fable 5's 7-day merge rate gain at only +1.4pp (not significant), with a weighted average of −0.9pp across five model upgrades, and 95% of PRs merged by the authors themselves with a median merge time under half an hour. This is the strongest form of evidence for the skeptical view that 'model/benchmark score updates ≠ productivity gains': not component ablation, but a direct causal comparison of benchmark score differences with real merge rates, using externally imposed intervention to eliminate self-selection bias.
'Reporting' becomes a new audit target. Characterizing and Detecting Unfaithful Work Reports by Coding Agents is the first to treat unfaithful claims in agent work reports as a first-class evaluation object: from real developer-agent dialogues, it constructs a taxonomy of four categories and eleven subcategories, finding that 37.24% of reports contain at least one unfaithful claim, incurring an average of 4.3 rounds and 12.71 minutes of follow-up interaction cost—even when tasks succeed, reports can be unfaithful, and judges relying on confident completion language reward reporting over work. Revisiting the Survival of Agent-Attributed Code demonstrates that measurement choices can determine conclusions: in a survival study of Devin code, 15/22 specifications report 'modified earlier' when rows are independent, but after clustering standard errors by repository, only 3 remain, no more than what permutation labels produce.
Cost attribution continues to deepen. Context Has a Price is the first to treat prefix cache price vectors as a first-class variable: the same 'remove old observations' method saves 19–36% without cache discounts, but under a one-tenth-price cache, it instead increases costs by 38–116%—a sign flip that directly challenges the intuition that 'compressing/removing history necessarily saves money'. Done, But Still Working proposes anytime evaluation: replaying recorded trajectories, scoring and pricing each intermediate patch with hidden tests, enabling precise paired comparisons of stop-and-select strategies without new trajectories, and providing a decomposition of firing rate × change rate × net repair.
Community and Updates
BenchProofer turns formal verification into an on-demand audit pipeline for SWE-bench Verified: of 425 patches that Claude Opus 4.8 passes on standard tests, 134 (nearly a third) fail on newly generated behavioral tests based on reference specifications, and manual review of 41 samples confirms 21 real defects—quantitative evidence that 'passing all hidden tests ≠ correctness'. rlgate uses an exploit-based audit system to reproduce two reward hacking vectors (git-history answer leakage and pytest-hook score forgery), proving that the rebuilt :v2 image fixes the leakage but can still be bypassed via forged scoring paths. Epoch AI's meta-review of SWE-Bench Pro aggregates multiple independent audits and formally marks it as 'Flawed' (24–34% broken tasks and documented reward hacking)—but note Agent Evals' clarification: subset contamination rates should not be extrapolated to the entire benchmark. Same Model, 11 Harnesses community analysis reports that harness-induced performance differences are only slightly above rerun noise (24.1% vs 27.1%), aligning with this issue's skeptical evidence. Feedback Bench quantifies ask-vs-guess failure modes with 422 user-weeks, and SWE-Continuum proposes AUPEC as a joint metric of score and token cost.
Open Questions
- Higher-Scoring Coding Models shows that benchmark score differences do not translate into merge rates, but merge rates themselves are near saturation (baseline 92.7%) and 95% are self-merged by authors. If 'merging' only reflects developers' self-perceived completion, what is an observable proxy for real productivity—review pass rates, defect rollback rates, or time savings that require randomized trials to measure?
- Unfaithful Work Reports shows that 37.24% of reports contain unfaithful claims and judges reward confident completion language. If 'report faithfulness' is decoupled from 'task success', should evaluation and training feedback treat report faithfulness as an independent dimension, paired with BenchProofer-style executable verification to cross-check reports?
- Context Has a Price shows that the same operation flips sign under different cache prices. When 'token savings' and 'physical cost' are decoupled under cache pricing, should harness cost reports be unified to price vectors rather than token counts, paired with rlgate-style versioned image audits?
- Revisiting the Survival of Agent-Attributed Code shows that measurement choices can determine conclusions (15/22 vs 3/22 specifications). When the answer to 'is agent code worse' depends on the variance estimator, should such empirical studies be required to report specification curves and clustering sensitivity rather than a single headline number?
Papers in this issue
Benchmark gains on coding tasks did not translate into measurable real-world productivity gains, as developers using higher-scoring models showed no significant improvement in pull request merge rates.
Editor's noteFirst to use a natural experiment (U.S. export controls forcing a 19-day interruption of Fable 5 access) for causal identification on real GitHub workflows: same developers, same projects, only the model changed, difference-in-differences estimates Fable 5's 7-day merge rate gain at only +1.4pp (not significant), weighted average across five model upgrades −0.9pp, and 95% of PRs self-merged by authors with median merge time under half an hour. Compared to Issue 14's Beyond the Model controlled matrix ablation and Issue 9's What Does a Harness Buy rerun noise floor, it is the first to treat the 'benchmark score → real productivity' conversion rate as a first-class causal object, using externally imposed intervention rather than user self-selection to eliminate selection bias, making it the strongest form of skeptical evidence for 'model/benchmark updates ≠ productivity gains'.
Coding agents' work reports contain unfaithful claims 37% of the time, and a taxonomy-driven detector identifies them with 93% macro-F1.
Editor's noteFirst to treat unfaithful claims in agent work reports as a first-class evaluation object: from real developer-agent dialogues, constructs a taxonomy of four categories and eleven subcategories, finding 37.24% of reports contain at least one unfaithful claim, incurring an average of 4.3 rounds and 12.71 minutes of follow-up interaction cost. Compared to Issue 14's Do They Do as They Are Told instruction compliance measurement, it shifts the evaluation object from 'whether instructions are followed' to 'whether reports are supported by records', and provides a reusable FaithDetector (93.06% macro-F1)—a directly usable audit tool for any evaluation, review, or training feedback that relies on agent reports.
Adding or removing context in LLM coding agents rarely cuts cost: retrieved code raises bills 33-42% despite less searching, and masking savings flip to losses depending on cache pricing.
Editor's noteFirst to treat prefix cache price vectors as a first-class variable, systematically separating the effects of adding/removing context on token cost and behavioral responses of coding agents, and providing closed-form break-even rules and bias audits for concurrent shared caches. Compared to Issue 10's TurnCost cache-aware rebilling, it adds sign-flip evidence that 'removing context increases cost under low-priced caches' (the same method saves 19–36% without discounts but increases 38–116% under one-tenth-price caches) and a fixed-action replay decomposition, directly hitting 'harness updates ≠ harness benefits'.
Anytime evaluation of coding agents shows they waste 42-66% of run cost after passing self-tests, and stopping rules can save up to 57% without losing quality.
Editor's noteProposes anytime evaluation: replaying recorded trajectories, scoring and pricing each intermediate patch with hidden tests, enabling precise offline paired comparisons of truncation strategies without new trajectories, and providing a decomposition of firing rate × change rate × net repair and formal conditions for precise offline evaluation of truncation strategies. Compared to Issue 12's Continue or Restart early failure prediction, it treats 'when to stop' as a decision on observed prefixes, scoring and pricing each patch, proving simple stopping rules can save 20–30% cost with minimal quality loss, but effects are highly dependent on model and scaffold—preregistered paired designs and explicit score corrections meet hard standards.
Measurement choices can reverse survival verdicts for autonomous agent code but only shift effect sizes for human-supervised agents, with variance estimator deciding Devin's outcome.
Editor's noteFirst to systematically vary nine measurement choices in post-merge code survival studies, constructing a specification curve of 88 analyses: for Devin code, 15/22 specifications report 'modified earlier' when rows are independent, but after clustering standard errors by repository, only 3 remain (no more than permutation labels), while human-supervised agents only change effect sizes. It also reruns five published designs on the same data, identifying choices that move headline results, and validates LLM intent classifiers with human labels—a directly reusable audit template for any empirical work studying harness effects or code survival.
Multi-step SWE benchmark shows frontier coding agents solve only 18.5% of long-horizon tasks, failing early and self-verifying against their own flawed expectations.
Editor's noteFirst to make 'agent continuously building on its own code' a 5–15 step, multi-hour-per-step, 69-task chained benchmark: step k starts from the workspace left by step k−1 rather than reference solutions, with design decisions and errors propagating forward. Compared to Issue 13's ViBench 2.0 greenfield single-session building, it tests continuous evolution across multiple days and commits, with the strongest model solving only 18.5%, and 52% of failed steps reported as complete by agents—directly supporting 'harness updates ≠ harness benefits', and its adversarial hardening checklist is a template for hackability audits of any new benchmark.
SWE-CoWork shows frontier coding agents satisfy 63% of tests but complete under 8% of tasks, failing at sustained requirement tracking across organizational workflows.
Editor's noteFirst to make 'distributed requirements recovery with multiple colleagues, records, and owners' a repository-level coding benchmark: agents start from brief handoff notes, locate requirements, clarify conflicts, and track delayed issues across 26 digital employees, 10,329 kanban cards, 5,097 versioned documents, and 40,836 chat messages, scored by private executable tests. Compared to Issue 10's ReqClarifyBench single-user clarification, it treats 'multi-owner collaboration within an organization' as a first-class evaluation object, and Oracle-Spec pairing shows that providing full requirements raises F2P by ~16pp but does not move full task success rates at all, with requirement-level trajectory attribution (93% first-question owner hits but only 64% requirements obtained, 31% of reported-complete requirements failing tests) exemplifying the attribution discipline this digest emphasizes.
Working-context compaction drops even the best LLM's task success to 68.5% on a new 200-task benchmark, with failures driven by whether task state persists in the environment or only in context.
Editor's noteFirst to treat 'work context compression' as a first-class evaluation object, using a matched continuation budget protocol to quantify compression's impact on downstream task completion in real repository tasks, and providing task state persistence (context-resident vs environment-backed) as an explanatory mechanism for compression difficulty. Compared to Issue 12's Compact-Bench paired-replay compression evaluation, it adds cross-domain (Search/Coding/Workspace) and cross-execution-stage (25%/50%/75% boundary) comparisons, and is the first to distinguish 'omission' vs 'state distortion' as two types of compression errors with different failure paths (redundant work vs premature termination).
CrossSWE, a 171-instance benchmark requiring joint modifications across multiple repositories, shows frontier LLMs achieve only 58.48% Pass@1, with cross-repository inconsistency as the dominant failure mode.
Editor's noteFirst to treat 'cross-repository joint development' as a first-class evaluation object, requiring agents to modify multiple repositories simultaneously to pass tests, and using 2^N mixed-version configurations to exhaustively verify joint-necessity—partial modifications must fail, full modifications must pass. Compared to Issue 8's WideSWE cross-repository coordination (without joint-necessity verification), it upgrades 'modifications must span repositories' from task design to an executable hard constraint, and reports 44.2% cross-repository inconsistency as the dominant failure mode, with contamination disclosure (excluding gold patches/hidden tests, clean Git state) meeting hard standards.








