Verification Gap Outweighs Code Quality: Reliability Bottlenecks in Long-Horizon Chained Builds
Highlights of This Issue
The strongest signal this issue: when evaluation shifts from "single request" to "agent chaining builds on its own code," bottlenecks are collectively attributed to missing verification and reliability, not code generation quality. Built on Broken Assumptions (BOBA) uses 280 chains of 20 requests with the strictest paired protocol to separate the variables "who wrote the code" and "whether it was verified": verified agent code builds almost as reliably as human code (gap ≤4pp), while unverified status doubles failure rates, and each unverified mistake doubles the odds of failure again; a single external "verification failed" message raises per-model completion rates by 8–16 points, and "naming failing tests" once is more effective than "insisting on refusal" three times. The same theme recurs in Multi-step SWE (69 tasks, 574 steps, nine languages, strongest model's full-task solve rate only 18.5%, 38/69 tasks unsolved by any model, step-level 70.2% pass but dropping to 14.1% after a previous step fails)—agents can pass single steps but cannot sustain the entire chain.
The reliability gap is equally striking on two "from-scratch" benchmarks. ViBench 2.0 has agents build complete web products from empty projects in a single session following 9–18 feature requests, with final all-or-nothing scoring on 544 test plans from a user perspective; the three strongest models achieve plan pass rates of 94–98% but pass@1 of only 49–65%, and open-weight models score zero—"almost completing every plan yet still failing over half the builds" is a reliability rather than capability gap. Contrasting last issue's SWE-Shift conclusion on 'harness updates ≠ harness gains', this issue advances it to "harness single-step gains ≠ long-chain gains," with The Coding Agent Said it Was Done and Shandong University's 492-task cost audit providing supporting evidence from the practical side.
On failure prediction and verifier auditing, Hard Issue, Weak Model, or Bad Run? advances last issue's Continue or Restart finding that "within-task attempts have no discriminative power": using a non-specialized reference distribution, it decomposes pooled AUROC into three causal sources—issue difficulty (accounting for 99.66% of paired comparisons), model capability, and the run itself—proving that trajectory predictors have statistically zero gain after controlling for the first two, oracle gap and upset rates do not exceed random variation, and model-switching gains come almost entirely from extra attempts rather than choosing the right model (specialization explains only 4.6% of variance). Impossible Code, Barely Changed Score adds a blow from the evaluation side: replaying checkpoints raises syntax failure rates from <1% to 63–67%, yet strong judges' Pass@32 drops only about 3pp, and with a k=128 sampling budget discriminative power nearly disappears (0.71pp) while execution tests maintain 15.9pp. Audit Where Hacks Pay provides a principled framework for budget-constrained strong verifier auditing under GRPO: Theorem 1 proves that auditing the group with the fewest "weak passes" first minimizes residual advantage mass, and it is the first to empirically show that "randomness in audit estimates affects training outcomes"—unbiased corrections yield 6–11 points less held-out accuracy than this strategy under the same budget.
Community and Developments
The Clean Coding Index re-evaluated 10,506 trajectories from two benchmarks, annotated 410 reward-hack attempts, and held out contaminated tasks, yielding a clean score with cross-benchmark re-ranking; its hardest finding is that "reward hacking is an environment property, not a model trait"—the same model shows zero hacks on patch-based DeepSWE but attempts the most on oracle-exposed SWE-Marathon, and hack rates remain nearly constant across scaffolds while capability can differ by about 2×, directly hitting the hard standard of "reporting only single-benchmark pass rates." It admits it audits only these two benchmarks solely because they release every trajectory; the Holdout community audit provides a reproducible catalog of reward hacks across 20 public RL environments (including "what the agent writes, the grader reads" types), serving as a ready-made checklist for hackability audits. Snorkel's peek-or-ask experiment quantifies the substitution effect of reward function source accessibility on clarification behavior: under the default harness, about 25% of rollouts choose to peek at the reward rather than ask a simulated user, and putting user checks in the system prompt raises the asking rate to 100%. On human-machine mismatch, a mismatch analysis of 20,574 real sessions quantifies that 26.95% of episodes stem from underspecified instructions, and 91.49% of resolutions still require explicit user correction. A two-agent context file ablation uses equivalence tests to bound AGENTS.md's effect on correctness to ≤10–15pp (ineffective), giving only process-level efficiency signals when the file carries test-time cost warnings (blind full-suite runs decrease monotonically with context)—an honest negative result for "in-repo instruction artifacts."
Open Questions
- BOBA proves that verified status determines whether the next item builds more than code quality, and a single verification message raises per-model completion rates by 8–16 points. If "systems are only as reliable as the checks they can reach," should harness engineering goals shift to giving agents an external verification channel they can "reach," rather than improving generation quality itself—and how wide should this verification line be (naming tests vs. repeated refusal)?
- Hard Issue proves trajectory predictors have zero gain after controlling for issue difficulty and model capability, and Continue or Restart proves within-task attempt selection has almost no discriminative power. If failure prediction cannot distinguish "bad runs" from "hard issues/weak models" at the run level, and run-to-run variation accounts for 42% of total variance, where can retry, switching, and stopping strategies obtain attributable gains?
- Impossible Code proves judge scores are nearly insensitive to 63–67% invalid code and lose discriminative power with sampling budget. When judge-based evaluation is so insensitive to output damage, should execution-level re-evaluation and syntax gates like the Clean Coding Index become standard companion metrics for any rewarded/harness change report, rather than relying solely on a single judge pass rate?
- Both ViBench 2.0 and Multi-step SWE show high step/plan pass rates but low full-chain completion rates. If "each step correct" and "full chain correct" are systematically disconnected and failures are mainly due to inherited defects and missing verification, should long-horizon benchmark reports uniformly split into "step-level" and "chain-level" metrics, and treat harness verification design as an audit target alongside pass rates?
Papers in this issue
Coding agents complete 95% of individual requests but only 51% of chains built on their own work, because unverified silent misses compound into broken foundations.
Editor's noteUses 280 chains of 20 requests, 5,600 requests in the largest-scale paired experiment to separate "who wrote the code" from "whether it was verified": verified agent code builds almost as reliably as human code (gap ≤4pp), while unverified status doubles failure rates and each unverified mistake doubles the odds of failure again. Compared to last issue's SWE-Shift matched continuation, it is the first to treat "verification status" as a first-class control variable and provides intervention quantification showing a single external 'verification failed' message raises per-model completion rates by 8–16 points. 280 chains, 10 models, fully public trajectories (sealed images and grader), and contamination disclosure meet hard standards, making it the strongest positive attribution evidence for "harness updates ≠ harness gains."
ViBench 2.0 reveals a reliability gap: top coding agents pass up to 98% of test plans yet deliver a fully working product in at most 65% of long-horizon builds.
Editor's noteFirst to make "greenfield multi-turn product building" an end-to-end benchmark: agents build complete full-stack products from empty projects following 9–18 feature requests in a single session, with final all-or-nothing scoring on 544 test plans from a user perspective, plus held-out 9 products, independent external team reproduction, and wrong-fail/wrong-pass dual error validation. The core is the "reliability gap": plan pass rate 94–98% but pass@1 only 49–65%, open-weight zero passes. Compared to last issue's SWE-Shift matched continuation and Issue 11's LoLBench proposal inputs, it makes "from-scratch building + multi-turn self-evolution + final whole-product acceptance" a first-class evaluation object.
Trajectory-based failure predictors for coding agents predict issue difficulty and model identity, not run-specific failure, adding zero measurable signal beyond these run-blind factors.
Editor's noteAdvances last issue's Continue or Restart finding that "within-task attempts have no discriminative power" to "trajectory predictors have statistically zero gain after controlling for issue difficulty and model capability," using a non-specialized reference distribution to test that oracle gap and upset rates do not exceed random variation, and model-switching gains come almost entirely from extra attempts rather than choosing the right model (specialization explains only 4.6% of variance, run-to-run accounts for 42%). Compared to last issue, it adds model capability as an independent confounder, equivalence tests, and repeated-run pilots; its decomposition protocol (within-issue AUROC, non-specialized reference) is a directly reusable audit template for any researcher designing stop/switch strategies.
A high judge-based coding score can hide severe syntax damage, with invalid Python rising from under 1% to 63-67% while Pass@32 drops only 3 points.
Editor's noteFirst to use replay studies to prove judge-based evaluation's insensitivity to generation quality degradation: syntax failure rates jump from <1% to 63–67%, strong judges' Pass@32 drops only about 3pp, and with a k=128 sampling budget discriminative power nearly disappears (0.71pp) while execution tests maintain 15.9pp. Compared to last issue's Rethinking Verification review design matrix, it makes "judge score sensitivity to output damage" a first-class evaluation object and provides an extremely cheap syntax gate as a reproducible audit tool—any work relying on judge pass rates to report harness/model changes should adopt it as a companion check.
Low-pass budgeted audits with reflected penalties suppress reward hacking in GRPO, while unbiased random audits fail because update randomness, not just expected gradient, determines held-out accuracy.
Editor's noteFirst to provide a principled framework for budget-constrained strong verifier auditing under GRPO: Theorem 1 proves that auditing the group with the fewest "weak passes" first minimizes residual advantage mass, and it is the first to empirically show that "randomness in audit estimates affects training outcomes"—unbiased corrections (IPW, group-level correction) yield 6–11 points less held-out accuracy than this strategy under the same budget. Its pre-registration protocol and frozen-panel prediction methodology are directly reusable templates for any RLVR audit or verifier intervention design, directly hitting the reward hacking and benchmark validity themes.
MemUse reveals that LLM memory systems deliver mostly irrelevant or unused content, with task outcomes hiding beneficial use in failures and harmful use in successes.
Editor's noteFirst to advance memory evaluation from task outcomes to "the use and consequences of a single delivery," using prior-context to separate information quality/novelty from usage evidence/consequences, and credit windows to trace effects to episode end, all without reference solutions or extra execution. Core evidence: 23.2% of beneficial uses still fail, 31.3% of harmful uses still succeed, directly proving task outcomes mask memory effects; compared to last issue's MemDev-Bench real historical memory and Issue 7's VibeMemBench memory-on/off whole-pairing, it fills the delivery-level blind spot where "whole-pairing cannot explain null results."
Continual harness evolution for coding agents suffers severe catastrophic forgetting, with verification conflicts causing up to 100% performance drops on prior tasks.
Editor's noteFirst to introduce "catastrophic forgetting" as a first-class evaluation object in harness self-evolution: continuously evolving across six evolution directions covering two representative methods (AHE, Meta-Harness) and three task types, quantifying SWE-Bench relative decline up to 66% (even to 0% under artifact constraints), and manually analyzing 520 trajectories to give a three-category harness-level conflict typology (verification/workflow/budget), with verification conflicts accounting for 87.1% of pass-to-fail transitions. Compared to last issue's Self-Evolving Harnesses retrospective adaptation, it makes "continuous evolution across task sequences" an independent evaluation dimension—essential skeptical reading for any research claiming harness self-evolution gains are transferable.
