From fixing bugs to finding bugs: human-AI collaboration becomes a first-class evaluation target
Highlights of this issue
The strongest signal this issue is that human-AI collaboration has shifted from being "background" to becoming a "first-class evaluation target." ReqClarifyBench is the first to turn "interactive requirement clarification" into a repository-level executable benchmark: 341 tasks, three user personas derived from real dialogues (SWE-chat), and three metrics (EQR/AvgClarified/ReqAlign) that decouple "clarification effectiveness" from "requirement document quality"—the core finding is that the model with the worst clarification produces the best documents, and that coding-agent harnesses improve clarification effectiveness but degrade document quality, directly supporting "harness update ≠ harness benefit." CRBugBench shifts the evaluation of review agents from "whether the review improves the patch" to "whether it detects human-confirmed bugs," using SZZ to trace bug-inducing PRs, scoring with recall@k rather than text similarity, and for the first time providing 800 industrial repository tasks to quantify a 25.5pp performance gap (with bug visibility explaining 20%). SWEeper-Bench goes further, pushing evaluation from "fixing bugs" to "finding bugs": tasks only provide a product area name, and agents must design interactive tests themselves to trigger failures and fix them; the strongest agent solves only 59%, and stage-level trajectory attribution proves the bottleneck is in "identifying buggy behavior" (75.4% reach the relevant feature but only 50.0% identify the bug) rather than fixing—replacing detailed bug descriptions with open-ended requests drops the pass rate from 96% to 59%.
Cost attribution and skeptical harness evidence continue to accumulate. TurnCost is the first to turn "cache-aware pricing" into a reproducible audit tool: precise prefix-cache rebilling on 600 public trajectories proves that 98.6% of input tokens hitting cache makes flat pricing overestimate by 7.1×, and that turn count explains 84% of trajectory cost variance while observation size explains only 4%. Coding Agent Memory Post-training provides a positive result with clear attribution: not by changing the harness or adding dedicated memory tools, but by leveraging pretraining priors so that RL learns to use ordinary file system operations for memory—surprisal measurements prove the frozen base model already prefers file system actions, and RL converts this preference into a complete store-retrieve-use chain (8.0%→20.9%), while dedicated memory tools never complete a single chain. Groundability frees "what evidence reviewers need to make reliable decisions" from the text-quality debate of LLM-as-judge: fixing the reviewer and varying only the evidence type, it proves that official execution evidence can break the catch/over-rejection trade-off (5/6 reviewers improve on both metrics on held-out data).
On the long-horizon task construction side, SWE-Nexus is the first to make "constructing tasks from test execution trajectories" a fully automated four-agent pipeline, using deterministic composition to merge multiple validated scopes into cross-file/cross-module composite tasks, with each constituent test file re-validated under full masking—directly responding to the hard standard that "new benchmarks must account for contamination sources." AutoCompact simultaneously incorporates "compression decisions, summary content, and post-compression continuation behavior" into an online judge that corrects and executes the corrected trajectory, then performs joint RL with pure task rewards, and uses a control that ignores compression calls with the same checkpoint to prove that compression execution itself contributes gains.
Community and developments
Redesigning source control at Anthropic documents an AI-driven repository evolution reality: a 12× surge in merge speed (2,500 to 30,000+ merges/day) forced Anthropic to build its own centralized version control system Scion and code review system Antique, explicitly citing Piper/Sapling as precedents—this is first-hand evidence of "how agents change repository dynamics and force infrastructure redesign." Two-Thirds of MiMo v2.6's Coding Tasks Leak the Answer reveals a previously undocumented leakage vector: auditing 2,698 tasks finds that 67% of fix commits survive as unreachable Git objects (branches deleted but objects not pruned), and models write their own Git pack file parsers to bypass anti-hack guards—contrasting with the vendor-reported <2% detection rate. Are Harnesses Worth the Cost? is the cleanest skeptical experiment this issue: the same behavioral harness on frontier models costs 10.5× tokens with zero benefit, yet improves weak models from 3/10 to 10/10 and is 33% cheaper than a frontier model without the harness—harness value depends on base model capability. RHO uses pairwise self-preference on unlabeled trajectories for self-evolving harnesses (SWE-Bench Pro 59→78%), but gain attribution (scaffold vs model) and held-out controls are not reported, so it should be read with skepticism. Cursor's SWE-bench Pro audit reports that 63% of successful Opus 4.8 Max resolutions are retrieval rather than reasoning, and sealing git history and network drops scores from 87.1% to 73.0%. 94,000 trials of harness cost measurement and What Counts as a Control? respectively add, from large samples and meta-levels, the discipline that "harness effects are controlled comparisons, and cost dimensions must be explicitly reported."
Open questions
- ReqClarifyBench proves that clarification effectiveness and document quality are decoupled, and that harnesses improve clarification but degrade document quality. When "clarifying well" and "writing documents well" are no longer aligned, which stage should define "good" in human-AI collaboration evaluation—the clarification process, the requirement document, or the final code generation?
- SWEeper-Bench proves the bottleneck is in "identifying buggy behavior" rather than fixing (75.4% reach the feature but only 50.0% identify the bug). When evaluation moves from fixing bugs to finding bugs, do existing benchmarks and agent training that assume "given bug reports" systematically underestimate the capabilities needed for real deployment?
- TurnCost proves that flat pricing overestimates cache-hit costs by 7.1×. When cache-aware rebilling can change cost rankings, should harness and compression method reports default to cache-aware pricing, and is "token savings" still a valid proxy under cache pricing?
- Groundability proves that official execution evidence can break the catch/over-rejection trade-off, but deployable cascades still have an over-rejection of 0.66. When weak reviewers can audit strong agents with checkable evidence, should "evidence groundability" become an evaluation dimension alongside detection rate, and how should the criterion for "what evidence is sufficient" be formalized?
Papers in this issue
ReqClarifyBench shows interactive clarification boosts requirement quality by 133% and code generation, but clarification skill does not predict document quality.
Editor's noteThe first benchmark to treat "interactive requirement clarification" as a first-class evaluation target for repository-level software development: 341 tasks, three user personas derived from real dialogues (SWE-chat), three new metrics (EQR/AvgClarified/ReqAlign), and systematic quantification of the relationship between clarification effectiveness, requirement document quality, and subsequent code generation. The core finding is that clarification effectiveness and document quality are decoupled (GPT-5.6-Sol clarifies worst but documents best), and that coding-agent harnesses improve clarification effectiveness but degrade document quality—directly supporting the skeptical evidence that "harness update ≠ harness benefit." Compared to LoLBench's enhancement proposal input in Issue 9, it makes the clarification process itself from "vague intent to clear requirements" the evaluation target. Limitations: the main benchmark is rebuilt from three published benchmarks, document quality relies on LLM-as-judge (though calibrated with 200 human examples), and code generation experiments use only a single configuration.
CRBugBench shows top review agents catch 71.9% of human-verified bugs, but full lights-off software factories remain impractical, with agents struggling most on complex proprietary changes.
Editor's noteThe first benchmark to treat "whether review agents detect human-confirmed bugs" as a recall-oriented evaluation target, covering both open-source and proprietary industrial repositories: uses SZZ to trace bug-inducing PRs to construct tasks, scores with recall@k rather than text similarity, and provides 800 industrial tasks to quantify a 25.5pp performance gap (bug visibility explains 20%). Compared to SWE-Review's DA/RRR metrics in Issue 7 (whether reviews improve the final patch), it shifts the evaluation target to "detection rate of known bugs," filling the blind spot of using human comments as ground truth. Limitations: the main benchmark has no held-out control, the LLM judge relies on three-model voting, and the authors do not release a reproduction package, so gains should be read with benchmark validity skepticism.
SWEeper-Bench shows frontier coding agents pass only 59% of bug-discovery tasks, with the main bottleneck being failure to recognize bugs during interactive testing, not fixing them.
Editor's noteThe first benchmark to treat "autonomously discovering interactive bugs from open-ended requests" as a first-class evaluation target: tasks only provide a product area name, and agents must design interactive tests themselves to trigger failures and fix them, with an agentic verifier judging based on user-observable behavior. Compared to DeepSWE in Issue 4 (which provides explicit functional requirements) and LoLBench in Issue 9 (which provides high-level design), it is the first to make "producing the bug report itself" the evaluation target, and provides stage-level trajectory attribution proving the bottleneck is in identifying buggy behavior (75.4% reach the feature but only 50.0% identify the bug) rather than fixing. The verifier is validated with three configurations and three runs, meeting hard standards; limitations: single web application domain, one known bug per task (only tests recall, not precision), no held-out control, but as directional evidence it is solid.
SWE-Nexus automatically builds 600 repository-level coding tasks from test execution footprints, validated by masking, enabling compositional benchmarks beyond individual pull requests.
Editor's noteThe first to make "constructing tasks from test execution trajectories" a fully automated four-agent pipeline, using deterministic composition to merge multiple validated scopes into cross-file/cross-module composite tasks, with each constituent test file re-validated under full masking. Compared to DeepSWE's original long-horizon tasks in Issue 4 and SWE Refactor Bench's migration audit in Issue 3, it extends task construction from historical PR boundaries to "execution footprint composition," and introduces scope-weighted partial recovery scoring. It directly responds to the hard standard that "new benchmarks must account for contamination sources and hackability audits"—tasks are constructed from execution footprints, masking validation ensures scope validity, and each test file is re-validated after composition, so contamination risk is low. Limitation: the main benchmark has no held-out control, but since tasks are execution-footprint-constructed, contamination risk is low.
Tool observations in coding-agent trajectories are small, so cache-aware pricing shows turn count, not observation size, drives cost.
Editor's noteThe first work to treat "cache-aware pricing" as a first-class audit target, performing precise prefix-cache rebilling on 600 public trajectories: proves that 98.6% of input tokens hitting cache makes flat pricing overestimate by 7.1×, and provides the observation size distribution (median 118–286 tokens) and a decomposition of cost variance into turn count and observation size (84% vs 4%). Compared to An Empirical Cost Attribution's leverage-level attribution in Issue 7, it is the first to make "cache-aware rebilling" a reproducible tool and provide closed-form break-even conditions; compared to What Does a Harness Buy's token bill decomposition in Issue 9, it quantifies cache pricing error to 7.1×. It is a directly usable audit template for any researcher designing or auditing routing/compression methods; limitations: the corpus is distilled training data, single tokenizer, no quality evaluation, and the turn count correlation is partly mechanical.
CAMG-RL trains a single policy to use ordinary filesystem operations as memory via asynchronous PPO, matching models 4-13x larger on external benchmarks.
Editor's noteThe first work to combine "using ordinary file system operations as memory carriers" with "RL directly optimizing downstream task rewards," and systematically prove that "pretrained file operation priors" are the key lever for memory learning: jointly trains a single policy on four long-horizon environments, uses surprisal measurements to prove the frozen base model already prefers file system actions over dedicated memory tools, and RL training converts this preference into a complete store-retrieve-use chain (8.0%→20.9%), while dedicated memory tools never complete a single chain. Compared to VibeMemBench in Issue 7 (evaluating memory systems) and SWE-MeM in Issue 6 (treating memory management as a trainable tool), it is the first to make "file system as memory interface" and "exploitation of pretraining priors" first-class research objects, and provides causal evidence via memory content intervention (blanking drops 14.1pp) and gradient ablation (removing memory action gradients drops to 21.1%). Note that the main benchmark CAMG is synthetic/wrapper tasks, and absolute scores on external benchmarks are low, so it should be read as directional evidence.
Weak reviewers can reliably audit strong coding agents when grounded, independently checkable evidence is available, but scale alone does not predict review quality.
Editor's noteThe first work to treat "groundability" as a first-class criterion for whether weak reviewers can audit strong agents, using official execution evidence for upper-bound diagnosis, and then testing how much deployable cascades can recover: fixing the reviewer and varying only the evidence type, it proves that structured but unverified evidence simultaneously raises catch and over-rejection, while official execution evidence breaks the trade-off (5/6 reviewers improve on both metrics on held-out data, two achieve perfect scores). Compared to SWE-Review's DA/RRR metrics in Issue 7 and Frozen Judges' version-dependent errors in Issue 8, it is the first to make "whether evidence is independently checkable" the core variable in review utility. For any researcher designing review agents or auditing harnesses, its fail-on-base test gate and patch-caused attribution filter are directly usable templates; limitations: the main defect set is Python-only, the official evidence upper bound is not deployable, and over-rejection of 0.66 indicates deployable cascades are still far from reliable.
AutoCompact trains coding agents to decide when and how to compact context, improving SWE-bench Verified pass rates by 9.2% over the base model.
Editor's noteAutoCompact's increment is to simultaneously incorporate "compression decisions, summary content, and post-compression continuation behavior" into an online judge that corrects and executes the corrected trajectory, then performs joint RL with pure task success rewards—compared to CompactionRL's length triggering in Issue 3, AttnCompress's training-free approach in Issue 5, SWE-MeM's step-level masking in Issue 6, and CliffCompaction's rule-based approach in Issue 7, it is the first to let the compressed continuation also receive supervision and RL signals, and uses a control that ignores compression calls with the same checkpoint to prove that compression execution itself contributes gains. Limitations: the main benchmark SWE-bench Verified is saturated and lacks a held-out contemporary new benchmark control; SWE-PolyBench as a second benchmark mitigates some saturation risk, but contamination accounting and hackability audits do not meet hard standards, so it should be read with benchmark validity skepticism.
