Code Agent Trajectory SynthesisIssue 5Nov 7 – 14, 2026

Verifier Quality Becomes the Training Target: From Acceptance Gates to Predictable, Intervenable Mechanisms

Highlights of This Issue

The clearest signal this issue: the quality of verifiers and rewards themselves is shifting from an 'acceptance gate at evaluation time' to a 'research object during training'. Three independent works advance verifier reliability from empirical concern to quantifiable, predictable, and intervenable mechanisms from different angles. VerifierBench is the first to systematically evaluate the agent-generated executable verifier itself, rather than the solver or test generation, uncovering a previously unquantified systematic bias: generated verifiers are far more reliable at rejecting invalid solutions than accepting valid ones (across 19 models, TNR generally above 80%, TPR generally below 70%), and most false rejections concentrate on all-reject verifiers—contributing 87.9%–96.0% of false rejections per model in archival records. Verifier-Grounded Agentic RL offers a constructive solution: use verifier diagnostics (obligation ratio) rather than binary verdicts for dense rewards, and add an integrity audit layer above the verifier to prevent specification rewriting and assuming the conclusion. Honesty Is a Race derives a testable condition from first principles (p̃·r/b > 1+κ), predicting when verifiable-reward RL teaches models to bluff, and provides a generous-then-cut schedule to avoid collapse.

The second theme: 'single-reference evaluation' itself is questioned. TestPrism proposes a Joint Success Function requiring generated tests to fail on the initial state, accept all valid candidate implementations, and reject all invalid ones, explicitly constructing a candidate panel with 4 valid alternatives and 5 invalid implementations. Across 14 configurations, single-reference success is 59.67% while Joint Success Function is only 28.00%—directly addressing SWE-Bench Pro V2's concerns about weak tests and hacking from Issue 4, advancing 'test sufficiency' from single-reference pass rate to cross-implementation reliability.

Two community signals mutually reinforce the judgment that 'environment quality is the bottleneck'. SWE-rebench V2.1 RL from Issue 4 now provides key training-signal evidence: its core increment is using 'repairing broken tasks' rather than 'synthesizing new tasks' as the environment construction path, with coding agents reviewing and fixing defects per task, auditing mirror leaks, removing tasks with failing reference patches or unstable environments, and explicitly measuring the improvement in GRPO training signal (Qwen3.5-9B from 55 to 70)—attributing gains to environment quality rather than base model. Meanwhile, Open Models Have No Common Crawl for Agent Training quantifies the scarcity ceiling of naturally occurring agent training data: only 2,294 of 90,000 PRs become SWE-bench tasks. Together, they show that 'manufacturing' and 'repairing' environments and verifiers, rather than 'mining', is the path to scale.

Community and Updates

SWE-rebench V2.1 RL releases 13,337 executable RL environments derived from real GitHub issues (20 languages, each with prebuilt Docker images, tests, and graders). The core increment is using 'repairing broken tasks' rather than 'synthesizing new tasks' as the environment construction path: coding agents review and fix defects per task, audit mirror leaks, remove tasks with failing reference patches or unstable environments, and explicitly measure the improvement in GRPO training signal (Qwen3.5-9B from 55 to 70). Repair cost is about $0.24/task, and 20% of final tasks were initially deemed unqualified by LLM quality filtering—showing that 'repair' can recover environments discarded by filter-based mining. Limitations: training signal improvement comes from a 100-task sample and a single 9B model; final agent quality gains await independent verification.

Open Models Have No Common Crawl for Agent Training provides a quantified yield anchor: the original SWE-bench crawled about 90,000 PRs from 12 Python repositories, only about 1/8 mapped to the task framework, of which about 80% could not be built into executable tasks, leaving only 2,294; subsequent manual review for SWE-bench Verified removed more than two-thirds. The author argues there is 'no data commons for posttraining'—converting repositories into training tasks is more 'manufacturing' than 'curation'. This is quantitative support for the motivation behind synthetic task generation methods (like SWE-smith, R2E-Gym).

Multi-Harness RL proposes building an independent sandbox per rollout as a cost-control mechanism for scaling RL trajectories, and provides empirical evidence echoing CoTrace's provenance claim from Issue 1: the same weights solve 62% under Mini-SWE-Agent but only 33% under Claude Code; single-harness training only improves on the trained harness, while multi-harness training improves across all four harnesses—harness fingerprint's influence on training data is again quantified.

Open Questions

  1. VerifierBench attributes false rejections to two mechanisms: all-reject verifiers and confirmatory self-testing. Can these mechanisms be distilled into executable screening rules to automatically intercept unreliable verifiers in synthetic environment pipelines (like SWE-smith/R2E-Gym), rather than only post-hoc evaluation?

  2. Honesty Is a Race's predictive condition (p̃·r/b > 1+κ) is validated on small models and synthetic/code tasks. When verifiers are test suites rather than formal checkers, and model scale and task complexity increase significantly, how should the bluffing threshold be calibrated? Does the generous-then-cut schedule work equally in SWE scenarios?

  3. Last issue's SWE-Bench Pro V2 noted that training-time exposure cannot be eliminated by locked protocols; this issue's SWE-rebench V2.1 RL proves that repairing broken environments improves training signal. When environment repair cost (about $0.24/task) is comparable to synthesizing new tasks, how should the budget allocation between 'repair vs. synthesize' be decided?

  4. Verifier-Grounded's integrity audit relies on formal specifications. For typical SWE tasks without formal specifications, can an equivalent 'specification completeness' audit be constructed to prevent agents from hacking verifiers by rewriting tests or weakening assertions?

Papers in this issue

  1. VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

    VerifierBench shows LLM-generated verifiers reject invalid solutions reliably but accept valid ones poorly, with most failures from unjustified checks that self-testing fails to expose.

    Editor's note

    First systematic evaluation of the agent-generated executable verifier itself, rather than solver or test generation. Core finding: generated verifiers are far more reliable at rejecting invalid solutions than accepting valid ones (across 19 models, TNR generally above 80%, TPR generally below 70%), and most false rejections concentrate on all-reject verifiers—contributing 87.9%–96.0% of false rejections per model in archival records, a bias previously unquantified. Compared to SWT-Bench and TestGenEval which only test test/assertion generation, it extends verification from a single benchmark to three task types (tool calls, states, code) and proposes V4V six-item screening and a valid variants + attacks construction protocol. Provides empirical evidence for reward signal reliability in synthetic training pipelines; V4V screening and trace-level attribution are transferable to verifier quality control in SWE-smith/R2E-Gym-like environments; limitation: training gains not tested.

  2. Verifier-Grounded Agentic RL for Provably Correct Code

    Verifier diagnostics, not binary verdicts, as dense RL rewards plus integrity audits raise verified code generation from 66.1% to 93.5% on DafnyBench and 56.3% to 100% on SWE-Proof.

    Editor's note

    Advances verifier rewards from binary verdicts to dense, provably correct signals: uses verifier diagnostics (obligation ratio) rather than verdicts for scoring, and adds an integrity audit layer above the verifier that rejects any candidate rewriting specifications or assuming the conclusion. Compared to Re:Form/VeriGate which only reward verdicts, and test rewards lacking reliability guarantees, it validates on held-out benchmarks, including transfer to unseen verifiers/languages, controlled comparisons with binary outcome rewards and SFT, and frozen-expert ablations—attribution evidence fits this direction's taste. Limitation: focuses on formal verification rather than typical SWE-bench-style test tasks.

  3. Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff

    A GRPO reinforcement step teaches honesty only when holding rate times reward exceeds bluff payoff plus cross-class coupling, else it teaches models to bluff.

    Editor's note

    Provides the first testable, first-principles condition (p̃·r/b > 1+κ) predicting when verifiable-reward RL teaches models to bluff, derived from GRPO's group-mean advantage and gradient coupling, and introduces a 'race' mechanism and 'generous-then-cut' schedule to avoid collapse. Compared to prior empirical observations of reward hacking or abstention collapse, it offers a predictive threshold and causal control (lottery) to isolate the checker's role, and the 'race' mechanism explains why runs slightly below threshold may win while lottery cannot. Limitation: experiments on small models and synthetic/code tasks, not validated on full SWE-bench.

  4. TestPrism: Rethinking Test Evaluation Beyond a Single Reference

    TestPrism shows single-reference test evaluation overstates quality by 2x, with frontier agents achieving only 28% joint success across valid and invalid implementations.

    Editor's note

    Proposes a Joint Success Function requiring generated tests to fail on the initial state, accept all valid candidate implementations, and reject all invalid ones, breaking the limitation of single-reference evaluation. Explicitly constructs a candidate panel with 4 valid alternatives and 5 invalid implementations, paired with TestHelix heterogeneous synthesis and peer cross-validation to improve cross-implementation reliability. Across 14 configurations, single-reference success is 59.67% while Joint Success Function is only 28.00%, and provides error classification (coverage gaps, wrong assertions, execution logic errors)—a strict definition of test sufficiency transferable to training data filtering and reward design.

  5. Rethinking "RL Generalizes, SFT Memorizes": The Role of SFT Data

    SFT matches or beats RL on out-of-distribution reasoning when trained on data from a model close to the base and scaled to RL's total rollout budget.

    Editor's note

    Challenges the claim that 'RL generalizes, SFT memorizes': by matching SFT and RL rollout budgets (correct rollout count and total rollout count) and mixing data sources, it shows SFT can match or even surpass RL on general reasoning. The increment lies in advancing the SFT vs. RL comparison from prompt-level alignment to rollout-budget alignment, with long-tail method coverage as the explanatory mechanism; when matching total rollout budget, rollouts containing incorrect final answers still contain correct intermediate steps, which actually improves generalization. Although the domain is mathematical reasoning rather than code agents, its budget-matching and attribution ideas are transferable to SWE scenarios.