Failure Trajectories and Verifier Quality: Two New Fronts in Synthetic Data
Highlights of This Issue
The most prominent signal this issue is that "failure" has transformed from discarded waste into a repeatedly mined resource. At least four works handle failure trajectories from different angles: distilling persistently failing rollouts into transferable, leak-filtered procedural skills (FailForge); training recovery RL conditioned on failure patches and verifier feedback (FC-SWE); using the target model's own failures to drive new task synthesis (AutoSynthData's closed-loop approach); and ranking failure and success trajectories by structural quality rather than outcome (TQS). Together, these works point to a shared judgment: under a fixed sampling budget, the per-trajectory training value of failure samples often exceeds that of continuing to add successful samples—data quality is replacing data scale as the main battleground.
The second theme is the tightening of data attribution. Multiple works decompose gains to the data level rather than the base model or scaffold: ProgramGym uses ablations to separate the contributions of environment construction (SFT) and workflow (Program-Agent), FailForge uses data quality substitution under a fixed budget to prove recovery trajectories outperform additional sampling, and CoTrace uses same-chain paired comparisons to prove that trajectory provenance matching matters more than volume. Meanwhile, the quality of verifiers and benchmarks themselves is under unprecedented scrutiny—OpenAI's audit of SWE-Bench Pro estimates about 30% of tasks are flawed and retracts its previous recommendation, and the public/private gap in SWE-Bench Pro V2 reaches 17.8 points. Reporting synthetic data gains on benchmarks with uncontrolled contamination and flaws is rapidly losing persuasiveness.
The third direction worth noting is that "the environment itself becomes a first-class citizen." From ProgramGym's from-scratch program reproduction of environments, to TerminalHorizon's use of independent execution to detect and recover key work "stolen" by the construction process, to SWE-rebench's continuous decontamination evaluation pipeline—researchers are beginning to treat environment construction as an object requiring validation, not just a data source. This aligns with the perspective in The Workspace Is the Benchmark: generation-time verification and verifier reliability are becoming core constraints in synthetic data pipeline design.
Community and Developments
Community discussions this issue center on failure-driven data generation and trajectory cleaning. ServiceNow's AutoSynthData blog proposes a closed-loop synthetic system that uses the target model's failure trajectories as signals for generating new tasks, rather than only sampling from teacher rollouts or static bug datasets—this idea echoes the failure-reuse theme of several papers this issue, but lacks held-out benchmark validation, so caution is advised. Reid Marlow's Scaffolded Trajectories Make Terrible Agent Training Data makes a sharp argument: raw scaffolded trajectories bake verifier leakage and harness crutches into SFT data, and offers a Recursive Self-Rewrite cleaning method, complementing CoTrace's harness-aware data recipe.
On the task synthesis side, SciWalker uses operator graph sampling plus execution feedback repair to construct scientific computing problems, producing 8k+ verifiable tasks and yielding a 9.9-point RL gain on SciCode—a rare non-SWE synthetic work this issue. On the evaluation infrastructure front, SWE-rebench's automated continuous collection and decontamination pipeline, along with the public/private gap analysis of SWE-Bench Pro V2, jointly reinforce the core taste of this direction: validating training data gains on contamination-controlled held-out benchmarks.
Open Questions
- There is a fundamental tension in reusing failure trajectories: recovery training relies on verifier feedback (e.g., FC-SWE's verifier-assisted protocol, FailForge's leak filtering), but such feedback is often unavailable in deployment scenarios. How to make failure-driven training signals effective at inference time without verifier feedback remains an open problem.
- Several works this issue (TQS, FailForge, SWE-Prime) report that "a small number of high-quality trajectories outperform full data," but their quality metrics differ (structural convergence, procedural skills, semantic segments). Is there a unified trajectory quality metric that can predict SFT/RL gains across methods and benchmarks?
- When the public/private gap in SWE-Bench Pro V2 shows that training-period exposure can inflate scores by about 18 points, how much of the synthetic data gains are real capability improvements versus benchmark exposure? This requires synthetic data works to treat decontamination audits (like SWE-rebench's continuous mechanism) as standard practice rather than optional.
Papers in this issue
ProgramGym builds 1,648 verifiable program-reproduction environments and a role-isolated agent workflow, boosting a 27B model from 34.87% to 56.25% tests passed, exceeding flagship models.
Editor's noteShifts environment construction from 'patching existing codebases' to 'reproducing programs from scratch,' using a feedback-driven loop to automatically build verifiable, executable environments, with black-box test generation plus four-fold validation (pass/double/dummy/weak-assert) and coverage instrumentation to ensure verifier quality. Training trajectories are explicitly filtered by test pass rate, addressing MindForge's lack of behavioral filtering. The highlight is ablations that decompose gains into environment construction (SFT) and workflow (Program-Agent) layers, validated on held-out ProgramBench; limitations are that the new benchmark is not yet widely validated and the Resolved metric gain is near zero.
FailForge converts previously discarded failed software engineering trajectories into training data by distilling procedural skills, boosting SWE-bench pass@1 by 6.6 points over standard rejection sampling.
Editor's noteDistills persistently failing SWE rollouts into transferable, leak-filtered procedural skills to guide second attempts, then removes skills during training to let the student internalize behavior—a substantial advance over instance-level hints and stepwise masking. Under a fixed sampling budget, the number of recovered instances is nearly double that of additional sampling, and skills retain about 60% recovery rate after cross-instance shuffling, indicating they capture reusable investigation procedures rather than answers. Validated on SWE-bench Verified/MultiLingual with repository separation for held-out evaluation; ablations (leak filtering, skill removal, data quality substitution) provide the attribution evidence valued in this direction.
TQS, a structural scoring framework evaluating coding-agent trajectory process quality, boosts SWE-Bench Verified pass@1 to 27.6, outperforming random selection by 2.8 points at 43 ms per trajectory.
Editor's noteProposes TQS, a framework for scoring trajectory structure that requires no LLM judge, runs in linear time, and is fully deterministic, using seven event-log metrics (OEC, PSN, IAC, etc.) to operationalize 'structured convergence,' going beyond outcome-based pass/fail filtering to rank successful trajectories. Validates SFT gains from top-ranked trajectories on SWE-Bench Verified, with consistency, length-dependence, and diversity admission criteria to avoid mode collapse. Compared to AgentDiagnose's LLM scoring, it recovers most gains at minimal cost; note the single teacher model, fixed 1024-trajectory budget, and no multi-seed variance reported.
A 4B coding agent rivals 32B-100B models on SWE benchmarks via policy-adaptive synthetic task synthesis and iterative RL, without distillation.
Editor's noteTaskPilot is a rare online, policy-adaptive task synthesis pipeline: it not only filters but also refines synthetic SWE tasks based on current policy rollout feedback, anchoring to the learnability frontier (about 50% solve rate). The increment over SWE-smith and R2E-Gym is closed-loop task correction—tasks that are too easy or too hard are rewritten rather than discarded. Demonstrates that a 4B model can be trained purely on synthetic tasks via RL (without distillation) to achieve strong results on held-out benchmarks, and systematically studies harness design for compact models. A technical report from a well-known lab; reproducibility requires caution, but the methodological insights are solid.
SWE-Prime's two-stage trajectory and segment selection enables training on just 10% of resolved trajectories to outperform full-dataset SFT by up to 24.2%.
Editor's noteTwo-stage, multi-granularity selection: first, quality and representativeness filtering at the trajectory level (HDBSCAN clustering), then masking low-value segments at the semantic-segment level. Using 'semantic segments' (groupings of consecutive steps) rather than isolated steps as the unit of supervision quality assessment, with boundary-aware chunking for long contexts, this advances SWE-Lego's stepwise masking and STeCa's step calibration. Using public SWE-rebench OpenHands trajectories, a 10% subset outperforms full-data SFT, validated on SWE-Bench Pro/Verified with multiple base models; the main limitation is that code and data are not open-sourced.
FC-SWE trains software engineering agents to recover from failed patches via failure-conditioned RL, achieving 41.7% Resolved@1 on SWE-bench, outperforming GRPO by 2.9 points.
Editor's noteExplicitly trains recovery trajectories conditioned on failure patches and verifier feedback, using trajectory-local rewards and active-set advantage estimation to handle outcome-dependent chains—a direct advance over GRPO baselines and SWE-RL's 'independent attempts or inference-time-only feedback.' Controlled comparisons against GRPO/Base on SWE-bench Verified, with ablations isolating the contributions of failure context and reward design. The limitation is reliance on a verifier-assisted protocol (using benchmark test feedback), which limits direct applicability to deployment scenarios without such feedback.
CoTrace shows that trajectory value depends on the generating harness, and its harness-matched data recipe yields model gains where pooled corpora fail entirely.
Editor's noteTreats the data recipe itself as the research object: routes trajectory provenance by harness fingerprint, matching SFT data to the runtime used, and refreshes the curriculum only after tasks are solved and harvested. The core finding is that pooling trajectories across sibling harnesses yields zero accepted model updates, while a compact harness-matched corpus (30–50 trajectories) brings consistent gains—provenance matters more than volume. Validated on Tmax, Terminal-Bench 2.1, and SWE-bench Lite (including OOD transfer), with convincing same-chain paired comparisons; harness search relies on a proprietary meta-agent and the chain runs only once, so absolute numbers should be treated cautiously.
TerminalHorizon's automated synthesis of long-horizon terminal trajectories with connected behavioral patterns enables a 27B model to surpass prior methods using 10x less data, improving all 12 benchmarks.
Editor's noteAddresses a neglected issue in long-horizon terminal task synthesis: environment construction may prematurely complete or simplify the exploration/implementation/verification work the agent should perform. TerminalHorizon uses independent execution to identify key work omitted in construction, guides environment reconstruction accordingly, then progressively expands under the same objective to connect behavior patterns across stages—a substantive correction to 'creating long trajectories by adding rounds or complexity.' Validated on 12 benchmarks (including held-out Terminal-Bench 2.1), with open-sourced data and models (TerminalHorizon-3K/-27B); the paper is very new, and decontamination and evaluation protocols warrant independent review.
