FailForge: Distilling Procedural Competence from Persistent Failures into Code Agents

Summary (Overview)

• Core contribution: FailForge is an agentic framework that converts failed software engineering (SWE) trajectories—normally discarded by rejection sampling fine-tuning (RFT)—into training signal by inducing transferable procedural skills and using them to guide recovery attempts.
• Key mechanism: For each persistently failed instance, a diagnosing agent produces a high-level, leakage-filtered procedural skill; this skill guides a second attempt that succeeds, after which the skill is removed at training time so the student internalizes the procedure without inference-time hints.
• Main results: FailForge recovers 26.2% of previously failed instances (218/832) at marginal additional cost, improves SWE-bench Verified pass@1 by 6.6 points (from 59.6 to 66.2) over a strong RFT baseline, with gains concentrated on the hardest problems.
• Transferability: Recovered supervision transfers across model scales (4B and 9B), evaluation domains (SWE-bench Verified and MultiLingual), and agent harnesses (OpenHands and Claude Code), and induces more systematic investigation behavior.
• Efficiency: FailForge achieves the lowest token cost per point of pass@1 improvement (2.12B tokens/point) compared to additional sampling (4.32B) and instance-level hints (3.01B).

Introduction and Theoretical Foundation

Background and Motivation

Large language model (LLM) agents have made rapid progress on real-world software engineering tasks, autonomously navigating repositories, editing code, and validating fixes against test suites. Training such agents increasingly relies on rejection sampling fine-tuning (RFT), where the model samples multiple trajectories per task, retains those that pass execution-based verification (e.g., unit tests), and fine-tunes on the successful rollouts.

However, RFT has a structural inefficiency: tasks on which every sampled trajectory fails contribute nothing to learning. These persistently failed tasks are not random—they sit precisely at the model's capability frontier, where learning signals would be most valuable. The authors observe that:

"Persistent failures reflect missing capabilities rather than merely sampling variance."

Scaling the per-task sampling budget from 5× to 10× recovers only a small fraction of failed tasks (14.3%), confirming this claim.

Key Theoretical Insight

The central observation is that many persistent failures stem not from missing knowledge about the specific instance, but from the absence of transferable procedural competence—the reusable strategy for how to investigate a problem. This includes:

• How to localize a fault across a large repository
• How to read a complex test harness
• How to structure an efficient investigation

The authors distinguish this from instance-level hints (which point to a specific bug) and step-level guidance (which is fragile on long-horizon tasks where sparse outcome feedback makes individual action attribution difficult).

Formal Problem Definition

The training setup uses a teacher policy $\pi_T$ and a distinct student policy $\pi_\theta$ fine-tuned on verified trajectories. Let $\mathcal{D} = \{x_i\}_{i=1}^{M}$ be the task pool, where each $x_i$ pairs a repository snapshot and a natural-language issue with a hidden test suite $T_i$. A rollout yields a trajectory $\tau$ and final patch, with binary reward:

$$R(x_i, \tau) \in \{0, 1\}$$

Standard RFT retains successful pairs $\mathcal{C} = \{(x_i, \tau) \mid R(x_i, \tau) = 1\}$ and discards the persistently failed set:

$$\mathcal{F} = \left\{x_i \in \mathcal{D} \mid R(x_i, \tau_i^{(k)}) = 0 \ \forall k \in \{1, \dots, K\}\right\}$$

The goal is to recover trainable supervision from $\mathcal{F}$ for an unconditional student that samples from $\pi_\theta(\cdot \mid x_i)$ with no auxiliary context at test time.

Methodology

FailForge operates via a five-stage recovery pipeline over the persistently failed set $\mathcal{F}$:

1. Naive RFT and Failure Identification

Sample $K=5$ teacher trajectories per task, verify against test suites, forming the initial corpus $\mathcal{C}$ of successes and the persistently failed set $\mathcal{F}$ (832 tasks out of 2,401).

2. Agentic Skill Induction

For each failed instance $x_i \in \mathcal{F}$, a diagnosing agent $\mathcal{A}$ produces a skill:

$$s_i = \mathcal{A}\left(x_i, \{\tau_i^{(k)}, o_i^{(k)}\}_{k=1}^{K}, p_i\right)$$

where $\tau_i^{(k)}$ are failed trajectories, $o_i^{(k)}$ their test outputs and reasoning traces, and $p_i$ the gold patch (available as privileged supervision in offline SWE data construction).

The skill follows a fixed four-part schema:

1. Applicability condition: when the skill is relevant
2. Key principles: invariants a correct fix must respect
3. Common pitfalls: likely causes of similar failures
4. Recommended approach: ordered investigation and repair procedure

Crucially, skills are phrased at the level of methodology (e.g., "cross-module identifier consistency," "module import visibility") without referring to the specific repository or bug.

3. Leakage Filtering

A separate judge model $\mathcal{J}$ screens each skill for solution-specific leakage:

$$\mathcal{J}(s_i, p_i) = \begin{cases} 1 & \text{if } s_i \text{ leaks implementation detail of } p_i, \\ 0 & \text{otherwise} \end{cases}$$

The judge applies a reconstruction test: a skill leaks if a reader could reconstruct the gold patch from it. Skills failing the filter are regenerated. The judge achieves 96.7% agreement with human consensus (Cohen's $\kappa = 0.805$), with 0.875 precision and 0.778 recall on the leakage class.

4. Skill-Guided Re-Rollout

Each failed instance is re-attempted with the retained skill injected into the teacher's context:

$$\tilde{\tau}_i \sim \pi_T(\cdot \mid x_i \oplus s_i)$$

where $\oplus$ denotes context injection (e.g., writing into CLAUDE.md or system prompt). Trajectories passing verification form the recovered corpus:

$$\tilde{\mathcal{C}} = \left\{(x_i, \tilde{\tau}_i) \mid x_i \in \mathcal{F}, R(x_i, \tilde{\tau}_i) = 1\right\}$$

5. Skill Removal and Training

The skill enters only at synthesis time. The removal operator $\rho$ strips the injected skill from the recovered trajectory but leaves the agent's own actions and reasoning untouched. The student is trained on the union of original and decontaminated recovered corpora:

$$\mathcal{L}(\theta) = -\sum_{(x_i, \tau) \in \mathcal{C}} \log \pi_\theta(\tau \mid x_i) - \sum_{(x_i, \tilde{\tau}_i) \in \tilde{\mathcal{C}}} \log \pi_\theta(\rho(\tilde{\tau}_i) \mid x_i)$$

This distills the teacher's skill-conditioned success $\pi_T(\cdot \mid x_i \oplus s_i)$ into a skill-free student $\pi_\theta(\cdot \mid x_i)$.

Empirical Validation / Results

Experimental Setup

• Training data: SWE-Gym (2,401 Python task instances with executable environments)
• Evaluation: SWE-bench Verified (500 instances) and SWE-bench MultiLingual
• Teacher: Kimi-K2.6 with OpenHands scaffold; Leakage filter: GPT-5.4; Students: Qwen3.5-4B and Qwen3.5-9B
• Repository separation: Training repositories disjoint from evaluation benchmarks

Main Results (OpenHands Harness)

| Benchmark | Method | Qwen3.5-4B pass@1 | Qwen3.5-4B pass@3 | Qwen3.5-4B pass@5 | Qwen3.5-9B pass@1 | Qwen3.5-9B pass@3 | Qwen3.5-9B pass@5 | |---|---|---|---|---|---|---| | SWE-Bench Verified | Base | 54.0 | 68.8 | 72.0 | 61.2 | 71.4 | 74.6 | | | RFT (5×) | 59.6 | 70.4 | 73.0 | 63.6 | 73.8 | 75.4 | | | RFT (10×) | 63.0 | 71.6 | 75.6 | 66.4 | 72.4 | 75.6 | | | RFT + Hint (5×) | 63.8 | 71.4 | 73.4 | 65.2 | 73.0 | 74.4 | | | FAILFORGE | 66.2 | 73.4 | 76.0 | 67.2 | 74.0 | 76.0 | | SWE-Bench MultiLingual | Base | 35.7 | 53.0 | 56.3 | 47.0 | 62.0 | 65.0 | | | RFT (5×) | 52.3 | 62.3 | 65.3 | 59.7 | 66.7 | 70.7 | | | RFT (10×) | 55.7 | 67.0 | 70.0 | 61.3 | 69.0 | 72.7 | | | RFT + Hint (5×) | 51.7 | 64.3 | 67.3 | 60.7 | 69.3 | 71.3 | | | FAILFORGE | 55.7 | 67.0 | 70.7 | 61.3 | 71.0 | 74.0 |

Key findings:

• FailForge improves pass@1 by +6.6 points over RFT on Verified (4B model), with gains at every $k$.
• Additional sampling (10×) reaches only 63.0 pass@1, confirming persistent failures are not sampling variance.
• Instance-level hints recover more trajectories (352 vs. 218) but yield a weaker student (63.8 vs. 66.2 pass@1)—recovery count alone doesn't determine supervision quality.

Cost Efficiency

FailForge achieves the lowest token cost per point of pass@1 improvement (2.12B vs. 3.01B for hints, 4.32B for sampling).

Data Quality at Fixed Budget

Holding total training trajectories constant and replacing a fixed portion of RFT corpus with FailForge-recovered trajectories raises pass@1 from 59.6 to 63.8, isolating per-sample quality gains.

Cross-Harness Transfer (Claude Code Evaluation)

| Benchmark | Method | Qwen3.5-4B pass@1 | Qwen3.5-4B pass@3 | Qwen3.5-4B pass@5 | Qwen3.5-9B pass@1 | Qwen3.5-9B pass@3 | Qwen3.5-9B pass@5 | |---|---|---|---|---|---|---| | SWE-Bench Verified | Base | 47.6 | 65.0 | 69.0 | 60.6 | 75.0 | 77.2 | | | RFT (5×) | 59.6 | 75.0 | 78.0 | 61.0 | 75.2 | 77.2 | | | RFT (10×) | 59.4 | 72.2 | 77.0 | 61.4 | 75.0 | 78.2 | | | RFT + Hint | 60.0 | 74.2 | 78.2 | 62.2 | 74.6 | 77.4 | | | FAILFORGE | 62.6 | 74.0 | 78.4 | 63.0 | 75.8 | 78.6 | | SWE-Bench MultiLingual | Base | 39.0 | 54.7 | 59.3 | 48.0 | 68.0 | 74.7 | | | RFT (5×) | 41.0 | 60.3 | 65.3 | 49.7 | 66.0 | 69.3 | | | RFT (10×) | 47.7 | 62.0 | 65.7 | 48.0 | 66.7 | 70.3 | | | RFT + Hint | 45.0 | 63.3 | 68.0 | 50.7 | 67.0 | 71.0 | | | FAILFORGE | 50.3 | 65.3 | 68.3 | 51.0 | 66.0 | 71.3 |

FailForge attains the best pass@1 on both benchmarks under Claude Code, indicating the recovered investigation procedure transfers across harnesses.

Ablation Studies

Component ablation (SWE-bench Verified, pass@1):

• Skill removal (SR) is critical: retaining the skill in training drops pass@1 to 64.6, as the student leans on the supplied strategy rather than acquiring it.
• Leakage filtering (LF) matters: without it, skills drift toward solution-bound guidance, lowering pass@1 to 63.6 (close to the hint baseline).

Skill-synthesis design choices:

• Withholding the gold patch drops pass@1 to 64.2—the reference solution provides useful diagnostic context.
• Rewriting reasoning traces to remove references to the injected skill (trace rewrite) drops pass@1 to 62.0—aggressive rewriting distorts the investigation behavior the student should imitate.

Cross-Instance Transferability

Within-repository guidance transfer experiments show:

• Matched hints recover more instances than matched skills (tailored to the target problem).
• After shuffling to sibling instances, skills retain 59.3% of their matched recovery vs. only 20.8% for hints.

This indicates skills capture reusable investigation procedures rather than solution-specific information.

Agent Behavioral Analysis

FailForge achieves the highest localization precision (especially on unsolved instances), highest verification rate (97.2% test before editing), and fewest edits—indicating a more systematic investigation procedure.

Taxonomy of Extracted Skills

All 832 induced skills classify into nine root-cause categories plus Other. The four largest categories cover 72.8% of skills:

1. Test-contract grounding
2. Fix placement
3. Change propagation
4. Root-cause tracing

These recurring procedural capabilities align with the localization gap observed in behavioral analysis. Independent classifications by three models achieve Fleiss' $\kappa = 0.800$.

Theoretical and Practical Implications

Theoretical Implications

• Failure as signal, not noise: The paper challenges the standard RFT paradigm that discards failed trajectories, arguing that persistent failures sit precisely at the capability frontier where learning signals are most valuable.
• Procedural vs. declarative knowledge: The work distinguishes transferable procedural competence (how to investigate) from instance-specific declarative knowledge (where the bug is), showing that the former is the missing ingredient in persistent failures.
• Latent guidance distillation: The skill-removal mechanism introduces a novel training paradigm where a latent guidance variable steers the teacher into high-reward regions of trajectory space, then is stripped so the student must internalize the procedure in weights rather than copy visible hints.

Practical Implications

• Cost-effective supervision recovery: FailForge recovers 26.2% of previously failed instances at the lowest token cost per performance point, making it practical for large-scale training pipelines.
• Data quality over quantity: Recovered trajectories carry more useful supervision per sample than original RFT trajectories, suggesting that data curation strategies should prioritize procedural richness over raw volume.
• Cross-domain robustness: Gains persist across model scales, languages, and agent harnesses, indicating the recovered procedures are genuinely transferable rather than overfit to the training distribution.

Conclusion

FailForge presents a framework that converts failed software engineering instances into training supervision by inducing transferable procedural skills for guided recovery. At a matched budget, it recovers nearly twice as many instances as additional sampling, improves SWE-bench Verified by 6.6 points over a strong RFT baseline, and induces more systematic and transferable problem-solving behaviors.

Key takeaways:

1. Persistent failures reflect missing procedural competence, not sampling variance—additional sampling alone cannot address them.
2. Procedural skills outperform instance-level hints as guidance for recovery, despite recovering fewer instances.
3. Skill removal at training time is essential for internalizing the procedure rather than relying on external hints.
4. The recovered supervision transfers across models, benchmarks, and harnesses, and improves per-sample training value.

Future directions suggested by this work include:

• Applying FailForge to other domains with verifiable outcomes (e.g., mathematical reasoning, scientific discovery)
• Investigating whether skills can be composed or hierarchically organized for more complex tasks
• Exploring whether the leakage-filtered skill induction can be made fully automatic without gold-patch supervision