FC-SWE: Failure-Conditioned RL for Long-Horizon Software Engineering Agents

Summary (Overview)

• FC-SWE is a novel failure-conditioned reinforcement learning framework for repository-level software engineering (SWE) agents that incorporates post-failure recovery trajectories into policy training, going beyond standard independent-attempt RL approaches.
• The framework introduces two key mechanisms: trajectory-local reward assignment (preserving each attempt's own verifier outcome) and active-set advantage estimation (comparing only actually-executed trajectories for the same issue).
• On all 500 SWE-bench Verified tasks with Qwen3.5-4B, FC-SWE achieves 41.7% Resolved@1 and 52.8% Resolved@2, outperforming GRPO (38.9% and 48.5%) and the Base policy (27.9% and 36.3%).
• Despite training with at most two attempts per chain, FC-SWE scales to 70.7% Resolved@11 under an eleven-attempt test-time budget, surpassing GRPO (67.3%).
• Controlled ablations show FC-SWE uniquely benefits from verifier feedback (19.0% conditional recovery with feedback vs. 17.0% with resampling), reversing the pattern observed for Base and GRPO policies.

Introduction and Theoretical Foundation

Background and Motivation

Repository-level software engineering (SWE) is a challenging long-horizon setting where agents must:

• Reason over extended interactions
• Use tools across multiple turns
• Adapt to observations from stateful environments

Recent work applies reinforcement learning (RL) to SWE using execution-based pipelines where the sequence of model outputs, tool calls, and environment observations forms a complete trajectory culminating in a candidate code patch. A test-suite verifier evaluates the patch, and its outcome serves as a terminal reward for policy optimization.

Key Insight

The authors identify a critical limitation of standard RL approaches (e.g., GRPO): training only on independent attempts does not explicitly teach the policy to use diagnostic information from failed verification in subsequent recovery attempts. A failed verification provides valuable evidence—failing tests, error messages, and tracebacks—that can guide recovery.

Theoretical Challenge

Incorporating recovery into RL training is non-trivial because:

1. A recovery trajectory is generated only after a preceding trajectory fails
2. Its context is constructed from the failed patch and verifier feedback
3. Preceding executions determine both which recovery trajectories enter training and under what contexts they are generated
4. Explicit decisions are needed about reward assignment and comparison groups for advantage estimation

Methodology

Problem Setup

For a software issue $x$, an attempt is one complete agent run beginning from the issue's original repository state and ending after its submitted patch is evaluated by the verifier. Each executed attempt produces one trajectory $\tau_c$ with binary verifier outcome $y_c \in \{0, 1\}$.

GRPO Baseline

The GRPO baseline independently samples a fixed group of $G$ trajectories for the same issue, combining GRPO's critic-free group-relative policy update with a leave-one-out reward baseline:

$$b_{c}^{\mathrm{GRPO}} = \frac{1}{G-1} \sum_{d \neq c} y_{d}, \qquad s_{x, -c} = \mathrm{Std}(\{y_{d}: d \neq c\})$$ $$A_{c}^{\mathrm{GRPO}} = \begin{cases} \frac{y_{c} - b_{c}^{\mathrm{GRPO}}}{s_{x, -c} + \epsilon}, & s_{x, -c} > 0, \\ y_{c} - b_{c}^{\mathrm{GRPO}}, & s_{x, -c} = 0, \end{cases}$$

where $b_c^{\mathrm{GRPO}}$ and $s_{x,-c}$ are the mean and sample standard deviation of the other $G-1$ trajectory rewards, and $\epsilon = 10^{-6}$.

Outcome-Dependent Recovery Chains

FC-SWE extends this to chains of attempts. For each issue $x$, each of the $G$ initially sampled trajectories seeds one recovery chain indexed by $c \in \{1, \ldots, G\}$. Chain $c$ contains at most $R$ attempts. The activity indicator $q_{c,r}$ evolves recursively:

$$q_{c,1} = 1, \qquad q_{c,r} = \begin{cases} 1 - y_{c,r-1}, & q_{c,r-1} = 1, \\ 0, & q_{c,r-1} = 0, \end{cases} \quad r > 1$$

Thus, a recovery attempt is executed if and only if the preceding attempt was executed and failed. The main configuration uses $G = 8$ and $R = 2$.

Recovery Context Construction

When an executed attempt $r < R$ fails, the repository is restored to its original state, and the context for the next attempt is:

$$h_{c,r+1} = \mathcal{C}(x, \mathrm{patch}(\tau_{c,r}), \mathrm{verify}(\tau_{c,r}))$$

where $\mathcal{C}$ is a context-construction operator that formats and truncates the original issue and failure evidence to a fixed context budget.

Trajectory-Local Verifier Rewards

Each verifier outcome is assigned only to the trajectory that produced the corresponding patch:

$$r_{c,r} = \begin{cases} y_{c,r}, & q_{c,r} = 1, \\ \bot, & q_{c,r} = 0, \end{cases}$$

where $\bot$ denotes an attempt position that was never executed. For $R=2$, possible reward sequences are:

• [1, ⊥] after initial success
• [0, 1] after successful recovery
• [0, 0] after two failures

This prevents a later recovery success from incorrectly assigning positive reward to an earlier failed patch.

Active-Set Advantage Estimation

For each issue $x$, all executed trajectories are pooled:

$$\mathcal{V}_x = \{(c, r): q_{c,r} = 1\}$$

The baseline for trajectory $i = (c,r) \in \mathcal{V}_x$ is:

$$b_{i} = \frac{1}{|\mathcal{V}_x| - 1}\sum_{\substack{j\in \mathcal{V}_x\\ j\neq i}}r_{j}$$

Advantages are obtained by normalizing $r_i - b_i$ using the sample standard deviation of the comparison rewards, with the zero-variance fallback from Equation 2. Inactive rows enter neither reward statistics nor the loss.

Policy Optimization

Trajectory advantages are broadcast to valid policy-generated tokens and optimized with a clipped policy-gradient objective, normalized by total valid generated tokens. Prompts, environment observations, copied failure context, and padding receive zero loss weight.

Empirical Validation / Results

Main Results (Table 1)

Results on all 500 SWE-bench Verified tasks, averaged over three decoding seeds:

FC-SWE improves over GRPO by 2.9 and 4.3 percentage points on Resolved@1 and Resolved@2, respectively.

Training Ablations (Table 2)

Failure-Evidence Ablations

Feedback vs. resampling at matched two-attempt budget:

• Base: 11.6% (feedback) vs. 13.7% (resampling)
• GRPO: 15.7% (feedback) vs. 18.6% (resampling)
• FC-SWE: 19.0% (feedback) vs. 17.0% (resampling) — reverses the pattern

Context ablation (FC-SWE policy):

• No failure context: 17.0% conditional recovery
• Patch only: 15.7%
• Patch + verifier evidence: 19.0% (3.3-point improvement over patch-only)

Test-Time Scaling

• Training horizon: $K=2$ attempts
• At $K=11$: FC-SWE reaches 70.7% vs. 67.3% for GRPO and 61.1% for Base
• Token efficiency at 50% resolution: FC-SWE uses ~3.15M tokens/task vs. 4.39M (GRPO) and 8.22M (Base)

Theoretical and Practical Implications

Theoretical Contributions

1. Outcome-dependent recovery chains: FC-SWE formalizes the notion that recovery trajectories exist only after failure, making the training data distribution itself policy-dependent.

2. Trajectory-local rewards: The framework demonstrates that preserving individual verifier outcomes prevents credit-assignment errors where later successes would incorrectly reward earlier failures.

3. Active-set advantage estimation: By pooling only executed trajectories, FC-SWE maintains meaningful comparisons across attempts while excluding unexecuted recovery slots. The authors note this update "does not inherit the standard unbiasedness guarantee of an action-independent baseline."

Practical Implications

1. Data efficiency: FC-SWE extracts more learning signal from failed attempts by converting them into recovery training data, rather than discarding them.

2. Verifier feedback as training signal: The framework demonstrates that diagnostic verifier output (failing tests, error messages) can be productively used as conditioning context during RL training.

3. Test-time scaling: Training with short chains (R=2) transfers to much longer test-time budgets (K=11), suggesting that learning how to recover generalizes beyond the training horizon.

4. Model-scale independence: Improvements are demonstrated on a 4B-parameter model, suggesting the approach may be complementary to scaling.

Conclusion

FC-SWE introduces a failure-conditioned RL framework that trains SWE agents to recover from failed patches using:

• Trajectory-local rewards to preserve individual verifier outcomes
• Active-set advantage estimation to compare executed trajectories for the same issue

Key findings:

1. FC-SWE improves both initial solving (Resolved@1: 41.7% vs. 38.9% for GRPO) and recovery (conditional recovery: 19.0% vs. 15.7%)
2. Training with failure context is essential—removing it reduces Resolved@2 by 4.9 percentage points
3. The policy uniquely benefits from verifier feedback at inference time, reversing the pattern where feedback hurts Base and GRPO policies
4. Despite training with at most two attempts, the policy scales to 70.7% resolution with eleven test-time attempts

Future work: The authors note that recovery with deployment-time verifiers remains an open direction.