Continue or Restart? The Limits of Early Failure Prediction in Coding Agents

Summary (Overview)

• Core finding: Early execution prefixes from coding agents provide near-chance discrimination between successful and failed attempts on the same software issue (within-task AUC ≈ 0.50), even though they modestly improve cross-task ranking.
• Comprehensive evaluation: The study mines 147,002 execution traces across four model-framework settings, comparing pre-execution issue context against early execution prefixes using linear count models, frozen text embeddings, task-conditional rankers, and a fine-tuned Qwen3-0.6B neural monitor.
• Paradox resolution: Learned restart policies can beat matched random restarting (up to +1.98 percentage points) despite near-chance within-task ranking—an exact algebraic decomposition shows this gain comes almost entirely from between-task stopping allocation, not attempt-level selection.
• Practical dominance of continuation: Across 216 of 224 tested budget-threshold configurations, uninterrupted continuation achieves higher solve rates at lower mean token cost than both learned and random cold restart policies.
• Key contribution: Establishes evaluation standards for early-stopping in coding agents, requiring both attempt-level discrimination metrics (within-task AUC) and net compute savings relative to continuation.

Introduction and Theoretical Foundation

Background and Motivation

Autonomous coding agents pair language models with execution scaffolds to navigate codebases, modify source files, and run verification tests. Evaluating early stopping—terminating unpromising runs before completion—promises to reduce computational costs. However, this requires separating two distinct factors:

1. Baseline task difficulty: Whether an issue is inherently solvable by the agent framework.
2. Trajectory quality: Whether a specific attempt on that issue is progressing well.

A key confounding issue: if an agent consistently resolves one issue and fails another, a monitor estimating static task difficulty achieves strong corpus-wide ranking even while assigning identical scores to every attempt on the same issue.

Theoretical Foundations

The paper distinguishes between two evaluation regimes:

• Pooled prediction: Ranks outcomes across all eligible runs (captures both cross-task difficulty and attempt-level differences).
• Within-task discrimination: Compares successful and failed attempts on the exact same task using area under the ROC curve (AUCwithin, where 0.50 = random ranking).

Two intervention regimes are considered:

• Abort-only termination: Abandons low-scoring runs without replacement (strictly dominated by pre-execution filtering).
• Restart-based stopping: Terminates a stalled attempt to reallocate budget to a fresh attempt on the same issue.

The central research questions (RQs) are:

• RQ1: How much does observing an early execution prefix improve outcome prediction beyond pre-execution task context?
• RQ2: Do early execution prefixes distinguish successful from failed attempts on the same task?
• RQ3: What drives the performance difference between learned and random restarting, and how do both compare with uninterrupted continuation under a shared token budget?

Methodology

Data and Settings

Four model-framework settings are analyzed:

(OH: OpenHands; SA: SWE-agent; ICC = intraclass correlation measuring within-task outcome clustering)

Key Definitions

• Task: A repository issue.
• Run/attempt: A single execution trajectory on a task.
• Decision landmark b: A fixed transcript-token threshold (5k, 10k, 15k, 20k, 30k, 40k, 60k tokens) at which a monitor evaluates an ongoing run.
• Eligible runs: Attempts with total length $L_i > b$ (still active at the landmark).

Evaluation Metrics

The within-task AUC is defined as:

$$\mathrm{AUC}_{\text{within}} = \frac{\sum_{t} \sum_{i \in P_t, j \in F_t} \left[ \mathbf{1}(s_i > s_j) + \frac{1}{2}\mathbf{1}(s_i = s_j) \right]}{\sum_{t} |P_t| |F_t|}, \tag{1}$$

where $P_t$ and $F_t$ denote sets of successful and failed active runs on task $t$, respectively.

Prediction Models

1. Issue baseline: 8 pre-execution count features from problem statements.
2. Issue-plus-prefix: Issue features + 16–18 behavioral count features from the observed prefix.
3. Task-conditional logistic models: Condition on task identity, estimating weights from within-task attempt differences.
4. Frozen text embeddings: All-MiniLM-L6-v2 over issue, initial assistant turn, and first tool observation.
5. Neural monitor: LoRA-adapted Qwen3-0.6B with paired sampling (one success + one failure from same task per training step).

Budget-Constrained Restart Replay

For a shared token cap $B$ with stopping at landmark $b = 20k$:

$$R_j = Y_j \mathbf{1}(L_j \leq B - b), \tag{2}$$ $$K_j = b + \min(L_j, B - b). \tag{3}$$

The replay solve rate across all $N$ starting attempts is:

$$V(\pi) = \frac{1}{N} \sum_{i=1}^{N} \left[ C_i + S_i^{\pi}(G_i - C_i) \right], \tag{4}$$

where $C_i$ is continuation success, $G_i$ is expected replacement credit, and $S_i^{\pi}$ is the stopping indicator.

The learned-minus-random differential decomposes exactly into between-task and within-task terms:

$$\Delta_{\mathrm{between}} = \sum_{t: n_t > 0} \frac{n_t}{N} \bar{D}_t \bar{v}_t, \tag{5}$$ $$\Delta_{\mathrm{within}} = \frac{1}{N} \sum_{t: n_t > 0} \sum_{i \in A_t} (D_i - \bar{D}_t)(v_i - \bar{v}_t). \tag{6}$$

Empirical Validation / Results

RQ1: Value of Observing an Early Prefix

• At 5k tokens: Prefix action counts increase pooled AUC from 0.56 → 0.58; log loss unchanged (0.69).
• At 20k tokens: Pooled AUC increases from 0.57 → 0.61.
• Frozen text embeddings provide no consistent pooled improvement over pre-execution issue text alone (0.62 in both arms at 5k tokens).

RQ2: Within-Task Discrimination

Count models and embeddings:

• At 5k tokens: within-task AUC = 0.50 [0.49, 0.51] (chance level).
• At 20k tokens: within-task AUC remains 0.49–0.50 across all four settings.
• At later landmarks: 0.52 (30k), 0.54 (40k), 0.56 (60k), but surviving fraction drops to 23% by 60k.

Task-conditional and neural models:

At 20k tokens, the task-conditional model achieves within-task AUC 0.50 [0.49, 0.51]. The fine-tuned Qwen3-0.6B monitor across 32,406 attempts achieves:

Stratification by task solvability (all intervals contain 0.50):

RQ3: Restart Policies Under Shared Budgets

Learned vs. random restart at 75k cap:

Exact decomposition (percentage points, 75k cap):

Continuation dominance: Across 224 configurations (4 caps × 7 thresholds × 4 settings × 2 policies), continuation Pareto-dominates 216 configurations. At the 75k cap, all 56 configurations are dominated (continuation has both higher solve rate and lower mean token cost).

Theoretical and Practical Implications

The Timing Bottleneck

The near-chance within-task discrimination at 20k tokens cannot be explained by absence of agent activity—87.51% of active runs have already executed tests and 32.80% have edited code. Rather, detecting an early test or edit doesn't establish whether it was diagnostic of eventual resolution.

Why Task-Level Selection Confounds Stopping Baselines

When a monitor captures static issue difficulty, score thresholding acts as an implicit task filter. Learned stopping concentrates aborts on hard tasks (where continuation rarely succeeds), avoiding the forfeited successes on tractable issues that random stopping incurs. This produces positive $\Delta_{\mathrm{between}}$ even with $\mathrm{AUC}_{\mathrm{within}} \approx 0.50$.

Design Recommendations

1. Pre-execution routing and budgeting: Static signals support up-front token provisioning and solver routing. Scaffold choice alone shifts solve rates by 14.06 percentage points (SWE-agent 47.30% vs. OpenHands 33.24% on shared tasks).
2. State-preserving recovery: Since cold prefixes lack attempt-level signal, retaining intermediate edits and test diagnostics is essential—explaining why state-preserving recovery dramatically outperforms cold restarts.

Evaluation Standards

Future studies should:

• Report within-task AUC across repeated attempts alongside pre-execution baselines and permutation controls.
• Compare proposed stopping rules against both matched random stopping and uninterrupted continuation.
• Decompose outcomes into between-task and within-task selection terms.

Conclusion

Early failure prediction benefits coding agents only when it informs abandoning an active attempt, not merely identifying difficult issues. Mining 147,002 traces, the study finds that early execution prefixes add minimal signal beyond static task context—all evaluated approaches yield within-task AUC ≈ 0.50 at early landmarks.

The apparent success of learned restarting over random baselines stems from between-task stopping allocation, not genuine attempt-level detection. Uninterrupted continuation achieves higher solve rates at lower mean token cost than cold restarts across essentially the entire tested budget frontier.

Future early-stopping and recovery methods should:

1. Demonstrate genuine attempt-level discrimination (within-task AUC > 0.50).
2. Show net compute savings relative to uninterrupted continuation.
3. Consider state-preserving recovery and pre-execution routing as more promising directions than cold restarts.