Summary (Overview)

• Central claim: Trajectory-based failure predictors for coding agents mostly predict which issue is hard and which model is running, not how the current run is going. Beyond issue difficulty and model ability, what predictors gain from the trajectory is statistically equivalent to zero.
• Three assumptions questioned: (i) high AUROC implies run-specific learning; (ii) the oracle gap reflects issue-specific strengths exploitable by switching; (iii) alarms mark bad runs that retries can fix. All three fail on real data.
• Key evidence: A run-blind issue difficulty score (never looking at the run) reaches 0.92 pooled AUROC, vs. ~0.62 for trajectory predictors. Knowing which model ran predicts within-issue failure better than trajectory predictors (0.72 vs. 0.60–0.61 AUROC).
• Switching evidence collapses: The oracle gap (10.0 pp with 7 models) is no larger than chance alone produces; per-issue model choice gains only +0.09 pp on held-out runs. Escalation gains (+6.91 pp) come from the extra attempt, not from choosing a better-suited model.
• Retrying has real room but predictors can't target it: Run-to-run variation accounts for 42% of outcome variance (vs. 4.6% for specialization), but current predictors mostly flag hard issues/weak models where retries rarely help. The paper calls for cause-aware failure prediction evaluated on repeated-run benchmarks and in closed-loop, equal-budget settings.

---

Introduction and Theoretical Foundation

Background

Software engineering (SE) agents resolve repository issues by inspecting code, editing files, and running tests. Runs are long and often fail. A growing line of work predicts failure mid-run and acts on it: stopping to save tokens, restarting the same model, or switching to another model. This paper focuses on trajectory-based predictors, which read only what the agent has produced so far (reasoning, commands, tool outputs) and thus apply to closed frontier models.

The Three Causes of Failure

A run can fail for three distinct reasons, each calling for a different action:

This decomposition follows item response theory (IRT), which explains each outcome by item difficulty and respondent ability, combined with observed run-to-run variation.

Three Assumptions Under Attack

The paper identifies three assumptions current practice takes for granted:

1. (i) A predictor with high AUROC has learned run-specific information
2. (ii) The oracle gap (any-model success minus best-single-model success) reflects issue-specific strengths that switching can exploit
3. (iii) An alarm marks a bad run, so acting on it (e.g., retrying) pays off

Toy World Demonstration

The authors show via a toy world (Fig. 1) where each outcome depends only on the issue, the model, and a coin flip that all three assumptions can fail simultaneously:

• (a) A score knowing only issue difficulty reaches 0.90 pooled AUROC, yet cannot rank two runs of the same issue (0.50)
• (b) With model A succeeding at 0.7 and model B at 0.6 on every issue, an "any-model" oracle solves $1 - (0.3 \times 0.4) = 0.88$ of issues — an 18 pp gap over A alone — despite A being better on every issue. Running A twice does even better (0.91): the gap comes from a second attempt, not a better-suited model.
• (c) 90% of failures occur on hard issues, where retry succeeds only 10% of the time; a difficulty score flags exactly those failures a retry rarely fixes.

---

Methodology

Data

• Primary data: mini-swe-agent runs on 500 issues of SWE-bench Verified (leaderboard-style), 3,492 runs from 7 models (Claude Haiku 4.5, Claude Opus 4.5/4.6, Gemini 3 Flash, GLM-5, GPT-5-mini, MiniMax-M2.5), 994 failures
• 23-model data: Companion release with outcomes of 23 models on the same issues; 16 outside models used only to estimate issue difficulty
• Repeated-run pilot: Public SWE-agent corpus with two Llama models (70B and 8B), each run ≥4 times on 522 issues from SWE-bench-extra and SWE-bench dev

Predictors

• Mirror the design of Fail-Fast: read trajectory after $t \in \{10, 20, 40\}$ completed steps, using either the last 8 steps (R8) or full prefix (ALL)
• Crucially: predictors see neither the issue text nor the model name — anything learned must come from the trajectory alone
• Models: TF-IDF logistic regression (with issue-grouped 5-fold CV) plus Qwen3-0.6B fine-tuned with LoRA; 18 configurations total
• Reported on 2,840 runs still running at step 20 ($t_{20}$)

Run-Blind References

Pair Decomposition

AUROC is decomposed by pair type:

• Cross-issue pairs (99.66% of all pairs): different issues
• Within-issue pairs: same issue, different models
• Consistent pairs: failed run comes from the overall weaker model
• Upsets: failed run comes from the stronger model

A run-tracking predictor should rank both consistent and upset pairs above chance; a predictor following model ability ranks consistent pairs correctly but upsets incorrectly.

Statistical Tests

• Equivalence test: 90% interval of AUROC gain must lie within ±0.02 to claim "adds nothing" (Holm-corrected over 50 comparisons)
• Conditional logit: holds issue difficulty fixed, tests whether score adds to model identity M
• No-specialization reference: Rasch model (one ability per model, one difficulty per issue) and 2PL model (also lets issues separate models more sharply) — simulates worlds where every issue ranks models identically

Repeated-Run Evaluation

• Split each model's runs per issue into halves A and B; A for choosing models, B for measuring success
• Average over 200 random splits; bootstrap issues for intervals

---

Empirical Validation / Results

RQ1: Predictors Learn Mostly the Issue and the Model

Issue difficulty explains pooled AUROC. Of 494,109 failed–successful pairs, 99.66% compare different issues. The run-blind difficulty score $D_{ext}$ reaches 0.924 pooled AUROC, while trajectory predictors reach only 0.62–0.63 — not significantly better than length + M.

Model ability explains within-issue AUROC. Within an issue, model identity M reaches 0.72 within-issue AUROC (0.82 with 23 models), above both trajectory predictors (0.60–0.61). Predictors score 0.68–0.69 on consistent pairs but only 0.42 on upsets — all ten TF-IDF configurations fall below 0.5 on upsets, revealing they follow the ability ranking rather than the run.

The trajectory adds nothing measurable beyond both. Adding predictor scores to $D_{ext}$ + M changes pooled AUROC by an amount equivalent to zero in all 18 configurations (e.g., −0.001 for the full-prefix predictor, 90% interval [−0.003, 0.000]). Within issues, conditional logit shows no help identifying which runs of an issue failed (e.g., −0.011 within-issue AUROC, 95% interval [−0.024, 0.002]).

Table I: AUROC by Type of Pair (2,840 runs at step 20)

RQ2: The Oracle Gap Is Not a Gain from Switching

The gap is not evidence of specialization. Best single model solves 76.8% of issues; some model solves 86.8% (gap = 10.0 pp; 12.4 pp with 23 models). Upsets occur in 24.6% of one-success pairs (17.7% with 23 models). However, none of the eight comparisons (oracle gap and upset rate × 7/23 models × Rasch/2PL) is significantly above the no-specialization reference after Holm correction (Fig. 2).

The gap is not an attainable gain. 45% of failures (448 of 994) occur on 64 issues no model solves — no model to switch to. Choosing per-issue from past runs gains only +0.09 pp on held-out runs; a router for new issues gains only +0.41 pp over the best single model.

Table II: Resolution Rate of Strategy A Minus That of B (pp)

Specialization is real but sparse. Upsets found in half A recur in half B at 44% (vs. 19% chance; p = 0.004), but only on ~15 of 522 issues. Variance decomposition: 53% issue difficulty, 42% run-to-run variation, only 4.6% specialization.

Escalation gains come from the extra attempt. Escalating every failure to one fixed second model gains +6.91 pp — but this policy makes no issue-specific choice, so the gain comes from the second attempt, not from picking a better-suited model.

RQ3: Alarms Miss the Runs a Retry Can Fix

Bad runs are common. Run-to-run variation accounts for 42% of outcome variance — the same model often solves an issue in some runs and fails it in others. Each such failure is a bad run with real retry potential.

Alarms cannot distinguish causes. Since predictors carry nothing measurable beyond issue difficulty and model identity (RQ1), and 64–82% of their score variance lies between issues rather than within them, a high score signals a hard issue or weak model — where retries rarely succeed. This is consistent with Fail-Fast, where a cold restart triggered by its predictor adds only 0.2 pp.

---

Theoretical and Practical Implications

Rethinking Failure Prediction

The paper demonstrates that three seemingly strong pieces of evidence — high AUROC, large oracle gap, and actionable alarms — can each arise from issue difficulty, model ability, or chance alone. The overlooked factor is the cause of failure, which determines the correct action.

Three Recommended Evaluation Practices

1. Report within-issue AUROC alongside pooled AUROC, compared with run-blind references (issue difficulty, model ability, length)
2. Compare oracle gaps with no-specialization references and evaluate routers on held-out runs
3. Judge alarms by whether the triggered action fixes the failure, comparing actions at equal numbers of attempts

Practical Consequences

• Model switching as currently motivated is largely illusory: the gap between best-single-model and any-model oracle is within chance range, and per-issue routing gains nothing on held-out runs
• Retrying has real room to repair failures (42% of variance is run-to-run), but current predictors cannot target bad runs — they flag hard issues and weak models instead
• Stopping to save tokens may be the only action current predictors genuinely support, since they do identify hard issues

Threats to Validity

• Evidence comes mainly from released labels on one benchmark and scaffold
• Trajectory-based predictors only; repeated-run pilot uses older models
• Router uses shallow issue features
• Non-rejection of the no-specialization reference does not prove specialization is absent (power not yet measured)

---

Conclusion

The paper's answer to its title question — "Hard issue, weak model, or bad run?" — is that current failure predictors mostly predict the issue and the model, not the run. The evidence for acting on their alarms, whether by switching or retrying, inherits the same confusion.

Future Plans

1. Collect repeated runs of recent models across more benchmarks and scaffolds to separate run-to-run variation from issue difficulty and model ability
2. Build cause-aware predictors that distinguish hard issue, weak model, and bad run, so alarms can trigger stopping, switching, or retrying appropriately
3. Evaluate in closed loop at equal budgets, counting false alarms and monitoring costs, judged by the decisions improved rather than pooled AUROC

The central call: failure prediction must explain why a run fails, so that each failure is met with the action that can fix it.