# Hard Issue, Weak Model, or Bad Run? Rethinking Failure Prediction for Coding Agents

> Trajectory-based failure predictors for coding agents predict issue difficulty and model identity, not run-specific failure, adding zero measurable signal beyond these run-blind factors.

- **Source:** [arXiv](https://arxiv.org/abs/2611.08285)
- **Published:** 2026-11-07
- **Permalink:** https://picx.dev/p/4uC55J
- **Whiteboard:** https://picx.dev/p/4uC55J/image

## Summary

## Summary (Overview)

- **Central claim**: Trajectory-based failure predictors for coding agents mostly predict *which issue* is hard and *which model* is running, not *how the current run* is going. Beyond issue difficulty and model ability, what predictors gain from the trajectory is statistically equivalent to zero.
- **Three assumptions questioned**: (i) high AUROC implies run-specific learning; (ii) the oracle gap reflects issue-specific strengths exploitable by switching; (iii) alarms mark bad runs that retries can fix. All three fail on real data.
- **Key evidence**: A run-blind issue difficulty score (never looking at the run) reaches 0.92 pooled AUROC, vs. ~0.62 for trajectory predictors. Knowing which model ran predicts within-issue failure better than trajectory predictors (0.72 vs. 0.60–0.61 AUROC).
- **Switching evidence collapses**: The oracle gap (10.0 pp with 7 models) is no larger than chance alone produces; per-issue model choice gains only +0.09 pp on held-out runs. Escalation gains (+6.91 pp) come from the *extra attempt*, not from choosing a better-suited model.
- **Retrying has real room but predictors can't target it**: Run-to-run variation accounts for 42% of outcome variance (vs. 4.6% for specialization), but current predictors mostly flag hard issues/weak models where retries rarely help. The paper calls for cause-aware failure prediction evaluated on repeated-run benchmarks and in closed-loop, equal-budget settings.

---

## Introduction and Theoretical Foundation

### Background

Software engineering (SE) agents resolve repository issues by inspecting code, editing files, and running tests. Runs are long and often fail. A growing line of work predicts failure mid-run and acts on it: stopping to save tokens, restarting the same model, or switching to another model. This paper focuses on **trajectory-based predictors**, which read only what the agent has produced so far (reasoning, commands, tool outputs) and thus apply to closed frontier models.

### The Three Causes of Failure

A run can fail for three distinct reasons, each calling for a different action:

| Cause | Description | Appropriate Action |
|-------|-------------|-------------------|
| **Hard issue** | Most models fail it | Stop or human intervention |
| **Weak model** | The model fails many issues | Switch to a stronger model |
| **Bad run** | This particular attempt went wrong | Retry |

This decomposition follows **item response theory** (IRT), which explains each outcome by item difficulty and respondent ability, combined with observed run-to-run variation.

### Three Assumptions Under Attack

The paper identifies three assumptions current practice takes for granted:

1. **(i)** A predictor with high AUROC has learned run-specific information
2. **(ii)** The oracle gap (any-model success minus best-single-model success) reflects issue-specific strengths that switching can exploit
3. **(iii)** An alarm marks a bad run, so acting on it (e.g., retrying) pays off

### Toy World Demonstration

The authors show via a toy world (Fig. 1) where each outcome depends only on the issue, the model, and a coin flip that all three assumptions can fail simultaneously:

- **(a)** A score knowing only issue difficulty reaches 0.90 pooled AUROC, yet cannot rank two runs of the same issue (0.50)
- **(b)** With model A succeeding at 0.7 and model B at 0.6 on every issue, an "any-model" oracle solves $1 - (0.3 \times 0.4) = 0.88$ of issues — an 18 pp gap over A alone — despite A being better on every issue. Running A twice does even better (0.91): the gap comes from a *second attempt*, not a better-suited model.
- **(c)** 90% of failures occur on hard issues, where retry succeeds only 10% of the time; a difficulty score flags exactly those failures a retry rarely fixes.

---

## Methodology

### Data

- **Primary data**: mini-swe-agent runs on 500 issues of SWE-bench Verified (leaderboard-style), 3,492 runs from 7 models (Claude Haiku 4.5, Claude Opus 4.5/4.6, Gemini 3 Flash, GLM-5, GPT-5-mini, MiniMax-M2.5), 994 failures
- **23-model data**: Companion release with outcomes of 23 models on the same issues; 16 outside models used only to estimate issue difficulty
- **Repeated-run pilot**: Public SWE-agent corpus with two Llama models (70B and 8B), each run ≥4 times on 522 issues from SWE-bench-extra and SWE-bench dev

### Predictors

- Mirror the design of **Fail-Fast**: read trajectory after $t \in \{10, 20, 40\}$ completed steps, using either the last 8 steps (R8) or full prefix (ALL)
- **Crucially**: predictors see *neither the issue text nor the model name* — anything learned must come from the trajectory alone
- **Models**: TF-IDF logistic regression (with issue-grouped 5-fold CV) plus Qwen3-0.6B fine-tuned with LoRA; 18 configurations total
- Reported on 2,840 runs still running at step 20 ($t_{20}$)

### Run-Blind References

| Reference | Definition |
|-----------|------------|
| Issue difficulty $D_{ext}$ | Fraction of 16 outside models that fail the issue (same score for all runs of an issue) |
| Model identity (M) | Score by the model's failure rate on training issues |
| Length + M | Adds only prefix length to M |

### Pair Decomposition

AUROC is decomposed by pair type:
- **Cross-issue pairs** (99.66% of all pairs): different issues
- **Within-issue pairs**: same issue, different models
- **Consistent pairs**: failed run comes from the overall weaker model
- **Upsets**: failed run comes from the stronger model

A run-tracking predictor should rank both consistent and upset pairs above chance; a predictor following model ability ranks consistent pairs correctly but upsets incorrectly.

### Statistical Tests

- **Equivalence test**: 90% interval of AUROC gain must lie within ±0.02 to claim "adds nothing" (Holm-corrected over 50 comparisons)
- **Conditional logit**: holds issue difficulty fixed, tests whether score adds to model identity M
- **No-specialization reference**: Rasch model (one ability per model, one difficulty per issue) and 2PL model (also lets issues separate models more sharply) — simulates worlds where every issue ranks models identically

### Repeated-Run Evaluation

- Split each model's runs per issue into halves A and B; A for choosing models, B for measuring success
- Average over 200 random splits; bootstrap issues for intervals

---

## Empirical Validation / Results

### RQ1: Predictors Learn Mostly the Issue and the Model

**Issue difficulty explains pooled AUROC.** Of 494,109 failed–successful pairs, 99.66% compare different issues. The run-blind difficulty score $D_{ext}$ reaches **0.924 pooled AUROC**, while trajectory predictors reach only **0.62–0.63** — not significantly better than length + M.

**Model ability explains within-issue AUROC.** Within an issue, model identity M reaches 0.72 within-issue AUROC (0.82 with 23 models), above both trajectory predictors (0.60–0.61). Predictors score 0.68–0.69 on consistent pairs but only 0.42 on upsets — all ten TF-IDF configurations fall *below* 0.5 on upsets, revealing they follow the ability ranking rather than the run.

**The trajectory adds nothing measurable beyond both.** Adding predictor scores to $D_{ext}$ + M changes pooled AUROC by an amount equivalent to zero in all 18 configurations (e.g., −0.001 for the full-prefix predictor, 90% interval [−0.003, 0.000]). Within issues, conditional logit shows no help identifying which runs of an issue failed (e.g., −0.011 within-issue AUROC, 95% interval [−0.024, 0.002]).

**Table I: AUROC by Type of Pair (2,840 runs at step 20)**

| | Pooled | Within-issue All | Consistent | Upset |
|---|---|---|---|---|
| **Run-blind references** | | | | |
| Issue difficulty ($D_{ext}$) | 0.924 | 0.500 | 0.500 | 0.500 |
| Model identity (M) | 0.567 | 0.724 | 1.000 | 0.000 |
| $D_{ext}$ + M | 0.929 | 0.718 | 0.869 | 0.317 |
| Length + M | 0.607 | 0.648 | 0.756 | 0.373 |
| **Trajectory predictors** | | | | |
| TF-IDF, last 8 steps (R8) | 0.624 | 0.604 | 0.678 | 0.415 |
| TF-IDF, full prefix (ALL) | 0.630 | 0.610 | 0.687 | 0.416 |

### RQ2: The Oracle Gap Is Not a Gain from Switching

**The gap is not evidence of specialization.** Best single model solves 76.8% of issues; some model solves 86.8% (gap = 10.0 pp; 12.4 pp with 23 models). Upsets occur in 24.6% of one-success pairs (17.7% with 23 models). However, **none of the eight comparisons** (oracle gap and upset rate × 7/23 models × Rasch/2PL) is significantly above the no-specialization reference after Holm correction (Fig. 2).

**The gap is not an attainable gain.** 45% of failures (448 of 994) occur on 64 issues *no* model solves — no model to switch to. Choosing per-issue from past runs gains only **+0.09 pp** on held-out runs; a router for new issues gains only **+0.41 pp** over the best single model.

**Table II: Resolution Rate of Strategy A Minus That of B (pp)**

| A vs. B | A - B [95% CI] |
|---|---|
| **One attempt: choose a model per issue?** | |
| Choice from the issue's past runs vs. one model | +0.09 [-0.36, 0.63] |
| Router for new issues vs. best single model | +0.41 [-1.83, 2.85] |
| **One vs. two attempts: escalate after failure?** | |
| Escalation vs. best single model | +6.91 [4.67, 9.15] |

**Specialization is real but sparse.** Upsets found in half A recur in half B at 44% (vs. 19% chance; p = 0.004), but only on ~15 of 522 issues. Variance decomposition: 53% issue difficulty, 42% run-to-run variation, only **4.6% specialization**.

**Escalation gains come from the extra attempt.** Escalating every failure to one fixed second model gains +6.91 pp — but this policy makes no issue-specific choice, so the gain comes from the second attempt, not from picking a better-suited model.

### RQ3: Alarms Miss the Runs a Retry Can Fix

**Bad runs are common.** Run-to-run variation accounts for 42% of outcome variance — the same model often solves an issue in some runs and fails it in others. Each such failure is a *bad run* with real retry potential.

**Alarms cannot distinguish causes.** Since predictors carry nothing measurable beyond issue difficulty and model identity (RQ1), and 64–82% of their score variance lies *between* issues rather than within them, a high score signals a hard issue or weak model — where retries rarely succeed. This is consistent with Fail-Fast, where a cold restart triggered by its predictor adds only 0.2 pp.

---

## Theoretical and Practical Implications

### Rethinking Failure Prediction

The paper demonstrates that three seemingly strong pieces of evidence — high AUROC, large oracle gap, and actionable alarms — can each arise from issue difficulty, model ability, or chance alone. **The overlooked factor is the cause of failure**, which determines the correct action.

### Three Recommended Evaluation Practices

1. **Report within-issue AUROC** alongside pooled AUROC, compared with run-blind references (issue difficulty, model ability, length)
2. **Compare oracle gaps with no-specialization references** and evaluate routers on held-out runs
3. **Judge alarms by whether the triggered action fixes the failure**, comparing actions at equal numbers of attempts

### Practical Consequences

- **Model switching** as currently motivated is largely illusory: the gap between best-single-model and any-model oracle is within chance range, and per-issue routing gains nothing on held-out runs
- **Retrying** has real room to repair failures (42% of variance is run-to-run), but current predictors cannot target bad runs — they flag hard issues and weak models instead
- **Stopping** to save tokens may be the only action current predictors genuinely support, since they do identify hard issues

### Threats to Validity

- Evidence comes mainly from released labels on one benchmark and scaffold
- Trajectory-based predictors only; repeated-run pilot uses older models
- Router uses shallow issue features
- Non-rejection of the no-specialization reference does not prove specialization is absent (power not yet measured)

---

## Conclusion

The paper's answer to its title question — "Hard issue, weak model, or bad run?" — is that current failure predictors **mostly predict the issue and the model, not the run**. The evidence for acting on their alarms, whether by switching or retrying, inherits the same confusion.

### Future Plans

1. **Collect repeated runs** of recent models across more benchmarks and scaffolds to separate run-to-run variation from issue difficulty and model ability
2. **Build cause-aware predictors** that distinguish hard issue, weak model, and bad run, so alarms can trigger stopping, switching, or retrying appropriately
3. **Evaluate in closed loop at equal budgets**, counting false alarms and monitoring costs, judged by the decisions improved rather than pooled AUROC

The central call: failure prediction must explain *why* a run fails, so that each failure is met with the action that can fix it.

---

_Markdown view of https://picx.dev/p/4uC55J, served by PicX — AI-generated visual whiteboard summaries of research papers._