# Benchmarking the Residual: What Long-Horizon Evaluations Add Beyond Matched Short-Task Performance

> The horizon residual, a log-ratio of composed short-stage success to end-to-end rollout success, diagnoses true long-horizon failures by separating them from ordinary error compounding.

- **Source:** [arXiv](https://arxiv.org/abs/2607.27283)
- **Published:** 2026-09-05
- **Permalink:** https://picx.dev/p/EKzrU4
- **Whiteboard:** https://picx.dev/p/EKzrU4/image

## Summary

## Summary (Overview)

- **Core Contribution**: This position paper argues that raw performance declines on long-horizon benchmarks are insufficient to diagnose "long-horizon failures" or specific mechanisms like context rot. The authors propose a systematic methodology to distinguish genuine long-horizon degradation from ordinary error compounding.
- **Key Proposal**: The paper introduces the **horizon residual** $\Gamma_{\mathrm{H}}$, defined as the log-ratio between an expected success rate composed from matched short-stage evaluations and the observed success rate in a natural end-to-end rollout. This provides a principled, auditable diagnostic comparison.
>
- **Methodological Framework**: The authors advocate a pre-registered, agent-configuration-matched counterfactual protocol involving: (1) decomposing long tasks into verifiable semantic stages, (2) measuring conditional stage success from declared checkpoints, (3) composing these into a product baseline prediction, and (4) comparing against natural rollout performance with targeted interventions to explain any mismatch.
>
- **Key Findings from Literature Review**: A systematic review of 9+ representative benchmarks (SWE-EVO, ChainSWE, SWE-Milestone, HORIZON, etc.) shows that no existing study implements the full proposed sequence—each lacks at least one critical component (e.g., composed expected prediction, state compatibility handling, targeted interventions, or protocol sensitivity analysis).
>
- **Practical Guidance**: The paper provides a detailed 8-step benchmark design protocol for coding and terminal agents, includes uncertainty quantification via the delta method and bootstrap, and articulates falsification criteria and an empirical research agenda to validate the framework's utility.

## Introduction and Theoretical Foundation

### Background and Motivation

Language-model agents are increasingly evaluated on tasks extending beyond single responses, spanning repository-level issue resolution, release-sized evolution, chained package upgrades, and continuous milestone execution. Across these settings, success generally declines as the amount of work, inter-stage dependence, and accumulated execution history increase—a trend critical for deployment decisions.

>
However, the phrase "long horizon" conflates several distinct quantities:

>
- The minimum number of effective actions required;
- The number of separately verifiable milestones;
- The depth of the dependency chain;
- The duration of the agent's rollout; or
- The amount of prior history visible to the model.

These quantities often co-vary in naturally collected tasks but need not, obscuring what a benchmark actually measures.



### Theoretical Foundation: Three Confounded Effects

The paper identifies three distinct sources of performance degradation that are easily conflated:
>
**Table 1: Distinct sources of performance degradation**

| Effect | What changes? | Useful control |
|---|---|---|---|
| Error compounding | Local stages are no harder, but more must succeed. | Measure each matched stage and compose estimated success probabilities. |
| Harder local work | Longer tasks contain more ambiguous, specialized, or poorly verified stages. | Match the distribution of local stage difficulty. |
| History dependence | Later steps become harder as context, state, or earlier errors accumulate. | Compare retained history with reset, compressed, or repaired history. |

The core theoretical insight is that **trajectory-induced degradation** (accumulated execution—transcript, tool outputs, environment state, or earlier errors—making later work harder) must be distinguished from mere task size. The narrower case where growing visible text causes decline is termed **context rot** [Hong et al., 2025].

### Motivating Evidence

Several empirical results illustrate why these categories must be separated:

- **Controlled history manipulation** [Sinha et al.,, 2026]: Holding evaluation turn and history format fixed while varying prior error rates shows cleaner histories improve subsequent accuracy—demonstrating content matters beyond length..
- **ChainSWE** [Jin et al,,, 2026]: Per-bug accuracy falls from 58.9% (oracle-correct prior state) to 36.5% (agent-generated prior state) across 100 chains, with 318 of 663 downstream sequential failures occurring on bugs solvable from oracle state.
—>
- **SWE-Milestone** [Deng et al,,,, 2026]: Independent milestone scores reach ~80%+, while the best continuous-evolution score is only 38.03%, showing reset-state versus persistent-rollout protocols estimate materially different quantities.

>

## Methodology

### Decomposition and Checkpoints

The proposed methodology begins with a **pre-specified decomposition** of the long task into verifiable semantic stages (e.g., "find the bug", "change the data model", "update the API", "pass integration tests"). Each stage is defined by an acceptance condition rather than an arbitrary number of turns or tool calls. The **deployed agent**—the complete system including model, prompt, tools, memory, and execution framework—is then run from **declared checkpoints**.



### The Product Baseline

Let $S_i$ denote success at stage $i$, and let $I$ denote a pre-specified checkpoint protocol (environment state, visible history, revealed information, local budget). Along a fixed path where every stage is required, define:

$$q_i = \mathrm{Pr}(S_i = 1 \mid S_{<i} = 1, I)$$

The chain rule gives the auditable baseline:


**Equation (1)** 
$$P_{\mathrm{expected}} = \prod_{i} q_{i}.$$


*Example*: If an agent has a 90% chance of passing Stage 1 and an 80% chance of passing Stage 2 from clean, compatible states, the predicted overall success is 72% (0.90 × 0.80). If actual success is only 30%, the intuitive shortfall is 42 percentage points.

### Three Baseline Forms

1. **Product baseline**: Multiplies conditional stage-success rates when checkpoints form one compatible path..
2. **Oracle-scaffolded baseline**: Multiplies success rates from separately chosen, clean canonical states; requires extra compatibility and independence assumptions.
3. **Checkpoint simulator**: Needed when solutions branch, failures can be retried/recovered, or one stage changes state and odds of later stages.



### The Horizon Residual

Let $P_{\mathrm{observed}}$ denote success under the natural end-to-end rollout. The horizon residual is defined as:


**Equation (2)** 
$$\Gamma_{\mathrm{H}} = \log \frac{P_{\mathrm{expected}}}{P_{\mathrm{observed}}},$$

where log denotes the natural logarithm, so $\Gamma_{\mathrm{H}}$ is measured in nats. Interpretation relative to the specified counterfactual:



- $\Gamma_{\mathrm{H}} > 0$: lower success in natural rollout than the model predicts;
- $\Gamma_{\mathrm{H}} \approx 0$: agreement at the resolution supported by the data;
- $\Gamma_{\mathrm{H}} < 0$: higher success in natural rollout than the model predicts.



### Stage-wise Decomposition

When the natural rollout can be evaluated at the same semantic boundaries, the contrast admits a useful decomposition. Let $r_i = \mathrm{Pr}(S_i = 1 \mid S_{<i} = 1,,N)$ denote conditional stage success under the natural-rollout protocol $N$. Then:


**Equation (3)** 
$$P_{\mathrm{observed}} = \prod_{i} r_{i}, \qquad \Gamma_{\mathrm{H}} = \sum_{i} \log \frac{q_{i}}{r_{i}}.$$

This breakdown shows which step had the largest drop relative to baseline, though it identifies where the mismatch surfaces, not necessarily where it originates.



### Uncertainty Quantification

**Equation (4)** — Delta method approximation (ignoring covariance):
$$\mathrm{Var}(\widehat{\Gamma_{\mathrm{H}}}) \approx \sum_{i} \frac{1 - q_{i}}{m_{i} q_{i}} + \frac{1 - P_{\mathrm{observed}}}}{m_{0} P_{\mathrm{observed}}},$$

where $m_i$ and $m_0$ are numbers of independent trials. Uncertainty grows rapidly when success probabilities approach zero. Since estimates may share tasks/seeds/checkpoints (inducing positive covariance), the paper recommends paired or hierarchical bootstrap intervals over complete task units rather than the analytic approximation.



## Empirical Validation / Results

### Literature Review: Component-Level Audit

The paper reviews 9 representative benchmarks and assesses them against the six required protocol components:
>
**Table 4: Component-level delta between representative studies and the proposed protocol** (✓: implemented; ∼: partial/implicit; —: not part of reported protocol)

| Study | Matched local stages | Composed $P_{\text{expected}}$ | Natural rollout | State compatibility | Targeted intervention | Protocol sensitivity |
|---|---|---|---|---|---|---|
| SWE-EVO | — | — | ✓ | — | — | — |
| NL2Repo-Bench | — | — | ✓ | — | — | — |
| SWE-Chain | ∼ | — | ✓ | — | — | — |
| ChainSWE | ✓ | — | ✓ | ∼ | ∼ | — |
| SlopCodeBench | — | — | ✓ | — | — | — |
| SWE-Marathon | — | — | ✓ | — | — | — |
| SWE-Milestone | ✓ | — | ✓ | ∼ | — | — |
| Illusion | ✓ | ∼ | ∼ | — | ✓ | — |
| HORIZON | — | — | ✓ | — | ∼ | — |
| **This position (proposed)** | **✓** | **✓** | **✓** | **✓** | **✓** | **✓** |



### Key Empirical Observations from Reviewed Studies

- **SWE-EVO** [Le et al.,, 2025]: 48 release-sized tasks modify 20.9 files on average, checked by 874 tests on average; not designed to separate scale, local difficulty, and horizon.
- **NL2Repo-Bench** [Ding et al,,,, 2025]: Best reported mean test pass rate is 40.2% across 104 repository-generation tasks; no more than five repositories fully completed.
>
- **ChainSWE** [Jin et al,,,, 2026]: 58.9% per-bug accuracy with oracle-correct prior state versus 36.5% with agent-generated prior state—a direct oracle-versus-persistent-state comparison, but no composed chain-level counterfactual.
>
- **SlopCodeBench** [Orlanski et al,,,, 2026]: No evaluated agent completes any of 36 problems; erosion and verbosity increase in 77% and 75.5% of trajectories, respectively—no matched reset-state baseline.
>
- **SWE-Marathon** [Desai et al,,,,  ​​2026]: No configuration exceeds 30% pass@1 across 20 project-scale tasks and 1,300 rollouts; 0 of 71 compaction-path trials pass versus  ​​8.9% without—compaction correlates with failure but cause/marker remain confounded..
- **SWE-Milestone** [Deng et al,,,, 2026]: Independent milestone scores ~80%+; best continuous score is 38.03%; on scikit-learn, one configuration drops from 93.2% independently to 21.1% continuously.

- **Illusion** [Sinha et al,,,,  ​​2026]: At matched evaluation turn, increasing prior-history error rate reduces subsequent accuracy—controlled synthetic task, but no prevalence/effect-size data in open-ended agents.
>
- **HORIZON** [Wang et al,,,,  ​​2026]: >3,100 trajectories across four domains reveal distinct planning-, memory-, history-related failure patterns; trajectory-grounded judge reaches κ=0.84 against one human annotator—labels are post-hoc, potentially co-occurring, not causal mechanisms.





### Worked Example: Four Independent Repairs

Consider a coding agent tasked with four independent repairs, each succeeding with probability 80% when evaluated separately:

- If outcomes compose independently: $0.8^4 \approx 41\%$ end-to-end success—no additional long-horizon failure needed to explain a low rate..
- If actual success is only 10%: $\Gamma_{\mathrm{H}} = \log(0.41/0.10) \approx 1.41$ nats—natural rollout underperforms by a factor of ~4.
- Stage-wise terms of, say, (0.2, 0.5, 0.6,  ​​0.1) indicate the mismatch surfaces mainly at the third repair, targeting the first intervention there.



## Theoretical and Practical Implications

### Interpretive Framework

The expected and observed success rates answer distinct questions and must be interpreted jointly:

- **Both high**: Both protocols place the task within measured capability under stated budgets/information.
- **Both low**: The task remains a useful deployment stress test, but the comparison has little power to separate counterfactual-model error, compounding, and history-dependent effects.
>
- **Expected high, observed low**: Natural rollout has an unexplained deficit under the selected model—motivating targeted interventions, not immediate mechanism claims..
- **Observed exceeds expected**: Natural rollout may exploit feedback, redundancy, adaptation, or recovery omitted by the counterfactual; interventions needed to determine which explanation accounts for the advantage.



### Benchmark Design Protocol (8 Steps)

For coding and terminal environments:

1. **Pre-register**: Declare task, stages, acceptance conditions, dependency graph, rollout policy, tools, verifier, recovery rules.
2. **Define checkpoints**: Record environment state, visible history, revealed information, repair policy, rules for alternative valid implementations..
3. **Match budgets**: Report token, tool, time, verification, retry costs for both local and end-to-end runs; include equal per-stage budgets and aggregate-budget-matched comparison where possible..
4. **Audit local performance**: Report per-stage successes, trial counts, uncertainty, heterogeneity; pre-specify handling of zero/near-zero estimates..
5. **Choose the model**: Product baseline for fixed compatible path; conditional model or checkpoint simulator for branches, retries, recovery, shared-state dependence..
6. **Run and intervene**: Estimate $P_{\mathrm{expected}}$ before measuring $P_{\mathrm{observed}}$; then test targeted changes (history reset, state repair, stronger verification, rollback, explicit planning).
7. **Test sensitivity**: Recompute predictions/residuals under multiple reasonable decompositions, dependency graphs, admissible checkpoint protocols; conclusions robust only when sign is consistent across pre-declared set..
8. **Audit the benchmark**: Report task-inclusion criteria, exclusions, leakage checks, contamination risks for both agent and decomposition/grading procedures..



### Structural Annotations

The paper advocates two complementary annotations to separate task structure from local difficulty:

- **Work exposure** $N_{\mathrm{work}}$: How many required, separately verifiable stages the benchmark declares—more stages create more opportunities for local failure..
- **Dependency depth** $H_{\mathrm{dep}}$: How many stages lie on the deepest required chain—greater depth means later work must build on more earlier state or feedback..

Neither is an intrinsic property of a task; both depend on the chosen decomposition, acceptance conditions, and dependency graph. They can vary independently within a declared design. The paper also relates these to HORIZON's agent-independent intrinsic horizon $H^*$ (minimum effective actions required by an optimal policy), viewing them as complementary descriptions rather than substitutes..

**Table 2: Factorial coding-task design separating dependency depth from local difficulty**

| | Easier local stages | Harder local stages |
|---|---|---|
| **Shallow chain** | Independent repairs with diagnostic tests | Independent repairs with ambiguous failures or security constraints |
| **Deep chain** | An explicit migration with strong intermediate checks | A multi-module migration with sparse feedback and difficult recovery |



## Conclusion

### Main Takeaways

1. **A declining end-to-end curve licenses a deployment claim, not a mechanism claim**. Raw performance decline on longer tasks can arise purely from ordinary error compounding: more stages create more chances to fail, even when no distinctive long-horizon limitation exists..

2. **The horizon residual $\Gamma_{\mathrm{H}}$ is a disciplined diagnostic starting point**, not a causal answer. It quantifies the log-contrast between a pre-specified compositional prediction from matched short stages and observed natural rollout success, directing follow-up experiments on history, state, planning, verification, or recovery..

3. **No reviewed study implements the full proposed sequence**. Every component exists somewhere in the literature, but the complete combination—matched local measurement, declared composition, natural rollout, state-compatibility handling, targeted intervention, protocol sensitivity analysis—remains unimplemented under one pre-specified protocol..

4. **The framework turns "longer is harder" from a descriptive observation into an auditable research program**, with explicit falsification criteria: sign instability across admissible protocols, infeasible checkpoint construction, or no incremental diagnostic value beyond existing persistent-versus-reset comparisons..

### Future Directions

The paper closes with a five-point empirical agenda:

1. **Instantiate the comparison**: Apply the protocol to one existing persistent-state benchmark, reporting $P_{\mathrm{expected}}$, $P_{\mathrm{observed}}$,, and $\Gamma_{\mathrm{H}}$ with bootstrap intervals..
2. **Develop branching estimators**: Extend uncertainty treatment (Equation 4) to conditional transition models over branching state graphs, with explicit zero/near-zero probability rules..
3. **Run the intervention program**: Pair each recurring residual pattern with single-factor interventions (history reset/compression, state repair, verification, rollback, explicit planning) to turn residual patterns into testable causal hypotheses..
4. **Map protocol robustness**: Characterize when admissible protocols agree in sign and when they diverge, turning robustness criteria into empirically grounded admissibility rules..
5. **Transfer beyond coding**: Identify where replayable state and semantic acceptance conditions exist outside coding/terminal tasks, adapting the protocol where they do not..

> *"Until such residuals are reported, a declining curve should be read as evidence about deployment, not about mechanism."*

---

_Markdown view of https://picx.dev/p/EKzrU4, served by PicX — AI-generated visual whiteboard summaries of research papers._