# Long-Horizon-Terminal-Bench: Testing the Limits of Agents on Long-Horizon Terminal Tasks with Dense Reward-Based Grading

> Long-Horizon-Terminal-Bench reveals even the best model achieves only 15.2% pass rate, proving long-horizon completion is the core bottleneck for AI agents.

- **Source:** [arXiv](https://arxiv.org/abs/2607.08964)
- **Published:** 2026-07-14
- **Permalink:** https://picx.dev/p/vwqt8Z
- **Whiteboard:** https://picx.dev/p/vwqt8Z/image

## Summary

## Summary (Overview)

- **New Benchmark**: Introduces **Long-Horizon-Terminal-Bench (LHTB)** , a benchmark of 46 containerized terminal tasks spanning nine domains (e.g., experiment reproduction, software engineering, multimodal analysis, interactive games, scientific computing). Each task requires hundreds of episodes, tens of minutes to hours, and millions of tokens.
- **Dense Reward Grading**: Unlike prior binary pass/fail benchmarks, LHTB decomposes each task into fine-grained subtasks with deterministic, environment-grounded checks, providing partial credit and dense intermediate reward signals. The overall reward is:  
  $$R = \frac{\sum_{k=1}^K w_k r_k}{\sum_{k=1}^K w_k}$$  
  where $r_k \in [0,1]$ is the normalized score for subtask $k$, and $w_k$ are non-negative weights.
- **State-of-the-Art Results**: Even the strongest model (GPT-5.5) achieves only **15.2% pass@1** at reward threshold $\tau = 0.95$ and **10.9%** at $\tau = 1.0$. The mean pass rate across 15 frontier models is just **4.3%** and **1.7%** respectively, revealing substantial headroom.
- **Failure Analysis**: Dense rewards expose distinct failure patterns – timeouts dominate (79% of unresolved runs), but early exits reveal weak self-verification (false finishes). The main bottleneck is **long-horizon completion** rather than local reasoning.
- **Cost and Complexity**: Average per-task consumption: 9.9M tokens, 231 episodes, 85.1 minutes, and ~$10.2 API cost. The benchmark is far more demanding than prior terminal benchmarks (e.g., Terminal-Bench 2, SWE-Bench).

## Introduction and Theoretical Foundation

The paper addresses two key limitations of existing agent benchmarks:

1. **Short Horizons**: Current benchmarks (e.g., SWE-Bench, Terminal-Bench) focus on tasks that finish within a few minutes, understating the difficulty of real-world workflows that require sustained progress over tens of minutes to hours, multi-stage planning, iterative debugging, and error recovery.
2. **Outcome-Only Grading**: Binary pass/fail evaluation yields sparse reward signals. An agent that completes most intermediate steps but fails at the final step receives the same score as one that fails from the beginning. This hides meaningful differences in partial progress.

The theoretical foundation draws on **process reward models** and **rubric-based evaluation**, applying dense reward signals at evaluation time rather than only for training. Benchmark tasks are designed to stress **long-horizon planning, long-context management, and iterative debugging** rather than one-shot problem solving.

## Methodology

### Task Formulation
Each LHTB task follows the Harbor format with:
- Natural-language instruction describing the long-horizon goal.
- Docker image with all assets, code, data, tools, and helper scripts.
- Task configuration file.
- Oracle implementation or simulator for grading.

Tasks are interactive: agents issue shell commands, edit files, run scripts, and inspect intermediate outputs over hundreds of steps until success or timeout (90-minute budget).

### Subtask-Based Grading
Each task is decomposed into $K$ subtasks $\{s_1, \ldots, s_K\}$ with normalized scores $r_k \in [0,1]$. Three types of subtasks:
- **Binary subtasks**: Pass/fail Boolean conditions on environment state.
- **Continuous or thresholded subtasks**: Score proportional to closeness to target (e.g., linear decay from 1.0 to 0 as error grows).
- **Episode-aggregating subtasks**: Fraction of episodes/levels where internal success flag is triggered.

The overall reward $R$ is the weighted average of subtask scores (Equation 1). A task is considered resolved if $R \geq \tau$ (with $\tau = 0.95$ as relaxed threshold). Mean reward across tasks captures partial progress.

### Dataset Construction
- **Seed workflows**: Real professional problems (e.g., materials phase-diagram audit, robotics SLAM repair, climate NetCDF detection, audio-visual alignment, satellite flood mapping).
- **Construction**: For each problem, a deliberately broken terminal-only project is built, including public asset-generation scripts, weak baseline, official gold solution, multi-step `solve.sh`, and hidden verifier.
- **Difficulty calibration**: Repeated runs of DeepSeek-V4-Pro under 1.5-hour budgets; 120 candidate tasks generated, quality-filtered to 46.
- **Anti-overfitting**: Public checks carry low weight; most reward comes from hidden stress cases (nested manifests, gzip+base64 wrappers, missing values, injected noise, rotated frames, etc.). Gold solution must achieve verifier score 1.0 on hidden suite.

### Evaluation Framework
- **Harness**: Harbor framework with Terminus-2 agent (except GPT-5.3 uses Codex).
- **Models**: 15 frontier models evaluated: GPT-5.5, GPT-5.4, GPT-5.3 Codex, DeepSeek V4 Pro, Gemini 3.1 Pro, GLM 5.1/5.2, Kimi K2.6/K2.7 Code, MiniMax M3, Qwen3.7 Max/Qwen3.6 Plus, Doubao Seed 2.1 Pro, Hy3, Grok 4.20.
- **Metrics**: Pass@1 at reward thresholds 0.9, 0.95, 1.0; mean normalized reward; episodes, time, tokens, cost.

## Empirical Validation / Results

### Main Results (Figure 3, Table 1)
- **GPT-5.5** is strongest: 15.2% pass@1 at $R \geq 0.95$ (7/46 tasks), 10.9% at $R \geq 1.0$.
- **MiniMax M3**, **Kimi K2.7 Code**, **DeepSeek V4 Pro**: each 6.5% (3/46) at $R \geq 0.95$.
- **Mid tier** (4.3%): Qwen3.7 Max, Doubao Seed 2.1 Pro, Gemini 3.1 Pro, GLM 5.1, GPT-5.3 Codex.
- **Lowest**: Kimi K2.6 and Grok 4.20 solve zero tasks at $R \geq 0.95$; Grok 4.20 has mean reward $R = 0.08$.
- **Mean rewards** follow similar ordering, but many models achieve partial progress even on unsolved tasks.

### Dense Reward Necessity (Figure 4)
- Under strict binary ($R \geq 1.0$), 10 of 15 models pass zero tasks.
- Only **4.3%** of runs (30/690) pass at $R \geq 0.95$; **32.9%** make no progress ($R < 0.05$); **62.8%** achieve partial reward.
- **Near-misses** ($0.75 \leq R < 0.95$) occur twice as often as passes (73 vs. 30), indicating many runs come close but fail final checks.
- Correlations: Spearman $\rho = 0.56$ between pass rate and mean reward, showing they measure different aspects.

### Cost Analysis (Table 1, Figure 5)
| Model | Tokens/task (M) | Episodes/task | Time/task (min) | Avg cost/task ($) |
|---|---|---|---|---|
| GPT-5.5 | 4.16 | 208 | 72.9 | 21.46 |
| MiniMax M3 | 20.20 | 314 | 90.0 | 6.13 |
| Kimi K2.7 Code | 8.54 | 183 | 85.4 | 8.31 |
| DeepSeek V4 Pro | 14.45 | 321 | 83.6 | 6.32 |
| Qwen3.7 Max | 6.13 | 218 | 83.5 | 7.78 |
| Doubao Seed 2.1 Pro | 5.80 | 183 | 91.7 | 5.16 |
| Gemini 3.1 Pro | 3.55 | 148 | 85.0 | 7.61 |
| GLM 5.1 | 5.84 | 120 | 92.6 | 5.13 |
| GPT-5.3 Codex | 4.57 | 299 | 80.7 | 8.20 |
| GLM 5.2 | 8.43 | 195 | 89.3 | 11.93 |
| Qwen3.6 Plus | 8.67 | 194 | 88.6 | 4.47 |
| Hy3 | 17.21 | 258 | 91.3 | 2.47 |
| GPT-5.4 | 10.90 | 302 | 79.3 | 27.57 |
| Kimi K2.6 | 10.27 | 188 | 92.5 | 9.94 |
| Grok 4.20 | 16.23 | 288 | 69.5 | 20.63 |
| **Average** | **9.66** | **228** | **85.1** | **10.21** |

- GPT-5.5 highest pass rate but expensive ($21.46/task); GPT-5.4 most expensive ($27.57) with lower pass rate.
- Hy3 anchors low-cost Pareto frontier ($2.47/task, 2.2% pass), while MiniMax M3, Doubao Seed 2.1 Pro, and DeepSeek V4 Pro offer good cost-effectiveness.

### Failure Patterns (Figure 6)
- **Timeouts**: 79% of unresolved runs – agents still working when 90-minute budget expires. Mean reward for timed-out runs ranges 0.10–0.35.
- **Early exits**: 19% – agents stop on their own before completing. Some false finishes (e.g., Kimi K2.7 Code stops at $R=0.92$ on one task; seven models stop on a task between $R=0.80$ and $0.87$ with ~20 minutes remaining).
- **Harness errors**: 3%.
- Main bottleneck is **long-horizon completion** rather than local correctness. Agents can string together locally correct actions but cannot reliably turn progress into finished artifacts before horizon expires.

### Stopping Judgment (Section 3.5)
- 14 runs identified as false finishes ($R \geq 0.75$ early exit).
- Dense rewards reveal variation: Kimi K2.7 Code's early exits average $R=0.51$; MiniMax M3 average $R=0.42$; Kimi K2.6 exits at average $R=0.11$.
- This exposes weak self-verification: agents overestimate completion and under-invest in final verification.

## Theoretical and Practical Implications

- **Evaluation Methodology**: LHTB demonstrates that dense, subtask-level grading is essential for ranking models on long-horizon tasks. Binary pass/fail collapses a wide range of partial progress into a single failure bucket, masking capability differences. Mean reward and pass rate provide complementary signals.
- **Agent Capability Gap**: The low pass rates (even for top models) and high token/episode consumption indicate that **long-horizon execution remains a central bottleneck** for current AI agents. Improvements in planning, memory, progress tracking, and self-verification may yield larger gains than improvements in single-step reasoning alone.
- **Cost-Reward Trade-offs**: Pareto frontier analysis shows that higher inference cost does not guarantee better performance. Some cheaper models (e.g., Hy3 at $2.47/task) achieve comparable or better cost-effectiveness than expensive ones.
- **Practical Guidance for Agent Design**: The failure analysis suggests that agents need:
  - Better time budgeting to avoid timeouts.
  - More accurate self-verification to avoid false finishes.
  - Strategies to reduce redundant exploration and maintain state effectively.
- **Benchmark for Future Research**: LHTB provides a challenging, reproducible testbed for developing agents that can plan, verify, and execute reliably over long horizons.

## Conclusion

- **Key Contributions**: Long-Horizon-Terminal-Bench introduces 46 long-horizon terminal tasks with dense reward-based grading, covering nine domains. It is substantially more demanding than prior benchmarks (average 9.9M tokens, 231 episodes, 85.1 minutes per run).
- **Main Findings**: Current frontier models struggle significantly – best model achieves only 15.2% pass@1 at $\tau=0.95$, mean pass rate 4.3%. Timeouts dominate failures, but false finishes due to weak self-verification are also prevalent.
- **Bottleneck**: The main limitation is not local reasoning but **reliable long-horizon completion** – agents cannot sustain progress, verify completion, and finish within budget.
- **Future Directions**: Need for better planning, stronger memory and progress tracking, and more calibrated stopping decisions. The benchmark and evaluation harness are released to support future work on robust long-horizon terminal agents.

---

_Markdown view of https://picx.dev/p/vwqt8Z, served by PicX — AI-generated visual whiteboard summaries of research papers._