Summary of: "StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling"

Ziheng Qin, Yaxin Lu, Zhangyang "Atlas" Wang, Kai Wang

---

Summary (Overview)

• Core proposition — "harness scaling": The authors introduce StateM, an agent-native runtime that improves long-horizon agent performance by modifying the execution system around an agent (states, transitions, hooks, checks, recovery rules) rather than changing model weights. The central research question: "How much apparent model failure is actually failure of the harness that maintains state, constrains execution, verifies progress, and recovers from errors?"
• Quality frontier: On Terminal-Bench 2.1, GPT‑5.5 xhigh with StateM reaches 92.1% accuracy (vs. 83.1% reference), a gain resembling a full model generation. A runbook frozen from GPT‑5.5 transfers unchanged to GPT‑5.6 Sol xhigh, achieving 95.28% raw accuracy (424/445 trials) and success on all 89 tasks at least once. The same frozen profile lifts GPT-5.6 Luna from 76.7% → 85.4%.
• Cost frontier: Using the same runtime/runbook structure, adapting concrete practices to DeepSeek-V4-Flash costs less than $38 in adaptation API usage. The complete DeepSeek final-score evidence costs ≈ $15.20, reaching 88.09% on the full suite (89.09% on the 88-task common core; a descriptive aggregate of 88.76% matches the reported GPT-5.6 Sol max score of 88.8%).
• Transfer hierarchy: Exact control profiles transfer frozen across nearby model generations, development principles and runbook structure transfer across providers (with inexpensive adaptation), and only the development methodology (locating consequential execution boundaries) transfers across task distributions.
• Three operational failure gaps are targeted by three control mechanisms: epistemic gaps → state-local context; procedural-memory gaps → versioned practices; procedural-compliance gaps → checked transitions.

---

Introduction and Theoretical Foundation

Background and Motivation

Long-horizon agents often fail in a revealing way: the underlying model can solve each local step, yet the complete run fails. Observed failure modes include losing track of mutable state, failing to reactivate lessons from earlier executions, skipping known procedures, and stopping prematurely — all before the requested deliverable is verifiably complete.

The dominant response is to improve the model (pretraining, post-training, test-time reasoning, more agents). The authors instead study an orthogonal question: how much apparent model failure is actually harness failure? Harness scaling is thus framed as a capability axis complementary to model scaling.

Two Operational Hypotheses

1. Control-signal dilution: A compact plan surrounded by an increasingly long trace of commands, observations, and repairs loses its control strength.
2. Mutable-state ambiguity: Completed goals, pending dependencies, failed attempts, and valid next actions must be reconstructed from append-only history rather than read from an authoritative current state.

Three Sources of Failure

The paper distinguishes three gaps (this is a key theoretical contribution):

• Epistemic gap: relevant knowledge or the appropriate method is unavailable at the decision point.
• Procedural-memory gap: a lesson from an earlier run is not retained, reactivated, or invoked when the risk recurs (across executions).
• Procedural-compliance gap: the correct procedure is active but remains incomplete or not fully followed by the agent (within a run).

StateM's design answers each gap: state-local context, versioned procedural practices, and checked transitions, respectively.

Design Space: Runtime Enforceability vs. Agent Autonomy

Existing systems trade off:

• State-machine / graph runtimes (e.g., LangGraph, StateFlow) provide persistent state, conditional routing, and recovery, but are typically organized around a developer-authored controller and externalize reasoning into fixed nodes/edges.
• General-purpose CLI agents (Codex, Claude Code) preserve broad reasoning/action space but their plans, instruction files, memories, and hooks do not form a unified, transition-aware control surface.

StateM targets the upper-right combination: broad agent autonomy with explicit, auditable, enforceable state transitions, in a control artifact jointly editable by agent and user.

---

Methodology

StateM Runtime and Runbook

StateM is a lightweight runtime configured by a human-readable YAML runbook containing states, valid transitions, state-local instructions, hooks, checks, and recovery rules. The separating insight: the runtime provides generic mechanisms (persistence, transition validation, hooks, history, recovery), while the control profile (runbook) specifies phase content, and enactment knowledge.

States as Context-and-Contract Boundaries

A state represents a meaningful stage of work (e.g., planning, implementation, task-contract checking, self-review, repair, handoff) — not a single model call. Two functions:

• Context boundary: entering a state refreshes the active instructions and durable task state via an in_hook (populates instructions, runs setup code, loads a compact progress record).
• Contract boundary: out_hook persists progress; the before_transfer block evaluates exit conditions before committing a transition.

Check types (by firmware strength):

• command/predicate checks — host-evaluated, independently reproducible; and manual checks — require explicit user/operator decision.
• checklist/message checks — self-attestation by the agent (structured, but not independent verification).
• llm_review checks — separate semantic judgment, but not deterministic.
• A structural field or agent declaration is not treated as proof merely because it appears in a structured receipt.

Formal Runbook Definition

A runbook defines a directed state machine:

$$\mathcal{B} = \left(\mathcal{S}, s_{0}, \mathcal{S}_{T}, \mathcal{E}, \Phi\right),$$

where $\mathcal{S}$ is the set of phase-level states, $s_0$ is the initial state, $\mathcal{S}_T \subseteq \mathcal{S}$ is the set of terminal states, $\mathcal{E} \subseteq \mathcal{S} \times \mathcal{S}$ is the set of permitted transitions, and the state specification $\Phi(s)$ holds the phase-local prompt, entry/exit hooks, transfer checks, and state-local artifacts.

All phase changes use the core operation goto. The transition protocol:

1. Verify the requested edge from current state to target exists.
2. Evaluate the current state's before_transfer checks.
3. Run the current state's out_hook end-of-hook operations.
4. Evaluate edge guards and edge-specific transfer hooks.
5. Commit the target state and append transition history only if all pre-commit steps succeed; otherwise remain in the source state and record the failure.
6. Create the target-state entry and execute its in_hook.

This is described as checked, logged, and recoverable — not fully transactional (external side effects cannot generally be rolled back, so recovery means restoring the StateM execution record and executing the configured repair procedure).

Persistent State, Recovery, and Enforced Completion

• Per-run storage: run identifier, current state, transition history, hook/check outcomes, timestamps, evidence-file references — separated from the reusable runbook.
• After a restart or context refresh, the agent queries current state, failed conditions, and evidence; it need not reconstruct the workflow from a long terminal transcript.
• Stopping conditions: a stop hook inspects StateM status; if the run is in a terminal state or explicitly blocked on an external dependency, the stop proceeds — otherwise the agent is returned the current phase and unmet obligations and requested to continue. This reduces premature termination.

Failure-Driven Harness Optimization

After runs, failures are classified (missing context, invalid transitions, weak checks, premature handoffs, ineffective recovery) and converted into versioned proposals: changes to state boundaries, prompts, hooks, checks, recovery rules, and practice activation conditions. Proposals are reviewed, regression-tested, and incorporated into the versioned runbook. The harness is both the runtime, an audit surface, and the operator's own optimization target — the search space for improvement.

---

Empirical Validation / Results

Evaluation Protocol

• Terminal-Bench 2.1: 89 tasks × 5 trials = 445 trials per full run; primary metric is trial-level success; also reported is 5-trial task coverage. Models: Codex agents with GPT-5.5 xhigh, GPT-5.6 Sol xhigh, GPT-5.6 Luna, and DeepSeek-V4-Flash.
• Profile boundaries adhered: development never used task identifiers, hashes, hidden tests, verifier implementations, or answer artifacts as activation data.
• BusinessBench: 477 eligible instances across 7 families (one family, attendance-payroll, abstained); 405 treated instances across 6 families = 810 treatment/control executions; family-level profiles frozen before the first held-out evaluation.

Same Model, Better Harness: A Model-Generation-Sized Gain

The Sol xhigh StateM public-submission record: 445 trials, 439 no-error completions, 6 AgentInitializerErrors, 424 successes. For contrast, the model-generation shift (GPT‑5.5 → GPT‑5.6 Sol) under the reference harness is only +1.8 points (83.1 → 84.9). Harness scaling changed completed-task performance more than the observed generation-to-generation model shift.

Frozen Transfer Within the GPT Family (Zero Target-Model Runbook Changes)

• GPT-5.5-developed profile frozen and applied to GPT-5.6 Sol xhigh → 95.28%; applied to GPT-5.6 Luna → 76.7% → 85.4%, which numerically exceeds the 84.9% Sol xhigh reference despite being the lower-tier model.

Cross-Provider Adaptation: The Cost Frontier

• Direct frozen transfer of the GPT profile to DeepSeek-V4-Flash fails: 82.7% → 82.0% baseline.
• With provider-specific adaptation (using the same runtime, runbook structure, golden rules, failure-analysis loop):

$$\frac{392}{445} = 88.09\%$$

on the full 89-task benchmark under standard timeouts.

$$\frac{392}{440} = 89.09\%$$

on the disclosed 88-task common core (excluding the latency-sensitive gpt2-codegolf).

Extending the timeout only for that task: 3/5 successes, giving a descriptive full-suite aggregate:

$$\frac{392 + 3}{445} = \frac{395}{445} = 88.76\% \approx 88.8\%,$$

which matches the separately reported GPT-5.6 Sol max score at one-decimal precision.

Economic results:

The DeepSeek evidence uses 1/37.8 of the public GPT submission's recorded cost; the entire adaptation-and-evaluation campaign is ≈1/11 of that cost.

Task-Level Generalization: BusinessBench (Frozen One-Shot)

Table 1 — Frozen one-shot BusinessBench results with Codex + GPT-5.6 Luna (aggregates over six treated families; attendance-payroll abstained):

†The 10.04-point subgroup Δ is computed from unrounded source values (displayed values differ by 10.03 after rounding).

Key interpretation: gains are concentrated in families where the control profile matches a specific execution boundary (Budget Approval: decimal-exact calculation, policy reconciliation, mandatory-notes closure; Machine Operating: task-derived query planning, interval coverage, durable publication). Negative transfer (−2.78 RefactorBench, −3.70 WooCommerce) occurs when control attaches to the wrong boundary: later diagnostic re-runs with corrected profiles remove or reverse these regressions (e.g., RefactorBench overall 76.39 → 79.17; WooCommerce overall 86.42 → 90.12). Methodology generalizes; concrete rules transfer only when tasks share the relevant execution structure.

Representative Task-Level Evidence (GPT-5.5)

Table 3 — Representative Terminal-Bench 2.1 task-level improvements (success out of 5 trials).

The canonical case: configure-git-webserver — the baseline agent can configure Git, SSH, hooks, and HTTP servers yet scores 0/5 because it does not preserve and validate the required end-to-end live state. With StateM, the final handoff is gated on fresh consumer-facing evidence (clone → commit → push → curl before leaving verification); the run stays in a repairable state if verification mutates the environment. StateM adds no new capability. It returns 5/5 by composing checks that the model already possessed around the consequential handoff.

Operational Endurance

One profile-development run sustained a 22-hour autonomous hyper-agent run (context refresh/compaction, stop-hook continuation) while the durable StateM record preserved phase, transition history, unresolved obligations, and a recovery anchor.

---

Theoretical and Practical Implications

1. Harness scaling is a real, measurable axis: An execution harness can move completed-task performance more than a model generation does (9–10.4 points vs. 1.8 points). This re-frames measurement: model identity is not a sufficient description of an agent system. Evaluation claims must identify the runtime and profile.

2. Transfer follows distance (hierarchy of abstraction):

   • Nearby models (GPT-5.5 → GPT-5.6): exact frozen runbook transfers with zero changes — the "transferable object" is the ordered, frozen YAML control profile.
   • Across providers (GPT → DeepSeek): exact profiles transfer; but the runtime, runbook structure, routing strategy, applicable controls, failure-analysis loop, and golden rules remain reusable at adaptation costs that are drastically cheaper than a new model run — ~$38 adaptation vs. $574.68 on the GPT reference.
   • Across task distributions (BusinessBench): the methodology's boundary-locating practice transfers; concrete runbook content does not (it even causes negative transfer if matched to the wrong behavioral boundary).
   • Result: "the transferable object becomes more abstract as distance grows."

3. Theoretical model: three gaps — three control points. The paper operationalizes three epistemic gaps (epistemic, procedural-memory, procedural-compliance) into distinct runtime constructs (state-local context, versioned practices, checked transitions). This supplies a design-language mapping (failure diagnosis → concrete control intervention).

4. Control is not more rules. Both the negative results (RefactorBench, WooCommerce) and positive results (Budget Approval, Machine Operating) show that optimal harness size is sparse, mechanism-matched control attached at consequential phase boundaries. "More remembered procedure is not better memory"; experience must be filtered before it becomes durable memory.

5. Economic realism: exact transfer failing across providers means that blanket "one control for all models" is decouplable; correct the question to routing where the pattern of execution failure differs even when the task set is identical. Better Harnesses, Smaller Models-style adaptation can make medium-cost models competitive with frontier models in syntax-completed tasks while reducing API expenditure by >10–30×.

6. Auditability and public disclosure. The paper repeatedly emphasizes verification strength distinctions (command/predicate vs. self-attestation vs. LLM review), the need to disclose whether results are raw vs. post-adjudication, the separation of development vs. frozen evaluation, and what the members "adaptation cost", "realized charges", and "submission-reported model cost" mean. This is itself a methodological contribution for reproducible agent benchmarking.

---

Conclusion

StateM shows that modeling is not the only improvement axis; the surrounding execution system can convert a model's existing capability into reliably completed work, with quantitative gains rivaling a model generation. Key takeaways:

• StateM runtime + frozen profile: GPT-5.6 Sol achieves 95.28% raw (424/445), solving more tasks at least once, on Terminal-Bench 2.1 — with a cross-family transfer gain (Luna 76.7→85.4%) and a cost-accuracy frontier where DeepSeek-V4-Flash matches GPT-5.6 Sol max (88.8%) on the suite for ≈ $15 of evidence cost.
• The transfer hierarchy — exact profiles transfer within a model family; structure and development principles transfer across providers; only the method of locating control boundaries transfers across task families.
• A learning rule for harnesses: the responsible adaptation loop turns selected postmortem findings into versioned, executable preconditions — but only generalizable and mechanism-matched lessons should be promoted.
• Limitations acknowledged: Model scaling still matters (groups are not universally strong under one runbook); concrete practices require adaptation across providers; semantically strict checks (semantic self-review, agent-authored property recipients) remain fallible; StateM does not undo longitudinal memory-driven aging (compression/retrieval aging remain out of scope); and the system targets single-agent within-run reliability.

Future Directions

The authors note a natural extension: role-isolated multi-agent runtimes — splitting verification, delegation, and decision-making into role/write-permission boundaries to reduce self-reinforcing errors within one model context. A second explicit future direction is the integration of lifespan diagnostics (AgingBench — revision/compression aging) with StateM-style within-run procedural runbooks.

---

Source: Qin et al., "StateM: Reaching 95.3% Raw Accuracy, or a $15 Frontier Run, on Terminal-Bench 21 v1.T. via Harness Scaling," arXiv preprint.

Metrics marked \( \text{raw}\) are pre-adjudication public-submission numbers; the authors emphasize that 95.3% is the raw-score result and that finished leaderboard values may exist pending adjudication.