Summary of "Rethinking Verification in Agent Harnesses"

Summary (Overview)

• This paper systematically studies verification as an axis of test-time scaling in agent harnesses, organizing 17 verification designs across four layers of the agent lifecycle: initial instructions (L0), execution-time reminders (L1), completion gates (L2), and independent review (L3).
• L3 independent review by a separate critic delivers the most favorable accuracy–cost tradeoff, improving average accuracy by 5.9–9.5 percentage points on Terminal-Bench (TB) and 3.9–6.6 on SWE-bench Verified (SWE) at only 1.13–1.25× and 1.00–1.04× baseline cost, respectively.
• L0 initial instructions and L1 reminders yield modest, model-dependent gains on coding benchmarks without substantially increasing output token use (0.90–1.00× and 0.75–0.93× baseline tokens).
• L2 completion gates can yield larger gains but at substantially higher cost (1.60–1.76× baseline on TB), with rejection-based designs showing the strongest accuracy improvements.
• On the newly introduced VisualBench (18 visual/interactive tasks with blinded expert evaluation), all four independent-review designs achieve net preferences of +13.2 to +23.8 points over baseline, and scaling review through additional rounds or critics further improves average TB accuracy by up to 3.5 percentage points.

---

Introduction and Theoretical Foundation

Background and Motivation

Agent harnesses enable LLMs to act through tools, observe results, and iteratively work toward task goals. The harness itself specifies instructions, controls intervention timing, and coordinates verification—yet there is limited empirical guidance on how to organize verification effort effectively. Prior designs have been shaped by "engineering heuristics and iterative experience" (e.g., Claude Code hooks, Codex self-review), but systematic evidence remains limited on how to design and scale verification, when design improvements actually help, and whether gains justify computational costs.

Theoretical Foundation: The Verification Design Space

The paper frames verification around three fundamental questions:

1. When to intervene?
2. Who should check the work?
3. How should findings lead to revision?

These map to a four-layer framework following the agent lifecycle:

Related Work

The paper builds on three research strands:

• Learned verifiers for selecting solutions (Cobbe et al., 2021; Lightman et al., 2023) and trained critics for highlighting bugs (McAleese et al., 2024)
• Self-correction through critiques and reflections (Madaan et al., 2023; Shinn et al., 2023)
• Tool-assisted verification grounding error detection in execution evidence (Chen et al., 2023; Gou et al., 2023; Huang et al., 2023a)

The key gap identified: prior work establishes "useful assessment mechanisms and agent workflows, but offers limited comparative guidance on how a harness should organize checking and revision."

---

Methodology

Models Evaluated

Seven frontier coding models tested through Alibaba Cloud Model Studio:

• Qwen3.7-Max, Qwen3.8-Max (Qwen Team)
• GLM-5.2 (Z.ai)
• DeepSeek V4-Pro, DeepSeek V4-Flash
• Kimi-K2.7-Code (Moonshot AI)
• MiniMax-M2.5

Qwen3.7-Plus serves as the independent critic for every coding model.

Benchmarks

1. Terminal-Bench 2.1 (TB): 31-task study set selected across three empirical difficulty bands (baseline success rates)
2. SWE-bench Verified (SWE): 30 tasks selected evenly across human time-to-fix tiers
3. VisualBench (new): 18 visual and interactive tasks (games, shaders, 3D scenes, applications, generative art) evaluated through blinded expert comparisons of functionality and visual quality

The 17 Verification Designs

Environment

• All experiments run in E2B cloud sandboxes, orchestrated with Harbor
• Each trial starts in a fresh sandbox with task files and dependencies
• Critic has read-only access (cannot modify the deliverable)
• Three trials per model–task pair for coding benchmarks; two for VisualBench

Key Metrics

• Accuracy: percentage of tasks solved (average across trials)
• Cost: includes coding agent, critic, and revisions, priced separately by model (see Table 4 for API rates)

---

Empirical Validation / Results

L0: Initial Instructions

• System-role instructions consistently outperform user-role across both benchmarks
• Model-dependent effects: MiniMax-M2.5 gains +12.6/+8.0 points on SWE; Qwen3.8-Max declines under both
• Cost changes are modest: 1.08/1.03× baseline on TB

L1: Execution-time Reminders

• Average output tokens decrease by 14–25% on TB and 7–10% on SWE
• Baseline agents only check after 6.2–13.6% of editing turns—justifying interventions
• On VisualBench: L1 averages −3.2 points net preference

L2: Completion Gates

• Rejection-based designs yield largest gains but at substantial cost
• Earlier intervention consistently improves both rechecking and rejection designs
• Agent behavior after rejection: 12–19 additional steps (median), file-changing actions in 86–96% of runs

L3: Independent Review — The Key Finding

Key advantages over L2:

• Larger gains on TB (+9.5 vs. +6.0 for best L2)
• Comparable gains on SWE (+6.6 vs. +6.2)
• Lower cost on both benchmarks

VisualBench Results

Changes-based review outperforms trajectory-based review both with and without required visual inspection.

Scaling Independent Review

• Additional rounds/critics improve average TB accuracy by up to 3.5 percentage points
• Four critics approach four-round accuracy at 19% lower total cost
• Four critics: agent-side cost $0.48 vs.$0.63 for four rounds—width consolidates feedback before revision

---

Theoretical and Practical Implications

Theoretical Contributions

1. Verification as a test-time scaling axis: The paper establishes that verification effort—not just sampling or model scale—can systematically improve agent performance, with diminishing returns and clear cost tradeoffs.

2. The value of independent perspective: The consistent superiority of L3 over L0–L2 designs demonstrates that "an independent checking perspective can address gaps in the agent's self-verification"—a critic in a fresh context can find cases the agent overlooks.

3. Trajectory context is not uniformly beneficial: Despite costing 29–43% more in critic tokens, full trajectories do not consistently improve outcomes on TB/SWE, though they help more on VisualBench.

Practical Implications

1. Design guidance for harness engineers:

   • Use concise verification instructions in the system prompt (not user role)
   • Use after-edit reminders to focus checking without increasing output tokens
   • Avoid completion-time self-checking—gains are limited relative to cost
   • Deploy a separate, low-cost critic with task requirements and changes as default context

2. Cost-efficient accuracy improvement: Independent review achieves substantial accuracy gains at 1.00–1.25× baseline cost—a highly favorable tradeoff for production systems.

3. Evidence from case studies:

   • Critics find bugs the agent's own tests miss (e.g., date-before-IP regex case, COBOL unsigned-field arithmetic underflow)
   • Completion gates can create invalid final states—re-testing can change the deliverable being judged (e.g., committing test content to served branches)

---

Conclusion

This paper provides comprehensive empirical guidance for verification design in agent harnesses:

• L0 (initial instructions) and L1 (execution-time reminders) yield modest, model-dependent coding gains without substantially increasing output tokens
• L2 (completion gates) can improve accuracy but may substantially increase cost
• L3 (independent review) offers the most favorable accuracy–cost tradeoff across all three benchmarks, with consistent improvements across models and tasks
• Scaling verification through additional rounds (depth) or independent critics (width) further improves accuracy, with width being more cost-efficient

Future directions implied by the work include:

• Studying verification designs for non-coding agent tasks
• Exploring critic selection and training for specific domains
• Investigating optimal review depth/width configurations per task type
• Integrating verification design with other test-time scaling axes (e.g., sampling, context management)

The authors conclude that independent review is a "practical axis of test-time scaling," enabling users to configure verification effort for different tasks to improve agent performance at manageable cost.