# Rethinking Verification in Agent Harnesses

> Independent review by a separate critic at the final stage delivers the best accuracy-to-cost tradeoff, improving agent performance by up to 9.5 points at only 1.25x baseline cost.

- **Source:** [arXiv](https://arxiv.org/abs/2610.34577)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/ptXwZi
- **Whiteboard:** https://picx.dev/p/ptXwZi/image

## Summary

# Summary of "Rethinking Verification in Agent Harnesses"

## Summary (Overview)

- This paper systematically studies **verification as an axis of test-time scaling** in agent harnesses, organizing 17 verification designs across four layers of the agent lifecycle: initial instructions (L0), execution-time reminders (L1), completion gates (L2), and independent review (L3).
- **L3 independent review by a separate critic** delivers the most favorable accuracy–cost tradeoff, improving average accuracy by **5.9–9.5 percentage points on Terminal-Bench (TB)** and **3.9–6.6 on SWE-bench Verified (SWE)** at only **1.13–1.25× and 1.00–1.04× baseline cost**, respectively.
- L0 initial instructions and L1 reminders yield **modest, model-dependent gains** on coding benchmarks without substantially increasing output token use (0.90–1.00× and 0.75–0.93× baseline tokens).
- L2 completion gates **can yield larger gains but at substantially higher cost** (1.60–1.76× baseline on TB), with rejection-based designs showing the strongest accuracy improvements.
- On the newly introduced **VisualBench** (18 visual/interactive tasks with blinded expert evaluation), all four independent-review designs achieve **net preferences of +13.2 to +23.8 points** over baseline, and scaling review through additional rounds or critics further improves average TB accuracy by up to 3.5 percentage points.

---

## Introduction and Theoretical Foundation

### Background and Motivation

Agent harnesses enable LLMs to act through tools, observe results, and iteratively work toward task goals. The harness itself specifies instructions, controls intervention timing, and coordinates verification—yet there is **limited empirical guidance** on how to organize verification effort effectively. Prior designs have been shaped by "engineering heuristics and iterative experience" (e.g., Claude Code hooks, Codex self-review), but **systematic evidence remains limited** on how to design and scale verification, when design improvements actually help, and whether gains justify computational costs.

### Theoretical Foundation: The Verification Design Space

The paper frames verification around three fundamental questions:
1. **When to intervene?**
2. **Who should check the work?**
3. **How should findings lead to revision?**

These map to a **four-layer framework** following the agent lifecycle:

| Layer | When | Who Checks | How Execution Continues |
|-------|------|------------|------------------------|
| **L0** | At initialization | Agent | Use initial verification instructions to guide checking and revision |
| **L1** | During execution | Agent | Check recent changes and revise ongoing work after a reminder |
| **L2** | At completion | Agent | Resume work for self-checking and revision after a check request or rejected completion |
| **L3** | Before acceptance | Separate critic | Return critic findings to guide agent revision |

### Related Work

The paper builds on three research strands:
- **Learned verifiers** for selecting solutions (Cobbe et al., 2021; Lightman et al., 2023) and trained critics for highlighting bugs (McAleese et al., 2024)
- **Self-correction** through critiques and reflections (Madaan et al., 2023; Shinn et al., 2023)
- **Tool-assisted verification** grounding error detection in execution evidence (Chen et al., 2023; Gou et al., 2023; Huang et al., 2023a)

The key gap identified: prior work establishes "useful assessment mechanisms and agent workflows, but offers limited comparative guidance on how a harness should organize checking and revision."

---

## Methodology

### Models Evaluated

Seven frontier coding models tested through Alibaba Cloud Model Studio:
- Qwen3.7-Max, Qwen3.8-Max (Qwen Team)
- GLM-5.2 (Z.ai)
- DeepSeek V4-Pro, DeepSeek V4-Flash
- Kimi-K2.7-Code (Moonshot AI)
- MiniMax-M2.5

**Qwen3.7-Plus** serves as the independent critic for every coding model.

### Benchmarks

1. **Terminal-Bench 2.1 (TB)**: 31-task study set selected across three empirical difficulty bands (baseline success rates)
2. **SWE-bench Verified (SWE)**: 30 tasks selected evenly across human time-to-fix tiers
3. **VisualBench (new)**: 18 visual and interactive tasks (games, shaders, 3D scenes, applications, generative art) evaluated through **blinded expert comparisons** of functionality and visual quality

### The 17 Verification Designs

| ID | Layer | Method | Variant |
|----|-------|--------|---------|
| 1 | L0 | Verification instruction | System |
| 2 | L0 | Verification instruction | User |
| 3 | L1 | Periodic reminder | System |
| 4 | L1 | After-edit reminder | System |
| 5 | L1 | After-edit reminder | User |
| 6 | L2 | Inspect changes | Automated |
| 7 | L2 | Run and check | Automated |
| 8 | L2 | Run and check | System |
| 9 | L2 | Reject completion | Automated |
| 10 | L2 | Reject completion | User |
| 11 | L2 | Recheck before finishing | User |
| 12 | L2 | Recheck before finishing | User / early |
| 13 | L2 | Reject completion | User / early |
| 14 | L3 | Claim-guided review | Changes |
| 15 | L3 | Task-first review | Changes |
| 16 | L3 | Claim-guided review | Trajectory |
| 17 | L3 | Task-first review | Trajectory |

### Environment

- All experiments run in E2B cloud sandboxes, orchestrated with Harbor
- Each trial starts in a fresh sandbox with task files and dependencies
- Critic has **read-only access** (cannot modify the deliverable)
- Three trials per model–task pair for coding benchmarks; two for VisualBench

### Key Metrics

- **Accuracy**: percentage of tasks solved (average across trials)
- **Cost**: includes coding agent, critic, and revisions, priced separately by model (see Table 4 for API rates)

---

## Empirical Validation / Results

### L0: Initial Instructions

| Variant | TB Accuracy Change (pp) | SWE Accuracy Change (pp) |
|---------|--------------------------|--------------------------|
| System role | **+3.3** | **+3.9** |
| User role | +1.2 | −1.1 |

- **System-role instructions consistently outperform user-role** across both benchmarks
- Model-dependent effects: MiniMax-M2.5 gains +12.6/+8.0 points on SWE; Qwen3.8-Max declines under both
- Cost changes are modest: 1.08/1.03× baseline on TB

### L1: Execution-time Reminders

| Variant | TB Gain (pp) | SWE Gain (pp) |
|---------|--------------|---------------|
| Periodic (System) | +3.8 | +3.1 |
| After-edit (System) | +3.5 | +3.6 |
| After-edit (User) | **+5.4** | −0.3 |

- **Average output tokens decrease by 14–25% on TB** and 7–10% on SWE
- Baseline agents only check after 6.2–13.6% of editing turns—justifying interventions
- On VisualBench: L1 averages −3.2 points net preference

### L2: Completion Gates

| Variant | TB Gain (pp) | SWE Gain (pp) | TB Cost |
|---------|--------------|---------------|---------|
| Inspect changes | +0.7 | +0.5 | ~1.1× |
| Run and check (System) | +4.5 | +1.5 | — |
| Reject completion (Automated) | +4.0 | **+5.3** | 1.60–1.76× |
| Recheck (User / early) | +4.8 | +4.1 | **1.13×** |
| Reject (User / early) | **+6.0** | +6.2 | 1.48–1.56× |

- **Rejection-based designs yield largest gains but at substantial cost**
- Earlier intervention consistently improves both rechecking and rejection designs
- Agent behavior after rejection: 12–19 additional steps (median), file-changing actions in 86–96% of runs

### L3: Independent Review — The Key Finding

| Design | TB Gain (pp) | SWE Gain (pp) | Cost (TB) |
|--------|--------------|---------------|-----------|
| Claim-guided / Changes | +5.9 | +5.1 | 1.13× |
| Task-first / Changes | +6.9 | **+6.6** | 1.13× |
| Claim-guided / Trajectory | +8.4 | +6.1 | 1.25× |
| Task-first / Trajectory | **+9.5** | +3.9 | 1.25× |

**Key advantages over L2:**
- Larger gains on TB (+9.5 vs. +6.0 for best L2)
- Comparable gains on SWE (+6.6 vs. +6.2)
- **Lower cost on both benchmarks**

### VisualBench Results

| Design | Net Preference (pp) |
|--------|---------------------|
| L0 (System) | −3.2 |
| L1 (After-edit User) | −3.2 |
| L2 (Early user recheck) | +2.7 |
| L3: Changes | +21.2 |
| **L3: Changes + visual inspection** | **+23.8** |
| L3: Trajectory | +13.2 |
| L3: Trajectory + visual inspection | +21.6 |

Changes-based review outperforms trajectory-based review both with and without required visual inspection.

### Scaling Independent Review

| Setting | TB Accuracy | Mean Critic Tokens |
|---------|-------------|-------------------|
| 1 critic, 1 round (baseline) | 62.7% | 38.2k |
| 2 rounds (depth) | 65.6% | 79.5k |
| 2 critics (width) | 64.4% | 69.7k |
| 4 rounds | **66.1%** | — |
| 4 critics | 65.8% | — |

- Additional rounds/critics improve average TB accuracy by up to **3.5 percentage points**
- Four critics approach four-round accuracy at **19% lower total cost**
- Four critics: agent-side cost $0.48 vs. $0.63 for four rounds—**width consolidates feedback before revision**

---

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Verification as a test-time scaling axis**: The paper establishes that verification effort—not just sampling or model scale—can systematically improve agent performance, with diminishing returns and clear cost tradeoffs.

2. **The value of independent perspective**: The consistent superiority of L3 over L0–L2 designs demonstrates that "an independent checking perspective can address gaps in the agent's self-verification"—a critic in a fresh context can find cases the agent overlooks.

3. **Trajectory context is not uniformly beneficial**: Despite costing 29–43% more in critic tokens, full trajectories do not consistently improve outcomes on TB/SWE, though they help more on VisualBench.

### Practical Implications

1. **Design guidance for harness engineers**:
   - Use concise verification instructions in the **system prompt** (not user role)
   - Use **after-edit reminders** to focus checking without increasing output tokens
   - **Avoid completion-time self-checking**—gains are limited relative to cost
   - Deploy a **separate, low-cost critic** with task requirements and changes as default context

2. **Cost-efficient accuracy improvement**: Independent review achieves substantial accuracy gains at 1.00–1.25× baseline cost—a highly favorable tradeoff for production systems.

3. **Evidence from case studies**:
   - Critics find bugs the agent's own tests miss (e.g., date-before-IP regex case, COBOL unsigned-field arithmetic underflow)
   - Completion gates can create **invalid final states**—re-testing can change the deliverable being judged (e.g., committing test content to served branches)

---

## Conclusion

This paper provides comprehensive empirical guidance for verification design in agent harnesses:

- **L0 (initial instructions)** and **L1 (execution-time reminders)** yield modest, model-dependent coding gains without substantially increasing output tokens
- **L2 (completion gates)** can improve accuracy but may substantially increase cost
- **L3 (independent review)** offers the most favorable accuracy–cost tradeoff across all three benchmarks, with consistent improvements across models and tasks
- **Scaling verification** through additional rounds (depth) or independent critics (width) further improves accuracy, with width being more cost-efficient

**Future directions** implied by the work include:
- Studying verification designs for non-coding agent tasks
- Exploring critic selection and training for specific domains
- Investigating optimal review depth/width configurations per task type
- Integrating verification design with other test-time scaling axes (e.g., sampling, context management)

The authors conclude that independent review is a **"practical axis of test-time scaling,"** enabling users to configure verification effort for different tasks to improve agent performance at manageable cost.

---

_Markdown view of https://picx.dev/p/ptXwZi, served by PicX — AI-generated visual whiteboard summaries of research papers._