# VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

> VerifierBench shows LLM-generated verifiers reject invalid solutions reliably but accept valid ones poorly, with most failures from unjustified checks that self-testing fails to expose.

- **Source:** [arXiv](https://arxiv.org/abs/2610.28679)
- **Published:** 2026-11-14
- **Permalink:** https://picx.dev/p/gju641
- **Whiteboard:** https://picx.dev/p/gju641/image

## Summary

# VerifierBench: Benchmarking the Generation of Executable Verifiers for LLM Agents

## Summary (Overview)

- **New benchmark**: Introduces VerifierBench, a benchmark of 222 tasks from six agent benchmarks (BFCL v4, ACEBench, τ²-bench, AppWorld, AgentDojo, SWE-bench Verified) with 4,993 hidden test cases for evaluating verifier generation by LLM agents.
- **Verifier4Verifier (V4V) audit**: A six-check screening procedure (replay, repeatability, provenance, input-label consistency, trivial baselines, mutation testing) that validates source verifiers and test suites, excluding 45 of 267 initial tasks due to confirmed source defects.
- **Key finding**: Across 19 models, normalized balanced accuracy (BA_norm) ranges from 10.6 to 62.1, with generated verifiers consistently rejecting invalid solutions (TNR: 79.2–96.3%) more reliably than accepting valid ones (TPR: 21.1–70.2%).
- **Root cause analysis**: False rejections concentrate in "all-reject" verifiers (87.9–96.0% of false rejections) caused by unjustified checks; self-testing fails to expose these errors because tests share the verifier's mistaken assumptions or never confirm acceptance of any valid solution.
- **Public resources**: Dataset, code, and project page are publicly available.

## Introduction and Theoretical Foundation

### Background and Motivation

In **data-oriented recursive self-improvement (RSI)**, agents construct training data for their own subsequent learning and can jointly generate tasks and verification functions that provide reinforcement learning rewards. In this paradigm:

- Verifiers judge candidate solutions, determining which receive rewards or enter the training data
- An incorrect verifier can **repeatedly reward a violation or discard a valid solution**
- The reliability of agent-generated verifiers is therefore critical to the learning process

### Problem Formulation

The paper studies **verifier generation**: given the instruction and environment a solver would receive, an agent writes an executable verifier that returns **PASS** or **FAIL** for any candidate solution. Unlike a solver (which needs one valid solution), a verifier must:

> *"Accept every valid solution while rejecting invalid ones. It must therefore distinguish task requirements from choices that can vary across valid solutions."*

### Key Challenge

A verifier that accepts one expected solution may still reject other valid ways to complete a task, while one that catches ordinary mistakes may still accept solutions that exploit gaps in its checks. Evaluation therefore requires testing **both** acceptance of valid alternatives **and** rejection of invalid solutions, including deliberate attacks.

## Methodology

### Task Formulation

For task $t$, an agent receives the instruction and environment used for ordinary task solving, plus the verifier's input/output conventions. A harness (agent framework managing model calls, tool use, and generation loop) runs the agent until it delivers an executable verifier $\hat{V}_t$; the reference verifier and hidden test cases are withheld.

**Verifier types**:
- **Tool-call verifiers**: Receive raw call text with parsed function names and arguments
- **State verifiers**: Receive candidate's interaction record or solution code
- **Code verifiers**: Receive candidate patch and repository checkout

### Test Case Construction

Test cases form four groups with construction labels (PASS/FAIL):

| Test-case group | Label | Cases | Content |
|---|---|---|---|
| Reference solution | PASS | 222 | The designated solution (gold) |
| Valid variants | PASS | 1344 | Cosmetic changes (840) or alternative implementations/paths (504) |
| Ordinary errors | FAIL | 2629 | Initial, partial, near-miss, or regressing solutions |
| Attacks | FAIL | 798 | Exploited checking omissions, fabricated success signals, or interference |

### Verifier4Verifier (V4V) Screening

Six checks applied to constructed test cases and source verifiers:

1. **Replay**: Run source verifier on every test case; compare decision with construction label
2. **Repeatability**: Rerun on same case from same reset state; decision must not change
3. **Provenance**: Link every recorded decision to test-case contents and verifier version
4. **Input-label consistency**: Cases giving identical information must not carry opposite labels
5. **Trivial baselines**: Always-accept, always-reject, or reference-only verifiers must be exposed
6. **Mutation testing**: Modify one check in source verifier; some test case must receive different valid decisions

### Metrics

The primary metric is **normalized task-averaged balanced accuracy**:

$$
\mathrm{BA}_{\mathrm{norm}} = 2\mathrm{BA} - 100 = \mathrm{TPR} + \mathrm{TNR} - 100 \tag{1}
$$

where $\mathrm{BA} = (\mathrm{TPR} + \mathrm{TNR})/2$. Constant valid decisions score 0, perfect agreement scores 100.

### Experimental Setup

- **Main comparison**: 19 models in one common harness (Pi)
- **Alternative configurations**: 17 additional model–harness pairs (native harnesses: Claude Code, Codex, Gemini CLI; third-party: OpenClaw, mini-swe)
- **Generation limits**: 5,400 seconds per task, 32,768 output tokens per reply
- **Evidence sources**: Main release (final judgments), archived records (per-task judgments), Astra trace batch (generation traces for GPT-6-astra)

## Empirical Validation / Results

### Main Results

**BA_norm ranges from 10.6 to 62.1** across 19 models with the Pi harness:

| Model | BA_norm | TPR | TNR | FPR_h |
|---|---|---|---|---|
| GPT-6-astra | 62.1 | 67.4 | 94.6 | 7.4 |
| Claude Opus 4.8 | 54.0 | 67.7 | 86.3 | 13.5 |
| Kimi K3 | 53.4 | 70.2 | 83.2 | 13.4 |
| GPT-5.6-sol | 52.9 | 56.8 | 96.0 | 0.8 |
| GPT-5.6-terra | 52.9 | 56.5 | 96.3 | 3.6 |
| ... | ... | ... | ... | ... |
| Doubao Seed 2.0 Mini | 10.6 | 21.1 | 89.5 | 3.7 |

**Key pattern**: Every model has lower TPR than TNR. Explicit FAIL accounts for 57.1–100% of unaccepted positive cases, indicating active rejection rather than missing decisions.

### Task-Type Differences

- **Widest gap** on state and code tasks: TPR averaged over models is 27.7% and 40.2%, respectively
- **Tool-call tasks**: TPR of 79.2% on average
- Low attack acceptance does not imply reliability: GPT-5.6-sol has FPR_h = 0.0% on state tasks but accepts only 23.0% of positive cases

### Harness Effects

Native harnesses change scores in both directions:
- **Claude Code raises Claude Opus 4.8** by 12.2 points to 66.2 (best of all 36 configurations)
- **Gemini CLI raises Gemini 3.5 Flash** by 15.0 points
- **Codex lowers all four GPT models** by 4.1–26.6 points

### Reasoning Effort

Higher reasoning effort barely changes aggregate scores (+0.5 BA_norm with Pi, −1.3 with Claude Code) but hides opposite changes by task type: HIGH effort improves code tasks (+3.8, +1.4) but worsens state tasks (−5.4, −3.6). Higher acceptance does not imply better discrimination: on Pi tool-call tasks, HIGH effort accepts 63 more positives but introduces 128 false acceptances.

### Concentration of False Rejections

**All-reject verifiers** (returning valid FAIL on every test case of their task) contribute **87.9–96.0%** of each model's false rejections. In the Astra trace batch, 68–75 of 222 verifiers are all-reject, contributing 93.3–95.5% of false rejections.

### Case Study: Unjustified Checks

| Unjustified check | Tasks | Verifiers | Positive FAIL |
|---|---|---|---|
| **Self-tests used only allowed solutions** | | | |
| Overbroad source-file protection | 2 | 3 | 12 |
| Rejects permitted representations | 3 | 5 | 35 |
| **No positive self-test passed** | | | |
| Rejects authorized user-tool calls | 3 | 3 | 27 |
| Omits permitted read-only tools | 2 | 2 | 18 |
| Incomplete category vocabulary | 2 | 3 | 27 |
| **Total** | **12** | **16** | **119** |

**Concrete examples**:
- **Tool allowlist omission**: A verifier for τ²-bench banking task 044 omits the permitted read-only query `get_user_dispute_history_7291` from its allowlist, treating it as an "Unrelated or unsupported mutation" and rejecting all 9 positive cases
- **Source-file protection**: Matplotlib verifiers exclude legal changes under `lib/mpl_toolkits/`, including the 3D axes implementation
- **Category vocabulary**: A retail verifier identifies office items with a keyword list omitting "Headphones," counting only two office items instead of three

### Self-Testing Failures

**Confirmatory self-testing**: Tests confirm the verifier accepts a chosen solution but never ask whether its other conditions follow from the task. For example, in Django-11734, the positive self-test patches `query.py` but never edits an `__init__.py` file, so the unjustified source-file restriction remains unchallenged.

**Incomplete testing**: Eight verifiers' agents ran intended-positive self-tests but never obtained a complete positive PASS. In the running example, the intended-positive self-test fails when replay is unavailable after budget exhaustion, leaving the tool restriction untested.

## Theoretical and Practical Implications

### For Verifier Generation

Self-testing serves two goals:
1. **Confirming acceptance** of a chosen solution
2. **Checking non-rejection** of other solutions the task allows

The second requires tracing every acceptance condition to the instruction, tool definitions, or environment rules, and testing it with cases derived from those requirements. Key recommendations:

> *"One test varies a permitted property while keeping the task satisfied... another violates one required condition. Examples drawn from solutions a check already allows cannot challenge that check."*

### For Data-Oriented RSI

In data-oriented recursive self-improvement, where generated verifiers select training data:

> *"Where generated verifiers select training data, as in data-oriented RSI, such checks could repeatedly exclude legitimate solution strategies."*

### For Evaluation Methodology

- Scores measure agreement with retained reference verifiers on finite, constructed test suites
- Cross-harness differences describe configurations, not model capability alone
- Rankings are point estimates without repeated generations or significance tests

## Conclusion

VerifierBench evaluates whether agents can write verifiers from the information available for solving a task. Across all 36 configurations:

1. **Generated verifiers accept valid solutions less reliably than they reject invalid ones**
2. **Most false rejections come from all-reject verifiers** that reject every case of their task
3. **Unjustified checks** and **self-tests that either confirm only allowed solutions or never confirm acceptance** explain these failures

The findings motivate testing generated verifiers on **permitted variations**, not only on a chosen solution. Future directions include evaluating the effectiveness of the suggested testing strategies and developing methods to challenge each acceptance condition against task requirements.

### Limitations

- V4V excludes 45 tasks with confirmed source defects, but agreement between construction labels and reference decisions does not certify semantic correctness of every label
- Cross-harness differences do not isolate any harness component
- The code and trace analysis covers verifiers selected to explain observed failures (mostly GPT-6-astra), identifying mechanisms but not their prevalence
- Rejections are attributed by source-code inspection rather than intervention experiments

---

_Markdown view of https://picx.dev/p/gju641, served by PicX — AI-generated visual whiteboard summaries of research papers._