# Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

> Weak reviewers can reliably audit strong coding agents when grounded, independently checkable evidence is available, but scale alone does not predict review quality.

- **Source:** [arXiv](https://arxiv.org/abs/2610.01023)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/f124Nj
- **Whiteboard:** https://picx.dev/p/f124Nj/image

## Summary

# Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

## Summary (Overview)

- **Core finding**: The reliability of weak reviewers auditing strong coding agents depends on **groundability**—whether evidence includes independently checkable claims—rather than reviewer scale alone. Parameter count does not consistently predict review quality.
- **Key contribution**: A new benchmark of 411 execution-labeled real traces (GPT-5.4, Gemini, Claude-Sonnet) and 101 synthetic/controlled cases, with metrics for catch, over-rejection, and abstention.
- **Upper-bound result**: With official execution evidence, five of six reviewers improve both defect catch and over-rejection on held-out traces; two (GPT-OSS-120B, GPT-4.1) classify all 122 traces correctly.
- **Deployable cascade**: A frozen review cascade using patch-caused static errors and checked generated tests achieves coverage 0.89/0.86, risk 0.33/0.26, catch 0.76/0.80, and over-rejection 0.66/0.67 on held-out GPT-5.4 and Gemini sets.
- **Main bottleneck**: Producing reliable checks *without* official tests remains the primary limitation—one passing generated test cannot establish correctness (accept precision only 0.30).

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical problem in AI-assisted software development: coding agents (e.g., Claude Code, Cursor) return plausible patches with confident summaries that often **omit required behavior**. These omissions are invisible in diffs and hard to detect in long traces. In their GPT-5.4 core set, 101/154 traces are defective, with omissions as the largest defect class (76 traces).

### Theoretical Framework: Groundability

The central concept is **groundability**: whether an independent check can answer the disputed question about a patch. The authors distinguish three evidence types:

- **Testimony**: The agent's own text (summaries, trace text, in-trace test output)—unreliable.
- **Structured (non-grounded) evidence**: Selected and organized by the review pipeline but not independently checked—raises both catch and false rejection.
- **Grounded evidence**: Backed by an independent check (sound static analysis or test run)—breaks the catch/over-rejection trade-off.

### Key Insight

The paper builds on the principle of **checkable evidence from untrusted producers**, drawing from proof-carrying code (Necula and Lee 1996), certifying algorithms, and translation validation. The core idea: *"do not trust the producer; check its evidence."*

## Methodology

### Benchmark Design

**Real traces** (execution-labeled):
| Set | n | Composition |
|---|---|---|
| GPT-5.4 core | 154 | 53 acc. / 76 omis. / 9 regr. / 16 no change |
| design (Django) | 32 | 21 acc. / 9 omis. / 1 regr. / 1 no change |
| Gemini transfer | 59 | 15 acc. / 33 omis. / 1 regr. / 10 no change |
| GPT-5.4 multilingual | 78 | 38 acc. / 40 no change |
| Claude-Sonnet | 120 | 23 acc. / 97 no change |

**Synthetic/controlled cases** (101 total): 37 reference-patch controls, 34 ablations (import/surface/partial-logic), 30 naturalistic pilot cases.

### Reviewers Evaluated

Six reviewers spanning a heterogeneous size ladder: Llama-3.1-8B, Qwen-2.5-Coder-32B, Qwen-2.5-72B, GPT-OSS-120B, Qwen3-235B, and GPT-4.1.

### Metrics

For execution-labeled traces:
- **Catch** = P(reject | defect)
- **Miss rate** = P(accept | defect)  
- **Over-rejection** = P(reject | acceptable)
- **Coverage** = fraction of traces decided
- **Risk** = error rate among non-uncertain decisions (Geifman and El-Yaniv 2017)

### Protocol

- Temperature 0.8, k=5 votes; majority vote decides, ties become uncertain
- Design choices frozen on 32-trace set; remaining partitions evaluated once
- Official test IDs and construction information never appear in deployable prompts

### Checking Generated Tests

A generated test $p$ is kept only if it **fails on the unpatched repository** $b$ because the expected behavior is missing (fail-on-base check), then run on patched repository $b + \delta$:

$$\text{keep } p \iff p \text{ fails behaviorally on } b$$

## Empirical Validation / Results

### Raw Agent Evidence Is Unreliable

- Patch-only review: Llama-8B accepts **9/9** target omissions from diff alone; GPT-4.1 catches only 0.27
- Full raw trace: Qwen-32B catches only 0.17 of omissions—more text does not help

### Structured Evidence Has a Trade-Off

- On 15 synthetic partial-logic omissions, catch rises from 0.53→1.00 (GPT-4.1) and 0.07→1.00 (Llama-8B) with "what is missing" items
- On real traces: GPT-4.1 catches 0.91 of defects but rejects 0.86 of acceptable patches

### Official Checks Break the Trade-Off

**Held-out upper-bound results (122 traces: 90 defects, 32 accepts):**

| Reviewer (size) | risk notes | structured catch | structured over-rej. | official catch | official over-rej. |
|---|---|---|---|---|---|
| Llama-3.1 (8B) | removed | 0.70 | 0.75 | 0.98 | 0.00 |
| Qwen-2.5 (32B) | kept | 0.69 | 0.66 | 0.96 | 0.09 |
| Qwen-2.5 (72B) | kept | 0.43 | 0.44 | 0.91 | 0.47 |
| GPT-OSS (120B) | removed | 0.51 | 0.41 | 1.00 | 0.00 |
| Qwen3 (235B) | removed | 0.28 | 0.19 | 0.90 | 0.00 |
| GPT-4.1 | removed | 0.69 | 0.62 | 1.00 | 0.00 |

**Key observations:**
- Five of six reviewers achieve both higher catch and lower over-rejection
- Qwen3-235B has the *lowest* unchecked catch (0.28), below Llama-8B's 0.70—**scale does not predict quality**
- GPT-OSS-120B and GPT-4.1 classify all 122 traces correctly

### Deployable Cascade Results

**Algorithm 1: Review cascade** (frozen evaluation)
1. **Stage 0**: No code change ($\delta = \emptyset$) ⇒ reject
2. **Stage 1**: Patch-caused static error ⇒ reject
3. **Stage 2**: Generated test (≤3 attempts); keep only if base fails behaviorally; pass ⇒ accept (evaluated rule), fail ⇒ reject
4. **Stage 3**: Residual review by Llama-8B (k=5); uncertain ⇒ ABSTAIN

**Run-once test-set results:**

| Set | coverage | risk | catch | over-rej. |
|---|---|---|---|---|
| held-out (121) | 0.89 [0.84,0.94] | 0.33 [0.25,0.43] | 0.76 [0.67,0.85] | 0.66 [0.49,0.81] |
| Gemini (59) | 0.86 | 0.26 | 0.80 | 0.67 |

**Critical asymmetry**: A checked test that still fails on the patch has **reject precision 0.82**, while a passing test has **accept precision only 0.30**. One passing test cannot establish correctness.

### Failed Hypotheses

1. **Environment fidelity**: Re-running failed tests in Docker recovers no additional decisions
2. **Harness repair**: Only moves generated-test decisions from 13 to 16/32
3. **Patch-conditioned generation**: No test passes on agent patch but fails on reference patch

## Theoretical and Practical Implications

### Theoretical Contributions

1. **Groundability as a framework**: The paper formalizes why evidence organization alone is insufficient—what matters is whether claims can be independently checked.

2. **Scale-independence result**: The heterogeneous ladder shows no monotonic relationship between parameter count and review quality, challenging assumptions that larger reviewers are inherently better auditors.

3. **Reviewability ≠ Correctness**: A correct patch can be hard to review without checkable evidence; a defective patch can be easy to review with a failing check.

### Practical Implications

- **Design guidance**: Review systems should spend budget on targeted checks and use patch-caused failures directly; only unresolved cases should reach the reviewer.
- **Mechanical gates**: When grounded evidence gives a clear answer, use mechanical gates rather than having small models re-read evidence (the 8B reviewer misreads execution results).
- **Test asymmetry**: Patch-caused test failures support rejection, but passing tests are weak evidence of correctness—systems should not accept on a single passing test.

## Conclusion

The paper demonstrates that **weak reviewers can reliably audit strong coding agents when decisive, grounded checks are available**. Official execution evidence shows the upper bound of what's possible, with two reviewers achieving perfect classification. However, the deployable cascade reveals the core bottleneck: **producing reliable checks without official tests** remains unsolved, with over-rejection at 0.66 and most false rejections (19/21) occurring when unresolved cases reach the reviewer stage.

### Future Directions

- Better automated check generation to expand which claims can be verified
- Safer handling of unresolved cases to reduce false rejections
- More robust generated-test quality (some tests require too much, others test too little)
- Extending beyond Python issue-resolution tasks for broader validation

The practical message is clear: **prioritize reliable checks and safer handling of unresolved cases over larger reviewers alone**.

---

_Markdown view of https://picx.dev/p/f124Nj, served by PicX — AI-generated visual whiteboard summaries of research papers._