Groundability, Not Scale Alone: When Weak Reviewers Can Audit Strong Coding Agents

Summary (Overview)

• Core finding: The reliability of weak reviewers auditing strong coding agents depends on groundability—whether evidence includes independently checkable claims—rather than reviewer scale alone. Parameter count does not consistently predict review quality.
• Key contribution: A new benchmark of 411 execution-labeled real traces (GPT-5.4, Gemini, Claude-Sonnet) and 101 synthetic/controlled cases, with metrics for catch, over-rejection, and abstention.
• Upper-bound result: With official execution evidence, five of six reviewers improve both defect catch and over-rejection on held-out traces; two (GPT-OSS-120B, GPT-4.1) classify all 122 traces correctly.
• Deployable cascade: A frozen review cascade using patch-caused static errors and checked generated tests achieves coverage 0.89/0.86, risk 0.33/0.26, catch 0.76/0.80, and over-rejection 0.66/0.67 on held-out GPT-5.4 and Gemini sets.
• Main bottleneck: Producing reliable checks without official tests remains the primary limitation—one passing generated test cannot establish correctness (accept precision only 0.30).

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical problem in AI-assisted software development: coding agents (e.g., Claude Code, Cursor) return plausible patches with confident summaries that often omit required behavior. These omissions are invisible in diffs and hard to detect in long traces. In their GPT-5.4 core set, 101/154 traces are defective, with omissions as the largest defect class (76 traces).

Theoretical Framework: Groundability

The central concept is groundability: whether an independent check can answer the disputed question about a patch. The authors distinguish three evidence types:

• Testimony: The agent's own text (summaries, trace text, in-trace test output)—unreliable.
• Structured (non-grounded) evidence: Selected and organized by the review pipeline but not independently checked—raises both catch and false rejection.
• Grounded evidence: Backed by an independent check (sound static analysis or test run)—breaks the catch/over-rejection trade-off.

Key Insight

The paper builds on the principle of checkable evidence from untrusted producers, drawing from proof-carrying code (Necula and Lee 1996), certifying algorithms, and translation validation. The core idea: "do not trust the producer; check its evidence."

Methodology

Benchmark Design

Real traces (execution-labeled):

Synthetic/controlled cases (101 total): 37 reference-patch controls, 34 ablations (import/surface/partial-logic), 30 naturalistic pilot cases.

Reviewers Evaluated

Six reviewers spanning a heterogeneous size ladder: Llama-3.1-8B, Qwen-2.5-Coder-32B, Qwen-2.5-72B, GPT-OSS-120B, Qwen3-235B, and GPT-4.1.

Metrics

For execution-labeled traces:

• Catch = P(reject | defect)
• Miss rate = P(accept | defect)
• Over-rejection = P(reject | acceptable)
• Coverage = fraction of traces decided
• Risk = error rate among non-uncertain decisions (Geifman and El-Yaniv 2017)

Protocol

• Temperature 0.8, k=5 votes; majority vote decides, ties become uncertain
• Design choices frozen on 32-trace set; remaining partitions evaluated once
• Official test IDs and construction information never appear in deployable prompts

Checking Generated Tests

A generated test $p$ is kept only if it fails on the unpatched repository $b$ because the expected behavior is missing (fail-on-base check), then run on patched repository $b + \delta$:

$$\text{keep } p \iff p \text{ fails behaviorally on } b$$

Empirical Validation / Results

Raw Agent Evidence Is Unreliable

• Patch-only review: Llama-8B accepts 9/9 target omissions from diff alone; GPT-4.1 catches only 0.27
• Full raw trace: Qwen-32B catches only 0.17 of omissions—more text does not help

Structured Evidence Has a Trade-Off

• On 15 synthetic partial-logic omissions, catch rises from 0.53→1.00 (GPT-4.1) and 0.07→1.00 (Llama-8B) with "what is missing" items
• On real traces: GPT-4.1 catches 0.91 of defects but rejects 0.86 of acceptable patches

Official Checks Break the Trade-Off

Held-out upper-bound results (122 traces: 90 defects, 32 accepts):

Key observations:

• Five of six reviewers achieve both higher catch and lower over-rejection
• Qwen3-235B has the lowest unchecked catch (0.28), below Llama-8B's 0.70—scale does not predict quality
• GPT-OSS-120B and GPT-4.1 classify all 122 traces correctly

Deployable Cascade Results

Algorithm 1: Review cascade (frozen evaluation)

1. Stage 0: No code change ($\delta = \emptyset$) ⇒ reject
2. Stage 1: Patch-caused static error ⇒ reject
3. Stage 2: Generated test (≤3 attempts); keep only if base fails behaviorally; pass ⇒ accept (evaluated rule), fail ⇒ reject
4. Stage 3: Residual review by Llama-8B (k=5); uncertain ⇒ ABSTAIN

Run-once test-set results:

Critical asymmetry: A checked test that still fails on the patch has reject precision 0.82, while a passing test has accept precision only 0.30. One passing test cannot establish correctness.

Failed Hypotheses

1. Environment fidelity: Re-running failed tests in Docker recovers no additional decisions
2. Harness repair: Only moves generated-test decisions from 13 to 16/32
3. Patch-conditioned generation: No test passes on agent patch but fails on reference patch

Theoretical and Practical Implications

Theoretical Contributions

1. Groundability as a framework: The paper formalizes why evidence organization alone is insufficient—what matters is whether claims can be independently checked.

2. Scale-independence result: The heterogeneous ladder shows no monotonic relationship between parameter count and review quality, challenging assumptions that larger reviewers are inherently better auditors.

3. Reviewability ≠ Correctness: A correct patch can be hard to review without checkable evidence; a defective patch can be easy to review with a failing check.

Practical Implications

• Design guidance: Review systems should spend budget on targeted checks and use patch-caused failures directly; only unresolved cases should reach the reviewer.
• Mechanical gates: When grounded evidence gives a clear answer, use mechanical gates rather than having small models re-read evidence (the 8B reviewer misreads execution results).
• Test asymmetry: Patch-caused test failures support rejection, but passing tests are weak evidence of correctness—systems should not accept on a single passing test.

Conclusion

The paper demonstrates that weak reviewers can reliably audit strong coding agents when decisive, grounded checks are available. Official execution evidence shows the upper bound of what's possible, with two reviewers achieving perfect classification. However, the deployable cascade reveals the core bottleneck: producing reliable checks without official tests remains unsolved, with over-rejection at 0.66 and most false rejections (19/21) occurring when unresolved cases reach the reviewer stage.

Future Directions

• Better automated check generation to expand which claims can be verified
• Safer handling of unresolved cases to reduce false rejections
• More robust generated-test quality (some tests require too much, others test too little)
• Extending beyond Python issue-resolution tasks for broader validation

The practical message is clear: prioritize reliable checks and safer handling of unresolved cases over larger reviewers alone.