When the Reward Suite is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

Summary (Overview)

• Core contribution: A preregistered two-arm causal experiment isolating what a deployed test suite's natural false positives (FPs) do to RLVR training, measuring the gap between reward-side leakage and held-out capability across three model families (Qwen2.5-Coder-1.5B, deepseek-coder-1.3b, Llama-3.2-1B) at 400–800 GRPO steps.
• Key finding [C]: The average held-out effect of training on a leaky suite is bounded — non-inferior under a preregistered 1.5-pt margin (gap 0.20 pt, one-sided 95% upper bound 0.75 pt), consistent across all six estimators in all three families.
• Key finding [C]: A cheap static audit (sampling 16 base-model solutions per task pre-training) predicts where rewarded-FP mass will land (Spearman ρ = 0.80 raw, 0.79 under difficulty control), with train-side FP share +43.8 pt higher on leak-stratum tasks than clean tasks.
• Key finding [E]: Exhaustive human-adjudicated audit of all 2,821 rewarded FPs in family A finds 47.57% are verified genuinely wrong code (task-cluster bootstrap-95 [36.4, 60.5]) — the reward pays for real bugs, not merely suite artifacts; replication families show 45.37% and 62.78%.
• Key finding [E]: Mechanism evidence points to selection of pre-existing error modes, not learned exploitation: FPs are flat across training (not growing), identical wrong outputs appear in untrained base models, and no hacking signatures are detected; the 8.37-pt reward inflation has no measurable held-out counterpart.

---

Introduction and Theoretical Foundation

Background and Motivation

RLVR (Reinforcement Learning from Verifiable Rewards) trains code models against test suites, treating a pass as ground truth (Lambert et al., 2024; DeepSeek-AI, 2025). However, deployed test suites are imperfect: a recent audit found 25–28.5% of audited tasks in public benchmarks accept at least one wrong solution (Rajan, 2026). That audit measured the suites, not training dynamics, leaving open what the author named "a direct causal test (training on fixed-versus-untouched broken tasks at fixed compute)."

The Critical Distinction: Natural FPs vs. Label Noise

The paper's theoretical foundation rests on distinguishing natural false positives from the label noise studied in robustness literature:

"The verifier errors we study are not label noise. When a deployed suite accepts a wrong solution, it accepts it every time—the same wrong program passes the same weak tests on every rollout. These natural false positives (FPs) are therefore per-task and persistent, and the error runs in one direction only: wrong code gets rewarded."

Noise-robustness analyses (Plesner et al., 2026) model verifier error as random flips redrawn per attempt, which average out over training — RLVR tolerates up to 15% such noise. A persistent FP does not average out; it constitutes a standing reward for a specific wrong program on a specific task.

Theoretical Anchor: Youden Index Phase Transition

The paper connects to Rad et al.'s (2026) theory: RLVR under a noisy verifier exhibits a phase transition in the verifier's Youden index:

$$J = \mathrm{TPR} - \mathrm{FPR}$$

• Incorrect modes amplify when $J < 0$
• Incorrect modes die out when $J > 0$

A deployed-but-weak suite (too accepting but still separating right from wrong on average) sits inside the $J > 0$ region, predicting no amplification — which the paper's three-family measurements confirm, with family B even drifting toward the predicted decay direction.

---

Methodology

Experimental Design: Two-Arm Causal Contrast

The design isolates exactly one component — the test suite paying the reward:

Key operational detail: The two suites are mutually exclusive — on 313/378 tasks, base and extra tests are fully disjoint. The hardened arm is scored by extra tests only (not the official EvalPlus union scoring).

Three Families (Preregistration-Governed)

• Family A: Qwen2.5-Coder-1.5B-Instruct (deepest audit — all 2,821 FPs human-adjudicated)
• Family B: deepseek-coder-1.3b-instruct (preregistered as the decision family)
• Family C: Llama-3.2-1B-Instruct

Families B and C were registered before any of their data existed.

Static Leakiness Audit (Pre-Training)

Before training, 16 samples per task are drawn from the base model and scored against both suites. A task's leakiness is:

$$\text{leakiness} = \frac{\#\{\text{MBPP-passing samples that fail MBPP+}\}}{\#\{\text{MBPP-passing samples}\}}$$

This audit is cheap (one sampling pass per family) and selects leak-stratum vs. clean tasks.

Verified-Wrong Audit (Post-Hoc, Exhaustive)

All 2,821 rewarded FPs in family A were classified under signed predicate specifications as:

1. Verified-wrong: genuinely incorrect programs
2. Plus-suite false negatives: correct code rejected by defective extra tests
3. Contract-invalid test inputs
4. Canonical-convention mismatches
5. Resource-limit artifacts

The specification went through four signed generations, with the verified-wrong share falling 68.84% → 59.91% → 47.57% — a self-penalizing audit trail.

---

Empirical Validation / Results

1. Average Held-Out Effect [C] — Non-Inferiority

With gap defined as hardened-minus-leaky held-out pass rate over 292 paired tasks:

• Gap at step 400: 0.20 pt (CI90 [-0.42, +0.74])
• One-sided 95% upper bound: 0.75 pt — half the preregistered 1.5-pt margin
• All six estimator variants (percentile, basic, BCa, hierarchical, seed-t, sign-flip) have uppers in 0.70–1.07 pt
• Leaky-arm FP mass does not trend upward (quarter means 14.5% → 15.2%; trend CI90 [-0.012, +0.026])
• Both arms improve over base model (+2.3–2.9 pt held-out MBPP), so the bounded gap is not two failures together

2. Leakiness Tracking [C] — The Audit Predicts Where Reward Leaks

• Spearman ρ = 0.80 raw (task-level bootstrap-95 [0.74, 0.85], n=216 tasks); ρ = 0.79 after controlling for difficulty
• OLS decomposition: leakiness coefficient 0.896 vs. difficulty -0.049
• Restricted to nonzero-leakiness tasks: ρ = 0.920 (n=73)
• All three families replicate: ρ = 0.813 (B), 0.700 (C), all p < 10⁻¹²

3. Verified-Wrong Share [E] — What the Leaked Reward Paid For

Table 1: Verified-wrong share of rewarded FP mass [E]

Family A's interval is wide because FP records cluster by task; 94.3% of audited tasks contain at least one verified-wrong record. Naive counting (no artifact exclusion) roughly doubles all counts.

4. Stratification Results

Train-side [C] (preregistered P3 test):

• Leak-stratum tasks: 45.9% of rewarded rollouts are FPs
• Clean tasks: 2.1%
• Gap: +43.8 pt (seed t-CI90 [43.05, 44.57]) — classifier-free, invariant to audit revisions

Eval-side [E] (permanently exploratory): On 47 exposed held-out tasks, hardened arm ends +1.60 pt ahead (seed t-CI90 [+0.37, +2.82], 5/5 seeds positive). BUT: concentration is severe (top 5 carrier tasks hold 60.0% of the summed gap; dropping them crosses zero), and the specification-search correction (106 valid specifications) prices only one axis of freedom.

5. Mechanism Evidence [E] — Selection, Not Learning

Four independent observations converge:

1. No growth: Rewarded-FP mass flat (A, C) or declining (B); FP-only group share does not grow
2. No fixation: On the largest channel, 55/55 verified-wrong records are distinct programs
3. Base-model pre-existence: Untrained base models already produce the characteristic wrong outputs under the leaky filter (9/11 channels at or above trained rates — when filtered identically); the leaky filter, not training, does the selecting
4. No sub-behavioral sharpening: Per-token log-probability sharpening of FP vs. genuine completions is null after length correction (families B and C; floor 0.001–0.0024 nats/token)

6. Cross-Family Replication [C]

Table 2: Coarse-axis concordance across model families

All three preregistered criteria pass on family B (the decisive family). Eval-side task-level effects do not transfer: family B's exposed-stratum gap is reversed (-1.389 pt), and family C is underpowered. Family B's FP-mass trend is negative — anti-exploitation — which fails its "benign" clause in the good direction.

7. 800-Step Continuation [C]

Growth is refuted at twice the horizon: no run of 12 fires any of the frozen growth criteria (positive FP-only-share slope, second-half above first, secondary measure agreement). Family B is adequately powered (MDE +0.43 pp/100 steps vs. observed ≈ 0); family C is an underpowered non-detection (MDE +1.0 pp/100 steps).

---

Theoretical and Practical Implications

Theoretical: Where Deployed Suites Sit on the Noise Map

The results map cleanly onto the Youden-index phase transition theory. A weak-but-separating suite ($J > 0$) predicts no amplification — confirmed in all three families. The flat FP share in families A and C (vs. theory's asymptotic die-out) is flagged as an open resolution question.

Practical: Audit Economics and Reward QA

• FP mass is highly concentrated: Gini coefficient 0.79; repairing the top 18 task suites removes ~half of FP mass
• The static audit is upgradable for free: training rollouts act as natural mutants, flagging 74 tasks the static probe called clean (zero reversals; exact one-sided McNemar p ≈ 5.3 × 10⁻²³) — but this improves localization, not volume (14.95% vs. 14.92% escape mass)
• Hardening removes measurement inflation, not capability gaps: the 8.37-pt reward gap is the price of the leak; the hardened arm doesn't pay it, but held-out ability moves only 0.20 pt

The Practitioner Checklist

1. Run the static audit before RLVR: one sampling pass predicts where rewarded FP mass will land (ρ = 0.80/0.81/0.70)
2. Read suite failures through the taxonomy: ~half of FP mass was real bugs; naive counting doubles the number
3. Do not read train-task reward as capability: 8.37-pt inflation does not appear held-out
4. Harden for measurement, not capability: repair is cheap where mass concentrates

The Recursive Implication for Audits

The meta-audit's exploratory finding — that frontier judges separate their own FPs only weakly (AUC .617/.639) versus their hetero-judge anchors on weaker policies' FPs (.894/.786) — suggests error legibility may fall as generator strength rises. Two design principles follow for oversight without a stronger reader: heterogeneous redundancy (decorrelated failure modes) and mechanical anchoring (pushing load-bearing claims down to checkable artifacts).

---

Conclusion

Main Takeaways

This paper's contribution is a measurement layer: a preregistered causal contrast isolating what a deployed suite's natural FPs do to training, with instruments separating where reward leaks, what it pays for, and whether training climbs it. The headline synthesis:

"The reward stream exploits genuine false positives of the deployed suite — where 'exploits' means collects reward from, and the collection is selection rather than learning."

At 1–1.5B parameters and 400–800 steps, a weak suite is a measurement problem before it is a capability problem, and the measurement problem is auditable before training starts. The leak is: (1) predictable (static audit ρ = 0.80), (2) substantive (~half genuinely wrong code), (3) selected not learned (flat across training, present in base models), and (4) bounded in held-out damage (< 1.5 pt margin).

Future Directions and Open Questions

• Scale and horizon: Whether exploitation of natural FPs grows at larger scale or longer horizons is a falsifiable continuation hypothesis — the open axis is scale, not steps; the task axis (same 250 tasks reused) is untested
• Causal separation on eval side: "Does the leak damage these tasks?" vs. "Do these tasks attract FPs for other reasons?" requires per-sample dumps and retained checkpoints (absent in family A)
• The meta-audit's frontier question: whether stronger generators produce less legible errors (motivation from §5, not a law) — two subjects license nothing about frontier models in general
• Sub-behavioral effects: parameter-level sharpening below the 0.001–0.0024 nats/token floor remains unprobed
• Suite defects: "Hardened" ≠ perfect — plus-suite false negatives have their own, smaller error floor, quantified but not zero

The Paper's Own Discipline

Every claim is tagged [C] (confirmatory, preregistered), [E] (exploratory, post-hoc), or [I] (instrument-validity). The preregistration chain includes its failures (two voided launch gates), and the audit is self-penalizing by construction — every added artifact predicate can only shrink the verified-wrong share. The bound is stated one-way: "it says what did not happen in 400 steps at this scale, not what cannot happen."