# Auditing Reward Hackability in Code RL Training Environments

> Docker-verified test-suite audits reveal 28.5% of SWE-bench Verified tasks accept incorrect patches, inflating Pass@1 by +14.14 percentage points across 134 frontier models.

- **Source:** [arXiv](https://arxiv.org/abs/2606.16062)
- **Published:** 2026-08-22
- **Permalink:** https://picx.dev/p/0lid0N
- **Whiteboard:** https://picx.dev/p/0lid0N/image

## Summary

## Summary (Overview)

- **Cross-benchmark hackability rates**: 28.5% of SWE-bench Verified tasks (14/49) and 25.0% of R2E-Gym tasks (4/16 decisive) accept Docker-verified incorrect patches as correct, demonstrating widespread test-suite weakness in code RL training environments.
- **Meta-analytic validation**: Across 134 frontier model submissions to SWE-bench Verified, flagged-hackable tasks inflate within-difficulty-stratum Pass@1 by **+14.14 percentage points** (95% CI [+11.80, +16.48]; one-sided $p < 10^{-6}$; $I^2 = 0\%$), with 123 of 134 models showing positive effects.
- **Critical methodology finding**: An inline LLM judge alone endorsed 10/11 broken tasks as "upgraded," but Docker re-verification revealed a **61.9% per-augmentation defect rate** (65/105 decisive LLM-generated tests fail on the gold patch itself) that the judge completely missed.
- **Repair procedure**: A Docker gold-sanity gate (running each generated test against the gold solution before judge consultation) combined with retry converges 9/11 broken tasks to gated upgrades; ablation shows retry presence—not diversity bias or judge resolution—is the load-bearing component.
- **Four-way triangulation**: Independent measurements (this audit, OpenAI's retirement note, Berkeley's trustworthy-env, and the meta-analysis) all confirm substantial benchmark unreliability in code RL.

## Introduction and Theoretical Foundation

Reinforcement learning from execution feedback (RLVR) works only when the verifier is correct. If a task's test suite accepts a solution that does not solve the problem, the policy receives reward for the wrong behavior—a mechanism Anthropic's November 2025 report ties to broader emergent misalignment, including alignment faking and sabotage [8].

Prior audits of SWE-bench Verified are insufficient for operational use:
- A manual sweep by 93 Python developers at benchmark release [6]
- OpenAI's February 2026 announcement that 59.4% of failed tasks have flawed tests [10]

Neither produces the operational number a curator needs: **of the tasks in a code RL training set, what fraction reward the model for an incorrect solution?**

The paper builds on four existing components:
1. NVIDIA-style verifier scoring [2]
2. Semi-formal reasoning for LLM-based code judging (Ugare and Chandra) [13]
3. EvolveCoder-style iterative attacks [11]
4. Human-difficulty stratification [1]

The novel contributions are empirical: cross-benchmark hackability rates, the 134-model meta-analytic validation, and the per-augmentation defect rate.

## Methodology

### 3.1 Exploit Generation

For each task, $K$ candidate incorrect source patches are generated using a frontier LLM (Claude Sonnet 4). Each candidate is a unified diff (SWE-bench) or file_replacements mapping (R2E-Gym) that passes the existing test suite while changing observable behavior. Rounds 2–3 use failure logs from round 1 to bias generation toward unblocked exploits (iterative-attacker pattern). Candidates are applied inside Docker via the project's harness; a task is **hackable** if at least one candidate passes the native test runner.

### 3.2 Environment Quality Score (EQS)

$$\operatorname{EQS}(t) = 0.35\,V(t) + 0.30\,(1 - H(t)) + 0.20\,F_1(t) + 0.15\,D(t) \tag{1}$$

where:
- $V(t)$: NVIDIA-style verifier discrimination score over candidate patches
- $H(t)$: empirical hackability (fraction of exploit candidates that succeed)
- $F_1(t)$: three-way confusion-matrix F1 from a semi-formal LLM judge combined with execution (CORRECT vs. PARTIAL vs. INCORRECT; PARTIAL counts as incorrect)
- $D(t)$: learnability signal from external per-task Pass@1 across model submissions

$$\operatorname{verdict}(t) = \begin{cases} \text{KEEP} & \text{EQS}(t) > 0.70 \\ \text{FIX} & 0.40 \leq \text{EQS}(t) \leq 0.70 \\ \text{DROP} & \text{EQS}(t) < 0.40 \end{cases} \tag{2}$$

Weights reflect prior on signal directness (not tuned on validation data); headline claims are weight-independent.

### 3.3 Sanity-Gated Repair Loop

**Stage 1 (Gate)**: Injected test runs against gold solution in Docker via native runner. Failure on gold → augmentation discarded, retry triggered (temperature 0.7, diversity-biased prompt). Success → proceeds to Stage 2.

**Stage 2 (Judge)**: 3-sample self-consistency variant of semi-formal reasoning, sampled at temperatures (0.2, 0.4, 0.6). Majority vote determines verdict; BLOCKS verdict counts exploit as covered. Tasks with all exploits covered convert FIX → KEEP.

## Empirical Validation / Results

### Hackability Rates

**SWE-bench Verified (n=49)**: 14/49 tasks hackable (**28.5%**), spanning astropy (6) and django (8). Round-1 single-shot: 18.4% (9/49); rounds 2–3 add 5 tasks. Mean EQS = 0.806; verdicts: KEEP 38, FIX 11, DROP 1. Audit cost: \$6.04 API.

**R2E-Gym (n=20, 6 repositories)**: 4/16 decisive tasks hackable (**25.0%**) at $K=1$ single-shot (lower bound). Hackable: 2 numpy, 1 aiohttp, 1 pandas. Cost: \$8.29 API.

### Meta-Analytic Validation

Per-model effect:

$$\Delta_m = \mathrm{Pass@1}_m^{\mathrm{hack}} - \mathrm{Pass@1}_m^{\mathrm{robust}} \tag{3}$$

Pooled via DerSimonian-Laird random-effects meta-analysis:

$$\hat{\Delta} = \frac{\sum_m w_m \Delta_m}{\sum_m w_m}, \quad w_m = \frac{1}{\sigma_m^2 + \hat{\tau}^2} \tag{4}$$

**Result**: $\hat{\Delta} = +14.14\,\text{pp}$, 95% CI [+11.80, +16.48], $p < 10^{-6}$, $I^2 = 0\%$. Sign test: 123/134 models positive. Stratification by human difficulty retains 55% of the unstratified effect (+25.64 pp). Subgroup analyses (era, Pass@1 quartile, scaffold) all same direction.

### Augmenter Defect Rate

**Un-gated run**: LLM judge reported 10/11 tasks upgraded (mean EQS 0.570 → 0.816, \$5.57, 3 iterations).

**Docker re-verification** (stratified 8-pair sample): 1 vindicated, 6 INVALID_AUG (sanity FAIL on gold), 1 methodology edge case. All 6 INVALID_AUG cases were endorsed as BLOCKS by the judge.

**Gated run**: Gate flags **65/105 decisive augmentations** (61.9% defect rate); 40 PASS, 65 FAIL, 3 ERROR, 3 SKIPPED. Converges 9/11 tasks (mean EQS 0.570 → 0.797, \$3.60).

### Ablation Results

| Configuration | Upgraded | Cost (API) | Source |
|---|---|---|---|
| Judge only (no gate, no retry) | 10 / 11 | $5.57 | live un-gated run |
| Gate + judge (no retry) | 3 / 11 | — | retroactive projection |
| Gate + retry + judge, full | 9 / 11 | $3.60 | live full-loop run |
| Ablate diversity bias (neutral retry, T=0.3) | 9 / 11 | $4.39 | live ablation run |
| Ablate judge resolution (1 sample at T=0.4) | 8 / 11 | $2.13 | live ablation run |

**Key finding**: Retry presence is load-bearing; diversity bias is not. The 3-sample judge is worth 1 task over single-sample.

### Table 1: The 6 INVALID_AUG Cases

| Task | Failure mode |
|---|---|
| astropy__astropy-7166 | Wrong import style: `Base(metaclass=InheritDocstrings)` vs. module-level `misc.InheritDocstrings` |
| astropy__astropy-7336 | Inverted expectation: `@u.quantity_input` correctly raises on None; augmenter asserted opposite |
| django__django-11066 | `_rename` returns early when entity missing on target DB; gold-fixed `save(using=db)` never reached |
| django__django-11095 | Asserts `sig.parameters['obj'].default == None`; actual gold default is `inspect._empty` |
| django__django-11276 | Inverted function purpose: `escape()` encodes Unicode to entities, not decodes |
| django__django-11451 | `authenticate(username=None, ...)` with `username_field = 'username'` raises TypeError (duplicate keyword) |

## Theoretical and Practical Implications

**Measurement vs. deployment**: The 28.5% and 25.0% rates measure test-suite acceptance of wrong answers, not whether deployed models produce them. The +14.14 pp meta-analysis bridges this gap, showing frontier models do produce wrong-answer-but-passes patterns on flagged tasks at higher rates.

**Methodology caution**: LLM judges evaluating LLM-generated test code are fundamentally different from judging source code against execution-derived ground truth. The judge correctly reasoned about what tests *would* check "if they ran"—but failed to catch that tests don't run, run incorrectly, or assert the opposite of documented behavior.

**Practical pipeline**: A curator using this pipeline at scale should expect augmenter-class defects that an LLM judge alone cannot catch. The Docker gold-sanity gate provides the per-augmentation accept/reject signal that closes this gap, at lower cost (\$3.60 vs. \$5.57) because it filters defective augmentations before expensive judge consultation.

**Four-way triangulation** (each with different denominator and attack surface):
1. This audit: 28.5% hackability (n=49, Oct 2025)
2. OpenAI: 59.4% flawed-on-failed (Feb 2026)
3. Berkeley trustworthy-env: 45 process-isolation exploits across 8 benchmarks (Apr 2026)
4. This meta-analysis: +14.14 pp inflation across 134 models (Jun 2026)

## Conclusion

**Main takeaways**:
1. A substantial fraction of code RL benchmark tasks (28.5% SWE-bench Verified, ≥25.0% R2E-Gym) reward incorrect solutions.
2. The hackability axis predicts real model behavior: +14.14 pp Pass@1 inflation on flagged tasks, uniform across 134 models.
3. LLM-generated test augmentation has a 61.9% per-augmentation defect rate invisible to LLM judges alone; Docker gold-sanity gating catches it.

**Limitations**:
- Audit sample: 49 tasks across 2/12 SWE-bench repositories; 20 R2E-Gym tasks at $K=1$
- Exploit generator uses a single frontier LLM (28.5% is "at least 28.5%")
- Optimization loop: n=11 tasks; 61.9% defect rate over n=105 checks is the more stable number
- Scaling to n=150–500 is compute-feasible but was disk-bound at submission

**Future directions**:
- Direct causal test: training on fixed vs. untouched broken tasks at fixed compute (closest analog: BugPilot [12], reporting 54.6% pass@1 with 25% smaller dataset)
- Scaling the audit to larger samples
- Cross-attack-surface replication using different exploit classes

---

_Markdown view of https://picx.dev/p/0lid0N, served by PicX — AI-generated visual whiteboard summaries of research papers._