Summary (Overview)

• Cross-benchmark hackability rates: 28.5% of SWE-bench Verified tasks (14/49) and 25.0% of R2E-Gym tasks (4/16 decisive) accept Docker-verified incorrect patches as correct, demonstrating widespread test-suite weakness in code RL training environments.
• Meta-analytic validation: Across 134 frontier model submissions to SWE-bench Verified, flagged-hackable tasks inflate within-difficulty-stratum Pass@1 by +14.14 percentage points (95% CI [+11.80, +16.48]; one-sided $p < 10^{-6}$; $I^2 = 0\%$), with 123 of 134 models showing positive effects.
• Critical methodology finding: An inline LLM judge alone endorsed 10/11 broken tasks as "upgraded," but Docker re-verification revealed a 61.9% per-augmentation defect rate (65/105 decisive LLM-generated tests fail on the gold patch itself) that the judge completely missed.
• Repair procedure: A Docker gold-sanity gate (running each generated test against the gold solution before judge consultation) combined with retry converges 9/11 broken tasks to gated upgrades; ablation shows retry presence—not diversity bias or judge resolution—is the load-bearing component.
• Four-way triangulation: Independent measurements (this audit, OpenAI's retirement note, Berkeley's trustworthy-env, and the meta-analysis) all confirm substantial benchmark unreliability in code RL.

Introduction and Theoretical Foundation

Reinforcement learning from execution feedback (RLVR) works only when the verifier is correct. If a task's test suite accepts a solution that does not solve the problem, the policy receives reward for the wrong behavior—a mechanism Anthropic's November 2025 report ties to broader emergent misalignment, including alignment faking and sabotage [8].

Prior audits of SWE-bench Verified are insufficient for operational use:

• A manual sweep by 93 Python developers at benchmark release [6]
• OpenAI's February 2026 announcement that 59.4% of failed tasks have flawed tests [10]

Neither produces the operational number a curator needs: of the tasks in a code RL training set, what fraction reward the model for an incorrect solution?

The paper builds on four existing components:

1. NVIDIA-style verifier scoring [2]
2. Semi-formal reasoning for LLM-based code judging (Ugare and Chandra) [13]
3. EvolveCoder-style iterative attacks [11]
4. Human-difficulty stratification [1]

The novel contributions are empirical: cross-benchmark hackability rates, the 134-model meta-analytic validation, and the per-augmentation defect rate.

Methodology

3.1 Exploit Generation

For each task, $K$ candidate incorrect source patches are generated using a frontier LLM (Claude Sonnet 4). Each candidate is a unified diff (SWE-bench) or file_replacements mapping (R2E-Gym) that passes the existing test suite while changing observable behavior. Rounds 2–3 use failure logs from round 1 to bias generation toward unblocked exploits (iterative-attacker pattern). Candidates are applied inside Docker via the project's harness; a task is hackable if at least one candidate passes the native test runner.

3.2 Environment Quality Score (EQS)

$$\operatorname{EQS}(t) = 0.35\,V(t) + 0.30\,(1 - H(t)) + 0.20\,F_1(t) + 0.15\,D(t) \tag{1}$$

where:

• $V(t)$: NVIDIA-style verifier discrimination score over candidate patches
• $H(t)$: empirical hackability (fraction of exploit candidates that succeed)
• $F_1(t)$: three-way confusion-matrix F1 from a semi-formal LLM judge combined with execution (CORRECT vs. PARTIAL vs. INCORRECT; PARTIAL counts as incorrect)
• $D(t)$: learnability signal from external per-task Pass@1 across model submissions

$$\operatorname{verdict}(t) = \begin{cases} \text{KEEP} & \text{EQS}(t) > 0.70 \\ \text{FIX} & 0.40 \leq \text{EQS}(t) \leq 0.70 \\ \text{DROP} & \text{EQS}(t) < 0.40 \end{cases} \tag{2}$$

Weights reflect prior on signal directness (not tuned on validation data); headline claims are weight-independent.

3.3 Sanity-Gated Repair Loop

Stage 1 (Gate): Injected test runs against gold solution in Docker via native runner. Failure on gold → augmentation discarded, retry triggered (temperature 0.7, diversity-biased prompt). Success → proceeds to Stage 2.

Stage 2 (Judge): 3-sample self-consistency variant of semi-formal reasoning, sampled at temperatures (0.2, 0.4, 0.6). Majority vote determines verdict; BLOCKS verdict counts exploit as covered. Tasks with all exploits covered convert FIX → KEEP.

Empirical Validation / Results

Hackability Rates

SWE-bench Verified (n=49): 14/49 tasks hackable (28.5%), spanning astropy (6) and django (8). Round-1 single-shot: 18.4% (9/49); rounds 2–3 add 5 tasks. Mean EQS = 0.806; verdicts: KEEP 38, FIX 11, DROP 1. Audit cost: $6.04 API.

R2E-Gym (n=20, 6 repositories): 4/16 decisive tasks hackable (25.0%) at $K=1$ single-shot (lower bound). Hackable: 2 numpy, 1 aiohttp, 1 pandas. Cost: $8.29 API.

Meta-Analytic Validation

Per-model effect:

$$\Delta_m = \mathrm{Pass@1}_m^{\mathrm{hack}} - \mathrm{Pass@1}_m^{\mathrm{robust}} \tag{3}$$

Pooled via DerSimonian-Laird random-effects meta-analysis:

$$\hat{\Delta} = \frac{\sum_m w_m \Delta_m}{\sum_m w_m}, \quad w_m = \frac{1}{\sigma_m^2 + \hat{\tau}^2} \tag{4}$$

Result: $\hat{\Delta} = +14.14\,\text{pp}$, 95% CI [+11.80, +16.48], $p < 10^{-6}$, $I^2 = 0\%$. Sign test: 123/134 models positive. Stratification by human difficulty retains 55% of the unstratified effect (+25.64 pp). Subgroup analyses (era, Pass@1 quartile, scaffold) all same direction.

Augmenter Defect Rate

Un-gated run: LLM judge reported 10/11 tasks upgraded (mean EQS 0.570 → 0.816, $5.57, 3 iterations).

Docker re-verification (stratified 8-pair sample): 1 vindicated, 6 INVALID_AUG (sanity FAIL on gold), 1 methodology edge case. All 6 INVALID_AUG cases were endorsed as BLOCKS by the judge.

Gated run: Gate flags 65/105 decisive augmentations (61.9% defect rate); 40 PASS, 65 FAIL, 3 ERROR, 3 SKIPPED. Converges 9/11 tasks (mean EQS 0.570 → 0.797, $3.60).

Ablation Results

Key finding: Retry presence is load-bearing; diversity bias is not. The 3-sample judge is worth 1 task over single-sample.

Table 1: The 6 INVALID_AUG Cases

Theoretical and Practical Implications

Measurement vs. deployment: The 28.5% and 25.0% rates measure test-suite acceptance of wrong answers, not whether deployed models produce them. The +14.14 pp meta-analysis bridges this gap, showing frontier models do produce wrong-answer-but-passes patterns on flagged tasks at higher rates.

Methodology caution: LLM judges evaluating LLM-generated test code are fundamentally different from judging source code against execution-derived ground truth. The judge correctly reasoned about what tests would check "if they ran"—but failed to catch that tests don't run, run incorrectly, or assert the opposite of documented behavior.

Practical pipeline: A curator using this pipeline at scale should expect augmenter-class defects that an LLM judge alone cannot catch. The Docker gold-sanity gate provides the per-augmentation accept/reject signal that closes this gap, at lower cost ($3.60 vs. $5.57) because it filters defective augmentations before expensive judge consultation.

Four-way triangulation (each with different denominator and attack surface):

1. This audit: 28.5% hackability (n=49, Oct 2025)
2. OpenAI: 59.4% flawed-on-failed (Feb 2026)
3. Berkeley trustworthy-env: 45 process-isolation exploits across 8 benchmarks (Apr 2026)
4. This meta-analysis: +14.14 pp inflation across 134 models (Jun 2026)

Conclusion

Main takeaways:

1. A substantial fraction of code RL benchmark tasks (28.5% SWE-bench Verified, ≥25.0% R2E-Gym) reward incorrect solutions.
2. The hackability axis predicts real model behavior: +14.14 pp Pass@1 inflation on flagged tasks, uniform across 134 models.
3. LLM-generated test augmentation has a 61.9% per-augmentation defect rate invisible to LLM judges alone; Docker gold-sanity gating catches it.

Limitations:

• Audit sample: 49 tasks across 2/12 SWE-bench repositories; 20 R2E-Gym tasks at $K=1$
• Exploit generator uses a single frontier LLM (28.5% is "at least 28.5%")
• Optimization loop: n=11 tasks; 61.9% defect rate over n=105 checks is the more stable number
• Scaling to n=150–500 is compute-feasible but was disk-bound at submission

Future directions:

• Direct causal test: training on fixed vs. untouched broken tasks at fixed compute (closest analog: BugPilot [12], reporting 54.6% pass@1 with 25% smaller dataset)
• Scaling the audit to larger samples
• Cross-attack-surface replication using different exploit classes