Summary (Overview)

• This paper investigates whether judge-based (LLM-as-a-judge) evaluation scores reveal substantial changes in generated code quality. The authors replay a prior GRPO training evaluation on 421 coding problems (HumanEval + MBPP) using Qwen-2.5-7B-Instruct and two trained variants (CE-trained with execution rewards, SLJ-trained with judge rewards).
• The base model's syntax-failure rate remains stable (~0.4%), but both trained checkpoints produce invalid Python on 63–67% of attempts in the replay, while the strong judge's Pass@32 scores drop by only ~3 percentage points.
• With 128 attempts per question, the judge accepts at least one SLJ-trained answer on 96.44% of problems versus only 28.27% under execution tests—demonstrating that a high judge score can hide severe output damage.
• The dominant failure mode is a closing bracket with no matching opening bracket (98.6% of invalid samples), ruling out early-cutoff truncation as the main cause.
• The paper recommends reporting the share of answers that pass Python's syntax check alongside every judge-based score, as a cheap, test-free sensitivity check.

Introduction and Theoretical Foundation

Background: Reinforcement learning with verifiable rewards (RLVR) trains language models using automated feedback. For coding tasks, execution-based tests provide direct correctness evidence but require sandboxes and time. Many pipelines instead use LLM judges to read and grade code, which introduces weaknesses that high scores may conceal.

Motivation:

• Code-judge benchmarks already examine syntax errors, and judges can favor familiar-looking answers or be exploited by optimized models.
• Reported performance varies with evaluation code, decoding settings, random seeds, hardware, and numerical precision.
• Prior work shows measurement gaps can inflate apparent RLVR gains.

Key Research Question: Can a judge-based evaluation reveal whether generated answers have substantially deteriorated? The study examines a complementary risk: the reported score may remain high even when generated answers degrade.

Theoretical Foundation: Pass@k estimates the chance of at least one success in k attempts per question using the unbiased estimator of Chen et al. [2021]:

$$\text{Pass@}k = \mathbb{E}\left[1 - \frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$$

where $n$ is total samples, $c$ is correct samples, and $k$ is attempts allowed. Crucially, one approved answer among 32 attempts suffices for success—so high scores don't imply most individual answers pass.

Methodology

Models Tested:

• Base: Qwen-2.5-7B-Instruct (before additional GRPO training)
• CE-trained: GRPO rewarded by execution tests
• SLJ-trained: GRPO rewarded by a strong LLM judge

Dataset: 421 coding questions: 164 from HumanEval and 257 from sanitized MBPP test split.

Generation Runs:

• First run: 32 answers per question saved during the original study (13,472 per checkpoint)
• Second run: 128 answers per question generated from released checkpoints (53,888 per checkpoint) on different hardware

Evaluation Methods:

1. Syntax check: Code must pass Python's ast.parse — no tests, no execution required
2. Judge evaluation: Qwen-2.5-72B-Instruct (strong) and Qwen-2.5-7B-Instruct (weaker), both at temperature 0 via vLLM, issuing one correct/incorrect verdict per answer
3. Execution: Runs the full prompt and check function; a pass requires a success token after testing

Statistical Analysis: Paired bootstrap with 5000 resamples of the 421 questions for 95% confidence intervals on within-run score differences.

Empirical Validation / Results

Table 1: Invalid answer rates across runs

Key findings:

• Base model changes little (0.39% → 0.40%); trained checkpoints jump from under 1% to over 60%.
• For SLJ-trained: more than half of 128 answers are invalid on 283 of 421 questions; only 29 questions have all answers passing syntax.
• 35,562 of 36,051 SLJ-trained invalid samples (98.6%) involve a closing bracket with no matching opening bracket or mismatched type (e.g., for i in range(1, len(numbers))):).

Table 2: Four checks on the same second-run answers

Judge behavior: The judge accepts 60.56% of SLJ-trained answers that fail syntax checks, compared with 83.35% of those that pass. Of all approvals for this checkpoint, 59.49% are invalid code.

Score gap analysis (base vs. SLJ-trained on second-run outputs):

• Judge Pass@1 gap: 16.01 pp → shrinks to 0.71 pp at Pass@128
• Execution gap: 22.32 pp → 15.91 pp at Pass@128
• Syntax gate gap: 66.50 pp

Syntax gate effect on Pass@1:

• Base: −0.19 pp drop
• CE-trained: −37.40 pp drop
• SLJ-trained: −40.51 pp drop

Controlled check: Adding surplus brackets to 10–60% of base-model answers changes judge Pass@32 by at most 0.24 pp, while parse rate falls to between 89.3% and 39.9%.

Theoretical and Practical Implications

For Evaluation Methodology:

• A high judge-based Pass@k score can coexist with extensive output damage; one approved answer per question hides that most answers are invalid.
• Pass@k measures "finding at least one approved answer," not "consistently generating valid code"—these are fundamentally different properties.
• The syntax check is a cheap (no execution needed), effective sensitivity check that reveals changes the headline score hides.

For AI Training Pipelines:

• Self-rewarding training loops that select candidates via judges risk retaining invalid code if approval is too lenient.
• The results do not establish reward gaming or GRPO as responsible; the cause remains unknown, but the failure demonstrates the importance of sensitivity checking.

For Reproducibility:

• Each run should retain generated answers, judge inputs/verdicts, generation settings, and model file checksums.
• Missing records prevent identifying the cause of the discrepancy—a lesson for evaluation infrastructure.

Limitations:

• MBPP prompts omit the function name expected by tests; supplying names raises base-model MBPP Pass@128 from 12.45% to 79.38%, so absolute execution scores must be read with this caveat.

Conclusion

The study demonstrates that a high judge-based score can hide substantial damage to generated code. Reporting the share of answers that pass Python's syntax check alongside every judge-based score would make such incidents visible—the check requires no tests or execution and flags invalid answers the judge accepts, though it cannot establish functional correctness. The findings motivate checking an evaluation's sensitivity to change in the outputs it scores. Future work should investigate the cause of the damaged checkpoints and develop general monitoring thresholds for output validity.

Recommendation: Report the parse rate (share of answers passing syntax check) alongside every judge-based score and compare it across runs—a practice that distinguishes "finding an approved answer" from "consistently generating valid code."