BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs

Authors: Di Wu¹, Xu He¹, Shu Wang², Kun Sun¹
¹George Mason University, ²Palo Alto Networks, Inc.

---

Summary (Overview)

• Problem Addressed: Data contamination in bug-fixing benchmarks for Large Language Models (LLMs), where test samples appear in training sets, artificially inflating performance metrics. OpenAI stopped reporting SWE-bench Verified scores due to such concerns.
• Key Contribution: BenchChecker, the first framework for detecting data contamination in LLM bug-fixing benchmarks that works with both open-source and closed-source models using only textual outputs (no logits required).
• Method: A forensics-based two-stage detection process: (1) Repository Presence Test using hypothesis testing on code-prefix probing, and (2) Patch Presence Test using similarity alignment and test-suite verification against the training cutoff snapshot.
• Performance: Achieves AUC ≈ 0.9 across six benchmarks with FPR < 0.1, significantly outperforming logit-based baselines (which perform near random guessing with AUC ≈ 0.5).
• Empirical Finding: Filtering contaminated samples reduces reported resolution rates of most LLMs by >20% relative on medium-difficulty tasks (1–4h subset in SWE-bench Verified, Medium subset in Multi-SWE-bench), showing substantial overfitting in current evaluations.

---

Introduction and Theoretical Foundation

Background

Bug-fixing benchmarks (SWE-bench, Multi-SWE-bench, Defects4J, etc.) systematically evaluate LLMs' capability in automated program repair. However, data contamination undermines credibility: when benchmark patches appear in training data, models may rely on memorization rather than reasoning.

Two Root Causes of Contamination

1. Repository overlap: Over 85% of source repositories for six common benchmarks appear in The Stack v2 (the largest public code training dataset).
2. Patch persistence: More than 50% of test sample patches remain persistent in the corresponding repository snapshots used for training.

Unique Challenges

• LLMs typically do not disclose training datasets, preventing direct verification.
• Closed-source LLMs expose only textual outputs, not logits—making existing membership-inference (MI) methods inapplicable.

Theoretical Foundation

The framework rests on the observation that code LLM training sets include recent repository snapshots (up to the training cutoff date) to maintain data freshness. Therefore:

If a patch was introduced early and never removed before the cutoff date, it is highly likely contained in these snapshots → contamination.

The problem decomposes into two sub-problems:

1. Repository presence test: Is the repository in the training set?
2. Patch presence test: Does the patch persist in the repository before the training cutoff?

Contamination Categories

Four temporal overlap categories between patch lifecycle and snapshot collection interval $[T_{c,start}, T_{c,end}]$:

• ① True Positive: $T_{p,intro} \leq T_{c,end}$ and $T_{p,persist} \geq T_{latest}$ — patch introduced before collection ends and persists to cutoff.
• ② False Positive (negligible): Patch introduced between collection end and cutoff; interval is typically minimal.
• ③ False Negative (~10%): Patch removed before collection ends but after collection starts.
• ④ True Negative: Patch removed before collection starts.

---

Methodology

Stage I: Repository Presence Test

Task Modeling: Given repository $R$ as a collection of code snippets $C$, each snippet is split into a prefix query $q$ and suffix target $t$ such that $C = (q, t)$. A hit occurs when target $t$ is included in the model's response set $\text{Resp}(M; q)$.

Signature Snippets (Three Criteria)

1. Repetitiveness: The snippet must appear multiple times in the repository:

$$\operatorname{count}_{R}(C) > 1 \tag{1}$$

2. Distinctiveness: The target must be rare across a reference corpus:

$$\sum_{r \in \mathcal{R}} \mathbb{1}[t \in r] \leq \tau \tag{2}$$

3. Context-Independence: The query must not predict the target:

$$H(t \mid q) \geq \delta \tag{3}$$

Hypothesis Testing

Formulated as a binomial hypothesis test with:

• Null hypothesis $H_0$: Hit probability $P \leq P_0$
• Alternative $H_1$: Hit probability $P > P_0$

where $P_0$ is the default hit probability when $R \notin D$:

$$P_0 := P(t \in \operatorname{Resp}(M; q) \mid R \notin D) \tag{4}$$

The p-value is computed as:

$$p := \sum_{i=k}^{n} \binom{n}{i} \cdot P_0^{i} \cdot (1 - P_0)^{n-i} \tag{5}$$

A weighted hit-counting mechanism accounts for distinctiveness and context-independence (ratio of prediction entropy to target-word repository frequency).

Pattern Identification

Three pattern types extracted:

• Code block clones (via NiCad 6.2)
• Text duplications (via Simhash algorithm)
• Word co-occurrences (via PrefixSpan frequent-pattern mining)

Stage II: Patch Presence Test

Four steps:

1. Patch Preprocessing: Formatting, comment removal (Black for Python, JavaParser for Java).
2. Patch Localization: AST-based alignment using GumTree tree-edit distance within candidate functions.
3. Similarity Matching: Deep-learning code embeddings (jina-code-embeddings-0.5B) with threshold 0.7.
4. Testing Verification: Fallback—apply snapshot blocks as a modified diff and run the benchmark's test suite.

---

Empirical Validation / Results

Experimental Setup

• Models: StarCoder-15B, StarCoder2-15B, CodeGen2-16B (all trained on The Stack v2, cutoff September 2023)
• Benchmarks: SWE-bench Verified, BugsInPy, BugSwarm (Python); Multi-SWE-bench, Defects4J, GrowingBugs (Java)
• Baselines: LOSS, Zlib, Min-K%, Reference-based (Ref)—all logit-dependent

Dataset Contamination Analysis

Parentheses show Category ③ samples (patches absent at cutoff but present in Stack v2).

Main Results (Contaminated Sample Detection)

BenchChecker achieves AUC ≈ 0.9 across all benchmarks with FPR ≤ 0.1, while baselines perform near random (AUC ≈ 0.5):

Ablation Study (SWE-bench Verified, StarCoder-15B)

Empirical Study on Leaderboards

Finding 1: Contamination impact increases with bug difficulty. Relative drops on 1–4h subset exceed 20% vs. <10% on easier subsets.

Finding 2: Multi-SWE-bench shows more severe contamination impact (up to 55.9% drop) than SWE-bench Verified (up to 11.8%).

Finding 3: Contamination impact is consistent across agent frameworks using the same model (max difference 3.3%).

Finding 4: Contamination-induced degradation declines from 40% (2023) to 5% (2025), indicating newer LLMs generalize substantially better.

Finding 5: Models with higher reported performance show smaller drops after contamination removal—benchmarks still reflect genuine performance trends.

Bug-Fixing Pattern Analysis

• Finding 6: Bug-localization failure is the most common reason models fail on seen patches (16/30 cases).
• Finding 7: Bug-triggering cases and solution hints in issue descriptions make bugs easier to fix (35/61 contain triggering scenarios; 21/61 provide solution hints).
• Finding 8: Models struggle with cross-file bugs—all 25/25 unseen-patch failures involve multi-file dependencies.

---

Theoretical and Practical Implications

Theoretical Significance

• First black-box contamination detection framework for bug-fixing benchmarks—works without logit access, enabling evaluation of closed-source models.
• Forensics-based approach combining multiple external information sources (benchmark metadata, repository history, cutoff dates) rather than relying solely on model outputs.
• Demonstrates that repository-level memorization is a robust contamination indicator, validated by extension to RepoBench (code completion shows up to 60% exact-match increase for seen repositories).

Practical Implications

• For benchmark maintainers: BenchChecker provides an automated tool to filter contaminated samples and strengthen leaderboard credibility.
• For LLM vendors: Reveals true model capabilities; newer models show declining contamination dependence, suggesting genuine progress.
• For end users: Transparent reporting prevents over-reliance on inflated benchmark scores in production settings.
• For future benchmark design: Suggests using post-cutoff repositories or dynamic benchmarks to reduce contamination risk.

---

Conclusion

BenchChecker is a forensics-based two-stage framework that detects data contamination in bug-fixing benchmarks using only textual outputs, making it applicable to both open-source and closed-source LLMs. Key takeaways:

1. Method: Repository presence test (binomial hypothesis testing with signature snippets) + Patch presence test (similarity matching with test-suite verification).
2. Performance: AUC ≈ 0.9 with FPR < 0.1 across six benchmarks, vastly outperforming logit-based baselines (AUC ≈ 0.5).
3. Findings: Current leaderboard resolution rates are significantly overestimated (20%+ relative drop on medium-difficulty tasks), though contamination impact is decreasing over time (40% → 5% from 2023 to 2025).
4. Limitations: Lexical-level distinctiveness assessment; assumes patch persistence at cutoff snapshot; does not cover non-code artifacts (issue descriptions).
5. Future work: Extending to other modalities (natural language, artwork), addressing contamination through non-code artifacts, and applying to training strategies that include full commit histories.

Ethical note: The authors commit to contacting benchmark maintainers and providing transparent, open-source tooling to mitigate potential negative impacts on stakeholder trust.