# BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs

> BenchChecker detects benchmark contamination in bug-fixing LLMs with AUC around 0.9 using only text outputs, showing reported resolution rates are overestimated by over 20 percent on medium-difficulty tasks.

- **Source:** [arXiv](https://arxiv.org/abs/2610.19447)
- **Published:** 2026-10-24
- **Permalink:** https://picx.dev/p/yZJtLC
- **Whiteboard:** https://picx.dev/p/yZJtLC/image

## Summary

# BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs

**Authors:** Di Wu¹, Xu He¹, Shu Wang², Kun Sun¹  
¹George Mason University, ²Palo Alto Networks, Inc.

---

## Summary (Overview)

- **Problem Addressed:** Data contamination in bug-fixing benchmarks for Large Language Models (LLMs), where test samples appear in training sets, artificially inflating performance metrics. OpenAI stopped reporting SWE-bench Verified scores due to such concerns.
- **Key Contribution:** BenchChecker, the first framework for detecting data contamination in LLM bug-fixing benchmarks that works with **both open-source and closed-source models** using only textual outputs (no logits required).
- **Method:** A forensics-based two-stage detection process: (1) **Repository Presence Test** using hypothesis testing on code-prefix probing, and (2) **Patch Presence Test** using similarity alignment and test-suite verification against the training cutoff snapshot.
- **Performance:** Achieves AUC ≈ 0.9 across six benchmarks with FPR < 0.1, significantly outperforming logit-based baselines (which perform near random guessing with AUC ≈ 0.5).
- **Empirical Finding:** Filtering contaminated samples reduces reported resolution rates of most LLMs by **>20% relative** on medium-difficulty tasks (1–4h subset in SWE-bench Verified, Medium subset in Multi-SWE-bench), showing substantial overfitting in current evaluations.

---

## Introduction and Theoretical Foundation

### Background

Bug-fixing benchmarks (SWE-bench, Multi-SWE-bench, Defects4J, etc.) systematically evaluate LLMs' capability in automated program repair. However, **data contamination** undermines credibility: when benchmark patches appear in training data, models may rely on memorization rather than reasoning.

### Two Root Causes of Contamination

1. **Repository overlap:** Over 85% of source repositories for six common benchmarks appear in The Stack v2 (the largest public code training dataset).
2. **Patch persistence:** More than 50% of test sample patches remain persistent in the corresponding repository snapshots used for training.

### Unique Challenges

- LLMs typically do not disclose training datasets, preventing direct verification.
- Closed-source LLMs expose only textual outputs, not logits—making existing membership-inference (MI) methods inapplicable.

### Theoretical Foundation

The framework rests on the observation that **code LLM training sets include recent repository snapshots** (up to the training cutoff date) to maintain data freshness. Therefore:

> If a patch was introduced early and never removed before the cutoff date, it is highly likely contained in these snapshots → **contamination**.

The problem decomposes into two sub-problems:
1. **Repository presence test:** Is the repository in the training set?
2. **Patch presence test:** Does the patch persist in the repository before the training cutoff?

### Contamination Categories

Four temporal overlap categories between patch lifecycle and snapshot collection interval $[T_{c,start}, T_{c,end}]$:

- **① True Positive:** $T_{p,intro} \leq T_{c,end}$ and $T_{p,persist} \geq T_{latest}$ — patch introduced before collection ends and persists to cutoff.
- **② False Positive (negligible):** Patch introduced between collection end and cutoff; interval is typically minimal.
- **③ False Negative (~10%):** Patch removed before collection ends but after collection starts.
- **④ True Negative:** Patch removed before collection starts.

---

## Methodology

### Stage I: Repository Presence Test

**Task Modeling:** Given repository $R$ as a collection of code snippets $C$, each snippet is split into a prefix query $q$ and suffix target $t$ such that $C = (q, t)$. A **hit** occurs when target $t$ is included in the model's response set $\text{Resp}(M; q)$.

#### Signature Snippets (Three Criteria)

1. **Repetitiveness:** The snippet must appear multiple times in the repository:

$$
\operatorname{count}_{R}(C) > 1 \tag{1}
$$

2. **Distinctiveness:** The target must be rare across a reference corpus:

$$
\sum_{r \in \mathcal{R}} \mathbb{1}[t \in r] \leq \tau \tag{2}
$$

3. **Context-Independence:** The query must not predict the target:

$$
H(t \mid q) \geq \delta \tag{3}
$$

#### Hypothesis Testing

Formulated as a binomial hypothesis test with:
- **Null hypothesis $H_0$:** Hit probability $P \leq P_0$
- **Alternative $H_1$:** Hit probability $P > P_0$

where $P_0$ is the default hit probability when $R \notin D$:

$$
P_0 := P(t \in \operatorname{Resp}(M; q) \mid R \notin D) \tag{4}
$$

The p-value is computed as:

$$
p := \sum_{i=k}^{n} \binom{n}{i} \cdot P_0^{i} \cdot (1 - P_0)^{n-i} \tag{5}
$$

A weighted hit-counting mechanism accounts for distinctiveness and context-independence (ratio of prediction entropy to target-word repository frequency).

#### Pattern Identification

Three pattern types extracted:
- **Code block clones** (via NiCad 6.2)
- **Text duplications** (via Simhash algorithm)
- **Word co-occurrences** (via PrefixSpan frequent-pattern mining)

### Stage II: Patch Presence Test

Four steps:
1. **Patch Preprocessing:** Formatting, comment removal (Black for Python, JavaParser for Java).
2. **Patch Localization:** AST-based alignment using GumTree tree-edit distance within candidate functions.
3. **Similarity Matching:** Deep-learning code embeddings (jina-code-embeddings-0.5B) with threshold 0.7.
4. **Testing Verification:** Fallback—apply snapshot blocks as a modified diff and run the benchmark's test suite.

---

## Empirical Validation / Results

### Experimental Setup

- **Models:** StarCoder-15B, StarCoder2-15B, CodeGen2-16B (all trained on The Stack v2, cutoff September 2023)
- **Benchmarks:** SWE-bench Verified, BugsInPy, BugSwarm (Python); Multi-SWE-bench, Defects4J, GrowingBugs (Java)
- **Baselines:** LOSS, Zlib, Min-K%, Reference-based (Ref)—all logit-dependent

### Dataset Contamination Analysis

| Benchmark | SWE-bench Verified | BugsInPy | BugSwarm | Multi-SWE-bench | Defects4J | GrowingBugs |
|---|---|---|---|---|---|---|
| Repositories | 12 | 17 | 34 | 9 | 17 | 171 |
| Samples | 500 | 500 | 170 | 128 | 835 | 1839 |
| Repos in Stack v2 | 100.0% | 100.0% | 85.2% | 100.0% | 100.0% | 97.8% |
| Patches in Stack v2 | 78.9% (3.2%) | 67.0% (3.3%) | 77.0% (3.5%) | 62.4% (10.8%) | 71.8% (10.2%) | 78.0% (11.6%) |

*Parentheses show Category ③ samples (patches absent at cutoff but present in Stack v2).*

### Main Results (Contaminated Sample Detection)

BenchChecker achieves **AUC ≈ 0.9** across all benchmarks with **FPR ≤ 0.1**, while baselines perform near random (AUC ≈ 0.5):

| Benchmark | SWE-bench Verified | BugsInPy | BugSwarm | Multi-SWE-bench | Defects4J | GrowingBugs |
|---|---|---|---|---|---|---|
| **Ours AUC** | 0.945 | 0.909–0.919 | 0.865–0.934 | 0.919 | 0.850–0.863 | 0.838–0.863 |
| **Ours TPR** | 0.979 | 0.864–0.893 | 0.816–0.954 | 0.938 | 0.746–0.777 | 0.753–0.795 |
| **Ours FPR** | 0.090 | 0.055 | 0.065–0.087 | 0.100 | 0.046–0.051 | 0.069–0.078 |
| Best Baseline AUC | 0.535 | 0.539 | 0.654 | 0.602 | 0.610 | 0.519 |

### Ablation Study (SWE-bench Verified, StarCoder-15B)

| Stage I Variant | AUC | TPR | FPR |
|---|---|---|---|
| BenchChecker | 1.000 | 1.000 | 0.000 |
| w/o Repetitiveness | 0.663 | 0.428 | 0.064 |
| w/o Distinctiveness | 0.647 | 0.871 | 0.486 |
| w/o Context-Independence | 0.845 | 0.952 | 0.323 |

| Stage II Variant | AUC | TPR | FPR |
|---|---|---|---|
| BenchChecker | 0.938 | 0.979 | 0.103 |
| w/o Patch Preprocessing | 0.786 | 0.762 | 0.135 |
| w/o Patch Localization | 0.874 | 0.932 | 0.287 |
| w/o Similarity Matching | 0.688 | 0.652 | 0.050 |
| w/o Testing Verification | 0.868 | 0.891 | 0.092 |

### Empirical Study on Leaderboards

**Finding 1:** Contamination impact increases with bug difficulty. Relative drops on 1–4h subset exceed 20% vs. <10% on easier subsets.

**Finding 2:** Multi-SWE-bench shows more severe contamination impact (up to 55.9% drop) than SWE-bench Verified (up to 11.8%).

**Finding 3:** Contamination impact is consistent across agent frameworks using the same model (max difference 3.3%).

**Finding 4:** Contamination-induced degradation declines from **40% (2023) to 5% (2025)**, indicating newer LLMs generalize substantially better.

**Finding 5:** Models with higher reported performance show smaller drops after contamination removal—benchmarks still reflect genuine performance trends.

### Bug-Fixing Pattern Analysis

- **Finding 6:** Bug-localization failure is the most common reason models fail on seen patches (16/30 cases).
- **Finding 7:** Bug-triggering cases and solution hints in issue descriptions make bugs easier to fix (35/61 contain triggering scenarios; 21/61 provide solution hints).
- **Finding 8:** Models struggle with cross-file bugs—all 25/25 unseen-patch failures involve multi-file dependencies.

---

## Theoretical and Practical Implications

### Theoretical Significance

- **First black-box contamination detection framework** for bug-fixing benchmarks—works without logit access, enabling evaluation of closed-source models.
- **Forensics-based approach** combining multiple external information sources (benchmark metadata, repository history, cutoff dates) rather than relying solely on model outputs.
- Demonstrates that **repository-level memorization** is a robust contamination indicator, validated by extension to RepoBench (code completion shows up to 60% exact-match increase for seen repositories).

### Practical Implications

- **For benchmark maintainers:** BenchChecker provides an automated tool to filter contaminated samples and strengthen leaderboard credibility.
- **For LLM vendors:** Reveals true model capabilities; newer models show declining contamination dependence, suggesting genuine progress.
- **For end users:** Transparent reporting prevents over-reliance on inflated benchmark scores in production settings.
- **For future benchmark design:** Suggests using post-cutoff repositories or dynamic benchmarks to reduce contamination risk.

---

## Conclusion

BenchChecker is a forensics-based two-stage framework that detects data contamination in bug-fixing benchmarks using only textual outputs, making it applicable to both open-source and closed-source LLMs. Key takeaways:

1. **Method:** Repository presence test (binomial hypothesis testing with signature snippets) + Patch presence test (similarity matching with test-suite verification).
2. **Performance:** AUC ≈ 0.9 with FPR < 0.1 across six benchmarks, vastly outperforming logit-based baselines (AUC ≈ 0.5).
3. **Findings:** Current leaderboard resolution rates are significantly overestimated (20%+ relative drop on medium-difficulty tasks), though contamination impact is decreasing over time (40% → 5% from 2023 to 2025).
4. **Limitations:** Lexical-level distinctiveness assessment; assumes patch persistence at cutoff snapshot; does not cover non-code artifacts (issue descriptions).
5. **Future work:** Extending to other modalities (natural language, artwork), addressing contamination through non-code artifacts, and applying to training strategies that include full commit histories.

> **Ethical note:** The authors commit to contacting benchmark maintainers and providing transparent, open-source tooling to mitigate potential negative impacts on stakeholder trust.

---

_Markdown view of https://picx.dev/p/yZJtLC, served by PicX — AI-generated visual whiteboard summaries of research papers._