# PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix

> PROBE benchmark shows frontier coding agents find under 21% of maintainer-fixed bugs, discovering real but systematically different bugs focused on local errors rather than input handling.

- **Source:** [arXiv](https://arxiv.org/abs/2610.35987)
- **Published:** 2026-10-31
- **Permalink:** https://picx.dev/p/iOzKs5
- **Whiteboard:** https://picx.dev/p/iOzKs5/image

## Summary

# Summary of "PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix"

## Summary (Overview)

- **PROBE** is a novel benchmark for evaluating **whole-repository bug discovery** by coding agents, consisting of **2,457 real bugs** fixed by maintainers across **20 diverse Python, JavaScript, and TypeScript repositories** over an 18-month period.
- The benchmark enables **recall and precision measurement** against a reference set of maintainer-fixed bugs, using a validated LLM-based matching pipeline (99% precision, 98% recall on expert-labeled pairs).
- Across **10 frontier models**, all achieve **recall below 21%** and **precision below 9%** with respect to the reference set, revealing a significant gap between agent and human bug-finding capabilities.
- Despite low precision against the reference set, **85% of agent findings are validated as real bugs** by independent LLM judges, indicating agents find *different* bugs than maintainers fix, not merely invalid ones.
- The paper demonstrates that **agent-found bugs cluster around local correctness errors** (e.g., wrong arithmetic, unhandled degenerate values) while **maintainer-fixed bugs focus on incomplete input handling and public API errors**, suggesting systematic differences in bug discovery priorities.

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical gap in software engineering: while coding agents have demonstrated reliability in **fixing human-reported bugs** (Jimenez et al., 2024; Deng et al., 2026), their ability to **autonomously discover meaningful bugs** across entire repositories remains understudied. This capability is receiving increasing commercial attention (OpenAI, 2026; Anthropic, 2026c; Cursor, 2026) as a way to ease the burden of code review and application testing.

### Prior Work Limitations

Previous research on bug detection has been **incomplete** in several ways:
- **Targeted approaches**: Pointing agents to specific patches or files known to contain flaws (Zeng et al., 2026; Li et al., 2026)
- **Security-focused**: Concentrating on security vulnerabilities, often those reliably exploitable (Wen et al., 2026; Guo et al., 2025; Zhang et al., 2025b)
- **No whole-repository evaluation**: No prior benchmark measures whether agents can find bugs that maintainers would fix across an entire repository

### Key Theoretical Insight

The authors observe that **maintainers reveal which bugs matter to them by fixing them over time**. This insight forms the theoretical foundation of PROBE: by snapshotting repositories at a fixed time $T_0$ and collecting bugs fixed in a subsequent reference window $[T_0, T_1]$, one can construct a **partial reference set** of meaningful bugs.

The benchmark acknowledges that the reference set is **incomplete** (bugs still unfixed at $T_1$ are missing), and models this incompleteness using survival analysis to estimate the total bug pool.

## Methodology

### Benchmark Construction Pipeline

The benchmark is built through a **six-stage pipeline** applied to 20 curated repositories:

1. **Identify Bug-Fix Commits**: Filter 13,254 commits using deterministic rules and LLM classification to identify 4,296 bug-fix commits (67.6% reduction)
2. **Extract Bugs**: Use LLM to extract 4,763 distinct candidate bugs, splitting those without shared root causes
3. **Anchor Code Changes**: Map affected code ranges from fix time $T'$ back to snapshot time $T_0$, tracing file renames and line shifts
4. **Check Bug Presence**: Verify bugs existed at $T_0$ using coding agents with revision history access (29.8% filtered)
5. **Synthesize Bug Reports**: Generate detailed bug reports without referencing the fix
6. **Deduplicate Candidates**: Merge candidates describing the same defect using LLM-based matching

### Key Statistics

| Stage | Count | Δ |
|-------|-------|---|
| Commits | 13,254 | - |
| Bug-Fix Commits | 4,296 | -67.6% |
| Candidate Bugs | 4,763 | +10.9% |
| Anchored Candidates | 3,345 | -29.8% |
| Present Candidates | 2,501 | -25.2% |
| Deduplicated Bugs | **2,457** | -1.8% |

### Bug Matching Method

The evaluation uses a **two-step matching approach**:

1. **Rule-based Filter**: Computes six similarity metrics (line overlap, embedding cosine similarities, IDF-weighted Jaccard similarities) with thresholds tuned to avoid false negatives
2. **LLM-based Judge**: Gemini 3.5 Flash determines whether resolving the root cause of either bug would resolve the other

Validation results: **98.8% precision, 97.6% recall** across 5 runs on 17,177 pairs.

### Bug Finding Harness

To boost recall from <1% (vanilla agents) to ~10-20%, the authors propose a **multi-pass audit harness**:

- **File Partitioning**: Files partitioned by complexity (cross-file callers/callees, size, function count) into groups ≤60k tokens, yielding **740 passes** across 4,468 files
- **Context Documents**: A project context document (architecture, API, supported inputs) and function-level call graph
- **Execution Isolation**: Each pass runs in a separate Docker container with historically faithful dependencies, 3-hour timeout, no network access

## Empirical Validation / Results

### Main Findings

**Table 3: Recall, Precision, and F1 Scores (sorted by F1)**

| Model | Effort | Recall (%) | Precision (%) | F1 (%) | Findings | Cost |
|-------|--------|------------|---------------|--------|----------|------|
| DeepSeek V4.1 Flash | high | 15.2 | 8.1 | 10.5 | 5,019 | $73 |
| GPT-5.6 Sol | high | 20.8 | 6.7 | 10.2 | 8,285 | $1,185 |
| GPT-5.6 Luna | xhigh | 17.3 | 7.1 | 10.1 | 6,636 | $131 |
| GLM 5.3 Flash | high | 15.8 | 7.0 | 9.7 | 6,029 | $120 |
| GPT-5.6 Sol | medium | 16.9 | 6.7 | 9.6 | 6,675 | $678 |
| GPT-5.4 | medium | 11.0 | 6.9 | 8.5 | 4,204 | $655 |
| GPT-5.6 Terra | medium | 9.1 | 7.6 | 8.3 | 3,093 | $224 |
| Gemini 3.5 Flash | medium | 9.8 | 6.4 | 7.8 | 3,867 | $493 |
| Muse Spark 1.3 | high | 15.9 | 4.8 | 7.4 | 8,372 | $164 |
| Claude Sonnet 5 | medium | 7.9 | 5.8 | 6.7 | 3,472 | $421 |

### Key Observations

1. **Recall vs. Precision Trade-off**: Recall varies significantly across models (correlated with total findings, Pearson r = 0.94), while precision remains consistently low (4.8–8.1%)

2. **Reasoning Effort Effects**: Increasing reasoning effort raises recall (5.3% → 17.3% for GPT-5.6 Luna) without changing precision

3. **Bug Validity**: LLM judge panel confirms **96% of matched findings** and **84% of unmatched findings** as real bugs

4. **Bug Type Distribution**: Agents over-report local correctness errors while maintainers fix more incomplete input handling:

| Category | Human % | Agent % | Gap (pp) |
|----------|---------|---------|----------|
| Wrong arithmetic or comparison | 9.9 | 18.5 | +8.6 |
| Unhandled degenerate value | 9.1 | 13.9 | +4.8 |
| Missing branch for supported input | 18.0 | 7.7 | -10.3 |
| Missing dispatch/lookup table entry | 5.1 | 0.7 | -4.3 |

5. **Agent Agreement**: Within-category overlap between agent runs has **median ratio 6.3×** compared to overlap with human fixes, showing agents agree with each other more than with maintainers

### Harness Ablation Results

| Method | Recall (%) | Precision (%) | F1 (%) | Cost |
|--------|------------|---------------|--------|------|
| Vanilla (single session) | 0.6-0.9 | 17.4-32.3 | 1.2-1.7 | $1-94 |
| Passes (minimal prompt) | 5.3-5.8 | 8.0-8.3 | 6.4-6.8 | $18-487 |
| **Full harness** | **7.6-7.9** | **5.8-7.0** | **6.7-7.3** | **$15-421** |
| Hound | 3.2 | 5.0 | 3.9 | $129 |
| agentic-pbt | 0.9 | 8.9-15.5 | 1.6-1.7 | $4-9 |

## Theoretical and Practical Implications

### Precision Ceiling Estimation

The authors model the total bug pool using **exponential, Weibull, and lognormal distributions**, finding the lognormal fits best. They estimate:
- **5.2k–5.9k bugs** fixed within 5 years
- **7.2k–9.4k bugs** fixed within 10 years

This implies a **precision ceiling of 26–47%** even for a perfect bug finder, since the reference set is incomplete.

### Systematic Differences in Bug Discovery

The paper identifies **fundamental differences** between agent and human bug discovery:

1. **Trigger Requirements**: Humans find more bugs requiring specific environments (10.8% vs. 4.7%) or visual inspection (3.8% vs. 0.7%), while agents find bugs needing only specific inputs (83.9% vs. 74.3%)

2. **Bug Types**: Agents over-report local faults (wrong arithmetic, unhandled degenerate values) while humans focus on incomplete input handling and public API errors

3. **Contamination Analysis**: No evidence of training data contamination for models with published cut-offs (Holm-adjusted p-values ≥ 0.20)

### Practical Implications

- **Model Selection**: Cheaper models (DeepSeek V4.1 Flash at $73, GPT-5.6 Luna xhigh at $131) can outperform expensive ones (GPT-5.6 Sol at $1,185) in F1 score
- **Ensembling**: Combining multiple runs can improve performance—union of Muse Spark runs achieves 26.7% recall; intersection of Luna runs achieves 10.5% precision
- **Bug Finding vs. Fixing**: The gap between agent and human bug priorities suggests different evaluation criteria are needed for proactive bug discovery versus bug fixing

## Conclusion

### Main Takeaways

1. **PROBE** provides the first benchmark for whole-repository bug discovery, enabling quantitative evaluation of coding agents' proactive bug-finding capabilities
2. **Current frontier models achieve at most 20.8% recall and 8.1% precision** against maintainer-fixed bugs, with stronger models primarily producing more findings rather than better ones
3. **Agent-found bugs are largely real but systematically different** from maintainer-fixed bugs, with agents over-reporting local correctness errors and under-reporting incomplete input handling and API surface issues

### Future Directions

- **Methodological improvements**: Methods that exercise software as users do (e.g., testing, runtime analysis) could find bugs closer to those maintainers fix
- **Extended scope**: Applying the pipeline to additional languages and larger codebases
- **Bug-finding capability improvement**: Finding bugs that matter to humans is identified as "a key frontier for scaling automated software engineering"

### Key Quote

> "While agents find real bugs, they are mostly not the ones maintainers fix. Our benchmark PROBE makes this gap measurable."

---

_Markdown view of https://picx.dev/p/iOzKs5, served by PicX — AI-generated visual whiteboard summaries of research papers._