Summary of "PROBE: Frontier Coding Agents Find Different Bugs Than Maintainers Fix"

Summary (Overview)

• PROBE is a novel benchmark for evaluating whole-repository bug discovery by coding agents, consisting of 2,457 real bugs fixed by maintainers across 20 diverse Python, JavaScript, and TypeScript repositories over an 18-month period.
• The benchmark enables recall and precision measurement against a reference set of maintainer-fixed bugs, using a validated LLM-based matching pipeline (99% precision, 98% recall on expert-labeled pairs).
• Across 10 frontier models, all achieve recall below 21% and precision below 9% with respect to the reference set, revealing a significant gap between agent and human bug-finding capabilities.
• Despite low precision against the reference set, 85% of agent findings are validated as real bugs by independent LLM judges, indicating agents find different bugs than maintainers fix, not merely invalid ones.
• The paper demonstrates that agent-found bugs cluster around local correctness errors (e.g., wrong arithmetic, unhandled degenerate values) while maintainer-fixed bugs focus on incomplete input handling and public API errors, suggesting systematic differences in bug discovery priorities.

Introduction and Theoretical Foundation

Background and Motivation

The paper addresses a critical gap in software engineering: while coding agents have demonstrated reliability in fixing human-reported bugs (Jimenez et al., 2024; Deng et al., 2026), their ability to autonomously discover meaningful bugs across entire repositories remains understudied. This capability is receiving increasing commercial attention (OpenAI, 2026; Anthropic, 2026c; Cursor, 2026) as a way to ease the burden of code review and application testing.

Prior Work Limitations

Previous research on bug detection has been incomplete in several ways:

• Targeted approaches: Pointing agents to specific patches or files known to contain flaws (Zeng et al., 2026; Li et al., 2026)
• Security-focused: Concentrating on security vulnerabilities, often those reliably exploitable (Wen et al., 2026; Guo et al., 2025; Zhang et al., 2025b)
• No whole-repository evaluation: No prior benchmark measures whether agents can find bugs that maintainers would fix across an entire repository

Key Theoretical Insight

The authors observe that maintainers reveal which bugs matter to them by fixing them over time. This insight forms the theoretical foundation of PROBE: by snapshotting repositories at a fixed time $T_0$ and collecting bugs fixed in a subsequent reference window $[T_0, T_1]$, one can construct a partial reference set of meaningful bugs.

The benchmark acknowledges that the reference set is incomplete (bugs still unfixed at $T_1$ are missing), and models this incompleteness using survival analysis to estimate the total bug pool.

Methodology

Benchmark Construction Pipeline

The benchmark is built through a six-stage pipeline applied to 20 curated repositories:

1. Identify Bug-Fix Commits: Filter 13,254 commits using deterministic rules and LLM classification to identify 4,296 bug-fix commits (67.6% reduction)
2. Extract Bugs: Use LLM to extract 4,763 distinct candidate bugs, splitting those without shared root causes
3. Anchor Code Changes: Map affected code ranges from fix time $T'$ back to snapshot time $T_0$, tracing file renames and line shifts
4. Check Bug Presence: Verify bugs existed at $T_0$ using coding agents with revision history access (29.8% filtered)
5. Synthesize Bug Reports: Generate detailed bug reports without referencing the fix
6. Deduplicate Candidates: Merge candidates describing the same defect using LLM-based matching

Key Statistics

Bug Matching Method

The evaluation uses a two-step matching approach:

1. Rule-based Filter: Computes six similarity metrics (line overlap, embedding cosine similarities, IDF-weighted Jaccard similarities) with thresholds tuned to avoid false negatives
2. LLM-based Judge: Gemini 3.5 Flash determines whether resolving the root cause of either bug would resolve the other

Validation results: 98.8% precision, 97.6% recall across 5 runs on 17,177 pairs.

Bug Finding Harness

To boost recall from <1% (vanilla agents) to ~10-20%, the authors propose a multi-pass audit harness:

• File Partitioning: Files partitioned by complexity (cross-file callers/callees, size, function count) into groups ≤60k tokens, yielding 740 passes across 4,468 files
• Context Documents: A project context document (architecture, API, supported inputs) and function-level call graph
• Execution Isolation: Each pass runs in a separate Docker container with historically faithful dependencies, 3-hour timeout, no network access

Empirical Validation / Results

Main Findings

Table 3: Recall, Precision, and F1 Scores (sorted by F1)

Key Observations

1. Recall vs. Precision Trade-off: Recall varies significantly across models (correlated with total findings, Pearson r = 0.94), while precision remains consistently low (4.8–8.1%)

2. Reasoning Effort Effects: Increasing reasoning effort raises recall (5.3% → 17.3% for GPT-5.6 Luna) without changing precision

3. Bug Validity: LLM judge panel confirms 96% of matched findings and 84% of unmatched findings as real bugs

4. Bug Type Distribution: Agents over-report local correctness errors while maintainers fix more incomplete input handling:

5. Agent Agreement: Within-category overlap between agent runs has median ratio 6.3× compared to overlap with human fixes, showing agents agree with each other more than with maintainers

Harness Ablation Results

Theoretical and Practical Implications

Precision Ceiling Estimation

The authors model the total bug pool using exponential, Weibull, and lognormal distributions, finding the lognormal fits best. They estimate:

• 5.2k–5.9k bugs fixed within 5 years
• 7.2k–9.4k bugs fixed within 10 years

This implies a precision ceiling of 26–47% even for a perfect bug finder, since the reference set is incomplete.

Systematic Differences in Bug Discovery

The paper identifies fundamental differences between agent and human bug discovery:

1. Trigger Requirements: Humans find more bugs requiring specific environments (10.8% vs. 4.7%) or visual inspection (3.8% vs. 0.7%), while agents find bugs needing only specific inputs (83.9% vs. 74.3%)

2. Bug Types: Agents over-report local faults (wrong arithmetic, unhandled degenerate values) while humans focus on incomplete input handling and public API errors

3. Contamination Analysis: No evidence of training data contamination for models with published cut-offs (Holm-adjusted p-values ≥ 0.20)

Practical Implications

• Model Selection: Cheaper models (DeepSeek V4.1 Flash at $73, GPT-5.6 Luna xhigh at$131) can outperform expensive ones (GPT-5.6 Sol at $1,185) in F1 score
• Ensembling: Combining multiple runs can improve performance—union of Muse Spark runs achieves 26.7% recall; intersection of Luna runs achieves 10.5% precision
• Bug Finding vs. Fixing: The gap between agent and human bug priorities suggests different evaluation criteria are needed for proactive bug discovery versus bug fixing

Conclusion

Main Takeaways

1. PROBE provides the first benchmark for whole-repository bug discovery, enabling quantitative evaluation of coding agents' proactive bug-finding capabilities
2. Current frontier models achieve at most 20.8% recall and 8.1% precision against maintainer-fixed bugs, with stronger models primarily producing more findings rather than better ones
3. Agent-found bugs are largely real but systematically different from maintainer-fixed bugs, with agents over-reporting local correctness errors and under-reporting incomplete input handling and API surface issues

Future Directions

• Methodological improvements: Methods that exercise software as users do (e.g., testing, runtime analysis) could find bugs closer to those maintainers fix
• Extended scope: Applying the pipeline to additional languages and larger codebases
• Bug-finding capability improvement: Finding bugs that matter to humans is identified as "a key frontier for scaling automated software engineering"

Key Quote

"While agents find real bugs, they are mostly not the ones maintainers fix. Our benchmark PROBE makes this gap measurable."