# (When) Should a Software Factory be Lights-off? Benchmarking Code Review Agents on Bug Detection (CRBugBench)

> CRBugBench shows top review agents catch 71.9% of human-verified bugs, but full lights-off software factories remain impractical, with agents struggling most on complex proprietary changes.

- **Source:** [arXiv](https://arxiv.org/abs/2610.13668)
- **Published:** 2026-10-17
- **Permalink:** https://picx.dev/p/dgaEA9
- **Whiteboard:** https://picx.dev/p/dgaEA9/image

## Summary

## Summary (Overview)

- **CRBugBench** is a new benchmark for evaluating code review agents' ability to detect human-verified bugs, containing **1,000 review tasks** across 98 open-source GitHub repositories and **800 tasks** from proprietary industry repositories.
- The best review agent (claude-opus-5 with copilot-cli) catches **71.9% of bugs** within its top-5 comments, up from 47.7% (opus-4.6) in five months of model evolution.
- Agents struggle more on proprietary industry repositories (up to **25.5% performance drop**), particularly for bugs requiring non-local reasoning and project context.
- **Exploration-nudging prompts** improve performance by up to 5.8% at only 10% higher cost, while multi-agent setups provide smaller gains at 5.55x cost.
- The paper provides actionable guidance for developers on when and how to adopt review agents, concluding that a fully "lights-off" software factory is not yet practical.

---

## Introduction and Theoretical Foundation

### Background and Motivation

The paper addresses a critical tension in modern software engineering: while coding agents have dramatically accelerated code generation, human code review velocity has not kept pace. This raises the question of whether we can build a **"lights-off" software factory** where AI both writes and reviews code with minimal human supervision.

### Limitations of Existing Benchmarks

Existing code review benchmarks (CodeReviewer, SWR-Bench, c-CRAB) treat human comments as ground truth, which the authors argue is problematic because:

- Human comments can be arbitrary, containing nitpicking or less useful comments accepted due to social dynamics
- Human comments can be too narrow, missing valid points that AI reviews might raise
- AI may provide high-quality reviews that differ from human reviewers

### Key Insight

The authors propose a more objective approach: **a good PR review should at least catch bugs that humans have confirmed and fixed**. This shifts evaluation from matching human comments to detecting human-verified defects.

---

## Methodology

### Repository Selection

- Selected **5 popular programming languages**: Python, Rust, C#, JavaScript/TypeScript, Java
- Filtered top-1,000 starred GitHub repos to those with >2,000 PRs and >200 contributors
- Sampled one repository per application domain per language, yielding **106 repositories** across 31 domains

### Bug Identification Pipeline

**Bugs identified before merge** (from PR review comments):
1. Mining: 304,601 PRs with 1,436,120 review threads collected; deterministic quality filters reduced to 472,039 candidate threads
2. Sampling and LLM filtering: Stratified sample of 90 threads per repository, LLM-filtered to remove non-bug comments → **865 vetted bugs**

**Bugs identified after merge** (from bug-fix PRs):
1. Mining: 195,075 candidate fix PRs with closing issues
2. Sampling and LLM filtering: 200 PRs per repository sampled → 10,784 fix PRs
3. **SZZ algorithm** traces defects to inducing PRs via `git blame` → 8,846 fix-inducing pairs
4. Ground truth screening with LLM → **689 fix-inducing PR pairs**

### Task Creation

Each task contains:
- Docker snapshot of repository at the inducing PR's base commit
- PR title/description as context
- Change provided as both a file and uncommitted working-tree modifications
- Agent writes review to `review.md` with findings self-ranked by severity

### Task Scoring

- **LLM-as-judge panel** with three models (gpt-5.5, claude-opus-4.8, gemini-3.1-flash), majority vote
- Human-LLM agreement: κ = 0.89; across-LLM agreement: Fleiss' κ = 0.95
- Primary metric: **recall@k** (proportion of tasks where ground-truth issue is in top-k findings)

### Regression Model

The paper models bug detection probability using multivariable logistic regression:

$$
\begin{array}{l} \text {review\_caught} \sim \log_ {\text {num\_changed\_lines\_z\_score}} + \log_ {\text {dependent\_files\_z\_score}} + \log_ {\text {dependency\_files\_z\_score}} \\ \quad + \log_ {\text {repository\_nloc\_z\_score}} + \text {num\_subchange\_types} + \text {extension\_entropy\_z\_score} \\ \quad + \text {bug\_is\_local\_visible} + \text {high\_severity\_bug} + C (\text {change\_type}, \text {Treatment(ref = feature)}) \\ \quad + C (\text {language, Treatment(ref = Python)}) + C (\text {bug\_iso\_impact, Treatment(ref = Functional Suitability)}) \\ \quad + C (\text {model\_key}) + C (\text {track}) + C (\text {calendar\_time}) \end{array}
$$

---

## Empirical Validation / Results

### RQ1: How well do review agents detect PR bugs?

**Table 1. Bug-detection performance and cost across model configurations on GitHub repositories.**

| Run | Track | Recall@1 | Recall@3 | Recall@5 | Recall@10 | Cost ($) | Latency (min) |
|-----|-------|----------|----------|----------|-----------|----------|---------------|
| claude-opus-5 | Cmt. | 48.3 | 71.2 | **79.6** | 86.0 | 2.55 | 8.6 |
| claude-opus-5 | Fix | 36.2 | 55.4 | **64.2** | 68.6 | 3.38 | 11.1 |
| claude-opus-4.8 | Cmt. | 45.6 | 65.9 | 70.0 | 71.5 | 1.54 | 6.5 |
| claude-opus-4.8 | Fix | 32.0 | 44.6 | 46.6 | 47.8 | 1.98 | 7.9 |
| gpt-5.6-sol | Cmt. | 43.5 | 61.9 | 65.6 | 66.6 | 1.50 | 3.7 |
| gpt-5.6-sol | Fix | 29.6 | 45.6 | 49.0 | 51.8 | 2.03 | 5.2 |
| gpt-5.6-luna | Cmt. | 43.5 | 60.4 | 63.9 | 65.4 | 0.51 | 3.1 |
| gpt-5.6-luna | Fix | 30.0 | 44.6 | 47.8 | 50.2 | 0.72 | 4.3 |

**Key findings:**
- Models are strong at **ranking findings**: 89% of found bugs appear in top-3 items
- Small models are cost-effective: gpt-5.6-luna matches 78% of claude-opus-5 performance at 21% cost
- **Human-agent collaboration** catches 82.1% of bugs (vs. 50% human-only, 71.9% agent-only)
- Agents drop up to **25.5% performance** on proprietary industry repositories

### RQ2: What factors impact review quality?

Key odds ratios from the regression analysis:

| Factor | Odds Ratio | Significance |
|--------|-----------|--------------|
| Larger diffs | OR = 0.591 | *** |
| Heterogeneous diffs | OR = 0.795 | * |
| Bug-fixing PRs | OR = 0.521 | *** |
| Dif-visible bugs | OR = 2.621 | *** |
| High-severity bugs | OR = 1.906 | ** |
| Usability bugs | OR = 0.454 | *** |

- **Bug visibility explains 20%** of the GitHub-industry performance gap
- Humans are better at maintainability issues (OR = 2.702, *), agents better at reliability and severe bugs
- Language (H3) and repository size (H5) showed no significant effects

### RQ3: Where do agents spend time and how do they fail?

- Agents spend **75% of turns** on reading (39%) and searching (36%), only **8% on executing code**
- Claude models execute more often (9.7% vs 6.9% for GPT models)
- **Persistence varies**: half of claude-opus-5 runs end by turn 35 vs. turn 17 for gpt-5.6-sol
- **95.6%** of executions inspect the defect file, but only **56.2%** actually hit the bugs
- Agents can be **gullible**: trust misleading PR descriptions or code comments as evidence of correctness

### RQ4: What agent design choices improve performance?

**Table 2. Prompting strategies comparison (excerpt).**

| Model | Track | Prompt | Recall@5 | Recall@10 | Cost ($) |
|-------|-------|--------|----------|-----------|----------|
| claude-opus-5 | Cmt. | Default | 79.6 | 86.0 | 2.55 |
| claude-opus-5 | Cmt. | Comprehensive | 79.9 (+0.3) | 87.2 (+1.2) | 2.76 |
| claude-opus-4.8 | Cmt. | Default | 70.0 | 71.5 | 1.54 |
| claude-opus-4.8 | Cmt. | Comprehensive | 75.5 (+5.5) | 78.4 (+6.9) | 1.67 |
| gpt-5.6-sol | Cmt. | Default | 65.6 | 66.6 | 1.50 |
| gpt-5.6-sol | Cmt. | Comprehensive | 71.3 (+5.6) | 74.2 (+7.6) | 1.36 |

**Key findings:**
- **Comprehensive prompting** is most cost-effective: +5.8% performance at 1.10x cost
- **Multi-agent setups** provide smaller gains (+2.8% on average) at 3.68x cost
- **Reasoning effort scaling**: gpt-5.6-luna improves from ~36% to 57% at 23.22x cost
- Prompt benefits vary by model; claude-opus-5 shows minimal gains (behaviors already incorporated in training)

---

## Theoretical and Practical Implications

### When Can a Software Factory Be Lights-Off?

The evidence suggests agents are **not yet ready for fully autonomous operation**, especially for:

- Large-scale proprietary industry repositories
- Diffs requiring non-local or domain-specific reasoning
- Consequential changes where human accountability matters

The authors support "lights-off" operation only for **narrowly scoped, low-consequence PRs**.

### Recommendations for Practice

1. **Scale review efforts based on PR complexity**: Small, homogeneous changes can use cheaper configurations; large, heterogeneous PRs warrant stronger models and careful human review
2. **Steer review directions based on priorities**: Provide custom instructions for usability, portability, backward compatibility, or maintainability when these matter
3. **Incorporate repository knowledge**: Especially for large proprietary repositories, provide repository-specific instructions and knowledge bases
4. **Monitor review quality and adapt harness over model evolution**: Harnesses that help one model may provide little benefit to later models; build custom benchmarks using the CRBugBench methodology

### Research Opportunities

- **Self-improving review agents** with memory for continual adaptation to evolving repositories
- **Human-in-the-loop efficiency**: Build systems to assist human inspection and maximize value at minimal cognitive cost
- **Calibrated uncertainty**: Train review agents that communicate uncertainty well to humans

---

## Conclusion

The paper introduces **CRBugBench**, a bug-grounded code review benchmark, and uses it to systematically study when and how code review agents detect human-verified bugs. Key takeaways:

- Review agents are **improving rapidly** (47.7% → 71.9% in five months) but still miss 20%+ of bugs humans found
- Agents **struggle with complex changes** and bugs requiring project context, particularly in proprietary repositories
- **Comprehensive prompting** improves detection more cost-effectively than multi-agent setups
- **Human oversight remains essential**, with humans and agents showing complementary strengths

The findings support retaining human oversight while providing concrete guidance for adapting review effort, context, and agent configurations to project needs. The benchmark is available at https://github.com/malusamayo/CRBugBench.

---

_Markdown view of https://picx.dev/p/dgaEA9, served by PicX — AI-generated visual whiteboard summaries of research papers._