SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review

Summary (Overview)

• Novel framework: SWE-Review introduces agentic code review as a mechanism to close the loop on AI-generated pull requests (PRs), transforming one-shot PR generation into an iterative generate-review-revise process.
• New benchmarks and datasets: The authors construct SWE-Review-Bench (1,384 candidate PRs from 500 SWE-bench Verified issues across three PR generators) and SWE-Review-Traj (8,914 decision-correct agentic review trajectories for training open reviewers).
• Key results: Agentic review improves resolve rates from 27.5% → 56.9% (Qwen3-30B-A3B), 50.9% → 68.8% (Qwen3-Coder-30B-A3B), and 72.2% → 75.4% (GLM-5) through iterative review-revision loops.
• Superiority over single-turn review: Agentic review consistently outperforms fixed-context single-turn review in both Decision Accuracy (DA) and Resolve Rate after Revision (RRR), with gains widening on harder, non-local patches.
• Training and test-time benefits: Review trajectories improve issue-resolution models via mixed training (up to +5.6 points resolve rate) and enable efficient test-time scaling (22.9% → 38.4% resolve rate in only 4 revision rounds).

Introduction and Theoretical Foundation

Background and Motivation

AI coding tools have evolved from autocomplete assistants to autonomous agents capable of proposing pull requests for real-world software engineering (SWE) issues. However, one-shot PR generation remains open-loop: after an agent produces a candidate PR, there is no reliable mechanism for determining whether the issue has been resolved or for diagnosing how the PR should be revised.

Code review—the practice of inspecting changes before merging—is identified as the natural mechanism for closing this loop. A reviewer provides both:

1. A binary decision (accept or request changes)
2. A diagnosis of what is wrong and how to fix it

The authors identify three critical gaps in current research:

• Lack of systematic benchmarks for agentic code review
• No reliable metrics measuring review's role in issue resolution
• Scarcity of public training data for open reviewer models

Theoretical Foundation

The paper formalizes agentic code review as a repository-grounded task. For each review instance, the input consists of:

• A repository checkout at the relevant commit
• A natural-language issue
• A candidate PR with proposed diff, title, and body

The reviewer does not receive the golden patch or hidden test results. It may browse files, search code, inspect dependencies, and execute commands before submitting a review report containing:

• Decision: approve if the patch resolves the issue, or request changes
• Diagnosis: concrete defects, cited code locations, and actionable fixes

Methodology

Evaluation Metrics

Three complementary metrics are defined:

1. Completion Rate (CR): Measures whether the reviewer produces a parseable final review.

2. Decision Accuracy (DA): Compares the approve/request-changes decision against the patch's true resolve status. Incomplete reviews are assigned 0.5 (chance-level).

3. Resolve Rate after Revision (RRR): Evaluates diagnosis operationally as the final resolve rate after review and revision. Approved patches are kept unchanged; patches marked request-changes are returned to the original PR generator with review feedback for one revision.

SWE-Review-Bench Construction

• Based on SWE-bench Verified (500 real-world issues with executable test suites)
• Three PR generators spanning quality distributions:
  • GLM-5: 72.2% resolve rate (n=500)
  • Qwen3-Coder-30B-A3B: 50.9% resolve rate (n=462)
  • Qwen3-30B-A3B: 27.5% resolve rate (n=422)
• Total: 1,384 PRs after filtering empty patches

SWE-Review-Traj Dataset

• Sourced from SWE-rebench (~6k issues after filtering)
• 14,156 candidate PRs generated with three models
• Teacher: open-weight GLM-5 with thinking enabled (avoiding closed-model bias)
• Decision-correct filtering: 8,914 trajectories where reviewer correctly approves resolving patches or correctly requests changes on non-resolving ones

Trajectory Validation

Semantic validation (Table 1a): Two judges (Claude Opus 4.6 and GPT-5.4) rate diagnoses on 1–5 scale:

Cohen's κ = 0.72 agreement between judges; only 3.3% of ratings differ by ≥2 points.

Functional validation (Table 1b): RRR under varying review completeness:

Empirical Validation / Results

Agentic vs. Single-Turn Review

Agentic review consistently outperforms single-turn review (diff-only and diff + context) across all three PR generator splits (Figure 3). Key results:

• Qwen3-30B-A3B split: RRR rises from 44.1% (best single-turn) to 52.6% (agentic), nearly doubling from the 27.5% no-review baseline.
• Difficulty stratification (Figure 4): The gap between agentic and single-turn review widens monotonically from easy to hard patches, confirming that repository exploration matters most when patch correctness cannot be assessed from the diff alone.

Case Study: sympy-13877

The candidate PR adds a NaN guard in exprtools.py that prevents a crash but still leaves det() returning incorrect nan values. Single-turn reviewers approve this symptom fix, while agentic reviewers trace the call chain to matrices.py, identify that cancel(ret) is called without assigning its result, and reject with a one-line upstream fix.

Reviewer Distillation Results (Table 2)

Subscripts show percentage-point changes relative to no-review baseline.

Mixed Training Results (Table 3)

Adding review trajectories to issue-resolution training improves both roles:

Test-Time Scaling (Figure 5)

Three strategies compared (all using Qwen3-30B-A3B as generator, 8B judges):

• Verifier-selected best-of-N (SWE-Lego-Verifier-8B): No early exit; sample cost grows with budget
• Reviewer-gated resampling: Stops when reviewer approves; no diagnostic feedback
• Review-guided iterative revision: Each rejection provides feedback for next attempt

Results: Review-guided iterative revision reaches 38.4% resolve rate within 5-sample budget using only 2.44 samples on average, versus 32.3% (8.9 samples) for reviewer-gated resampling at 16-sample budget.

Theoretical and Practical Implications

Theoretical Contributions

1. Review as evidence-gathering: The paper reframes code review from a single-turn comment-generation task to an agentic capability requiring adaptive repository exploration. This is theoretically grounded in the observation that patch correctness often cannot be assessed from the diff alone—tracing call chains, checking related implementations, and verifying behavior require non-local evidence.

2. Transferable repository reasoning: Mixed training results suggest that review and repair share a transferable repository-reasoning skill rather than being isolated capabilities. This supports a unified view of code understanding in software engineering agents.

3. Richer interface for test-time scaling: Unlike scalar scorers that reduce each attempt to a single ranking score, agentic review exposes a discrete decision (for gating/early stopping) and structured diagnosis (for iterative revision), enabling a fundamentally more efficient scaling paradigm.

Practical Implications

1. Closed-loop issue resolution: SWE-Review provides a practical mechanism for moving from one-shot PR generation to reliable closed-loop issue resolution, with demonstrated gains across three PR generator quality distributions.

2. Open training data: SWE-Review-Traj fills a critical data-scarcity gap, enabling open-source researcher training of reviewer models (improving 8B model completion rate from ~4% to 71–84%).

3. Unified agents: Mixed training enables self-contained generate-review-revise agents where the same model writes, reviews, and revises patches, improving final resolve rate by up to 10.6 points.

4. Efficient test-time scaling: Review-guided iterative revision achieves better results with fewer samples than verifier-based best-of-N, making it a practical choice for compute-constrained deployment.

Conclusion

SWE-Review establishes agentic code review as a central mechanism for closing the loop on AI-assisted issue resolution. Key takeaways:

• Agentic review outperforms fixed-context single-turn review in both decision accuracy and downstream revision usefulness, with gains concentrated on harder, non-local patches.
• Review trajectories serve as valuable training data for both reviewer distillation and mixed training that improves issue-resolution capabilities.
• Review-guided iterative revision enables effective and efficient test-time scaling.

Limitations and Future Directions

• Scope: Focus on AI-generated PRs for SWE-style issue resolution; broader scenarios (feature implementation, refactoring, documentation, migration, architectural changes) remain unexplored.
• Functional metrics only: DA and RRR measure issue resolution but not style, readability, maintainability, security, performance, or project-specific conventions.
• Relative comparisons: Limited to selected executable tasks, repositories, model families, prompts, and the OpenHands-SDK scaffold.

Future work should extend agentic review to broader PR types and non-functional review criteria.