Summary (Overview)
- This paper presents the first large-scale empirical study of agentic code reviews using CodeRabbit as a case study, analyzing 31,073 pairs of code review comments and developer feedback from 10,191 pull requests across 239 GitHub repositories spanning ten programming languages.
- Results show mixed reception of agentic reviews: 36.4% were accepted, 7.3% triggered discussion, and 56.3% were rejected by developers.
- Rejections were primarily driven by invalid suggestions (false positives, redundant, out-of-scope) and misalignment with developer coding practices and intent.
- CodeRabbit reviews focus more on functional concerns (75.9%) than evolvability, yet functional reviews face higher rejection rates (48.6%–57.8%) largely due to invalidity.
- Lightweight learning-based methods achieve up to 76% F1 score for early prediction of rejected reviews, demonstrating learnable patterns between code reviews and developer feedback.
Introduction and Theoretical Foundation
Background
The paper addresses the emerging paradigm of agentic code review, where Large Language Models (LLMs) function as autonomous reviewer agents providing real-time, inline feedback on pull requests (PRs). CodeRabbit is identified as one of the most popular agentic code review tools, designed to provide context-aware, inline feedback directly within PRs.
Motivation
Despite widespread adoption of such tools, developer feedback to agentic code reviews remains largely unexplored. Prior work has examined:
- Human-written review quality and developer reactions (e.g., anti-patterns like confused, conflicting, brief, shallow, and toxic reviews in OpenStack)
- LLM-based tools' practical effectiveness in eliciting subsequent code changes
However, there lacks a large-scale analysis of how developers in open source actually respond to various types of agentic code reviews at the instance level, and the underlying reasons behind their feedback.
Research Questions
The study answers three research questions:
- RQ1: What feedback did developers provide on CodeRabbit's code reviews?
- RQ2: How do different types of concerns raised in CodeRabbit's code reviews relate to developer feedback?
- RQ3: Can LLM-based approaches predict developer rejections of agentic code reviews?
Methodology
Data Collection
- Queried GitHub REST API for pull requests containing review comments that directly mention CodeRabbit
- Filtered to projects with sustained agent interaction across ten popular programming languages
- Excluded reviews without developer replies, yielding 31,073 CodeRabbit reviews from 10,191 PRs across 239 repositories
Analysis Approach
- Lightweight coding on 297 sampled agentic code reviews (10% of dataset) via stratified sampling across repositories and languages
- For each sample, examined:
- The code diff under review
- The inline agentic code review comment
- The subsequent developer feedback
- Open coding on the 17,500 rejected suggestions to identify rejection reasons
- Statistical analysis including Kendall's Tau (𝜏), p-values, and Sen's Slope (𝛽) for trend analysis
- LLM-based prediction experiments testing various approaches with and without code dif context
Classification of Developer Feedback
Developer feedback was categorized into:
- Accepted: Review led to code changes, new commits, PRs, or issue tickets
- Triggered Discussion: Review induced further discussion but no direct action
- Rejected: Review was dismissed by the developer
Empirical Validation / Results
RQ1: Developer Feedback Distribution
Key Finding I: Developers accepted 36.4% of agentic code reviews, leading to new commits, PRs, or issue tickets. The proportion of accepted reviews increased by 12.5% since release, showing statistically significant improvement over time (p ≤ 0.05).
Key Finding II: For the 56.3% of rejected reviews, the primary reasons were:
| Rejection Reason | Percentage |
|---|---|
| Misalignment with coding practices | 42.0% |
| Invalid suggestions (total) | 58.0% |
| - False positives | 43.3% |
| - Out of scope | 10.0% |
| - Redundant | 4.7% |
Within misalignment with coding practices:
- Intended design trade-offs: 23.7%
- Personal coding preferences: remainder
Trend Analysis: Misalignment with coding practices has shown a statistically significant increase of 0.3% per month, while invalid suggestions decreased by 7.5% since release, suggesting CodeRabbit is becoming more technically proficient but remains misaligned with developer practices.
RQ2: Concern Types and Feedback
Key Finding III: Most CodeRabbit reviews focus on functionality (75.9%), with a growing emphasis on evolvability-related feedback (17.5% since release). However:
| Concern Type | Rejection Rate | Primary Rejection Reason |
|---|---|---|
| Functional | 56.2% | Invalid (35.8% false positives) |
| Evolvability | 56.7% | Misalignment with coding practices (31.6%) |
Functional defects were the most common sub-type but yielded a 56.5% rejection rate, with 35.8% of suggestions flagged as invalid by developers. Evolvability reviews were more likely to be technically valid but rejected due to misalignment with developer practices.
RQ3: Early Prediction of Rejections
Experimental setup tested various approaches using the agentic code review as input, with and without associated code dif:
| Approach | F1 Score |
|---|---|
| Zero-shot prediction | Unviable |
| Lightweight learning-based methods | Up to 76% |
Key findings:
- Zero-shot prediction of rejection at inference time is unviable when restricted to only the current review and code dif
- Prior developer feedback encodes learnable rejection patterns even within limited context
- Lightweight LLMs can address the precision bottleneck of direct-prompting approaches
Theoretical and Practical Implications
For Tool Builders
- Reduce fundamental program understanding errors with specific lines of code under inspection
- Develop deeper understanding of system component integration across repositories
- Enable runtime inspection of system behavior
- Better leverage historical software development artifacts to align with developer coding practices
For Developers
- Findings highlight strengths and weaknesses of agentic code reviews in current form
- Helps align expectations of these tools
- The 36.2% of functional defects accepted and 7.3% triggering discussion serve as valuable complements to human reviews
For Researchers
- Developer feedback provides meaningful insights into code review quality
- As agentic tools generate large volumes of interaction data, developer feedback could serve as an additional evaluation signal, complementing traditional metrics
Conclusion
This paper presents the first in-depth empirical study of CodeRabbit's agentic code reviews, grounded in developer feedback and underlying reasons. Key takeaways:
- Mixed reception: 36.4% acceptance, 56.3% rejection, 7.3% discussion-triggering
- Primary rejection drivers: Misalignment with coding practices and invalid suggestions (false positives being most common)
- Functional focus with high rejection: CodeRabbit increasingly focuses on functional defects but suffers from high invalidity rates
- Learnable patterns: Lightweight LLMs can effectively predict rejections (up to 76% F1), suggesting viable gating mechanisms
Future Directions
- Leverage developer feedback as an additional evaluation signal for code review quality
- Explore integration of predictive approaches as quality gates in code review workflows
- Further investigation into reducing non-helpful reviews to improve perceived reliability and foster long-term trust in agentic review tools
Limitations
- Content of developer responses alone provides sufficient signal (no analysis of non-textual feedback)
- Results capture behaviors likely shared across similar tools given CodeRabbit's common design patterns
- Stratified sampling and filtering of minimal-interaction projects ensures representative distributional characteristics
Related papers
- Recursive Harness Self-Improvement
Recursive Harness Self-Improvement lets a few cheap prompt-level iterations outperform maximum test-time scaling baselines, cutting inference cost by up to 60% through better inter-agent context management, not longer reasoning.
- AttnCompress: Dynamic Attention-Guided Trajectory Compression for Software Engineering Agents
AttnCompress achieves 53.17% pass rate on SWE-Bench-Verified, beating prior compression SOTA while cutting tokens by 21.6% and costs by 33.6%.
- Self-Authored Verification Is Unreliable in Heuristic Self-Improving Agents
Self-improving agents that grade their own work systematically lose alignment with deployment performance, requiring a sealed external acceptance signal to prevent regression.