# Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild

> A large-scale study of CodeRabbit finds 56.3% of agentic code reviews are rejected, primarily due to invalid suggestions and misalignment with developer practices.

- **Source:** [arXiv](https://arxiv.org/abs/2607.03316)
- **Published:** 2026-09-12
- **Permalink:** https://picx.dev/p/WDhHKB
- **Whiteboard:** https://picx.dev/p/WDhHKB/image

## Summary

## Summary (Overview)

- This paper presents the first large-scale empirical study of **agentic code reviews** using CodeRabbit as a case study, analyzing 31,073 pairs of code review comments and developer feedback from 10,191 pull requests across 239 GitHub repositories spanning ten programming languages.
- Results show **mixed reception** of agentic reviews: 36.4% were accepted, 7.3% triggered discussion, and 56.3% were rejected by developers.
- Rejections were primarily driven by **invalid suggestions** (false positives, redundant, out-of-scope) and **misalignment with developer coding practices** and intent.
- CodeRabbit reviews focus more on **functional concerns** (75.9%) than evolvability, yet functional reviews face higher rejection rates (48.6%–57.8%) largely due to invalidity.
- Lightweight learning-based methods achieve up to **76% F1 score** for early prediction of rejected reviews, demonstrating learnable patterns between code reviews and developer feedback.

## Introduction and Theoretical Foundation

### Background
The paper addresses the emerging paradigm of **agentic code review**, where Large Language Models (LLMs) function as autonomous reviewer agents providing real-time, inline feedback on pull requests (PRs). CodeRabbit is identified as one of the most popular agentic code review tools, designed to provide context-aware, inline feedback directly within PRs.

### Motivation
Despite widespread adoption of such tools, **developer feedback to agentic code reviews remains largely unexplored**. Prior work has examined:
- Human-written review quality and developer reactions (e.g., anti-patterns like confused, conflicting, brief, shallow, and toxic reviews in OpenStack)
- LLM-based tools' practical effectiveness in eliciting subsequent code changes

However, there lacks a **large-scale analysis** of how developers in open source actually respond to various types of agentic code reviews at the instance level, and the underlying reasons behind their feedback.

### Research Questions
The study answers three research questions:
1. **RQ1**: What feedback did developers provide on CodeRabbit's code reviews?
2. **RQ2**: How do different types of concerns raised in CodeRabbit's code reviews relate to developer feedback?
3. **RQ3**: Can LLM-based approaches predict developer rejections of agentic code reviews?

## Methodology

### Data Collection
- Queried **GitHub REST API** for pull requests containing review comments that directly mention CodeRabbit
- Filtered to projects with sustained agent interaction across ten popular programming languages
- Excluded reviews without developer replies, yielding **31,073 CodeRabbit reviews** from **10,191 PRs** across **239 repositories**

### Analysis Approach
1. **Lightweight coding** on 297 sampled agentic code reviews (10% of dataset) via stratified sampling across repositories and languages
2. For each sample, examined:
   - The code diff under review
   - The inline agentic code review comment
   - The subsequent developer feedback
3. **Open coding** on the 17,500 rejected suggestions to identify rejection reasons
4. **Statistical analysis** including Kendall's Tau (𝜏), p-values, and Sen's Slope (𝛽) for trend analysis
5. **LLM-based prediction experiments** testing various approaches with and without code dif context

### Classification of Developer Feedback
Developer feedback was categorized into:
- **Accepted**: Review led to code changes, new commits, PRs, or issue tickets
- **Triggered Discussion**: Review induced further discussion but no direct action
- **Rejected**: Review was dismissed by the developer

## Empirical Validation / Results

### RQ1: Developer Feedback Distribution

**Key Finding I:** Developers accepted 36.4% of agentic code reviews, leading to new commits, PRs, or issue tickets. The proportion of accepted reviews increased by **12.5% since release**, showing statistically significant improvement over time (p ≤ 0.05).

**Key Finding II:** For the 56.3% of rejected reviews, the primary reasons were:

| Rejection Reason | Percentage |
|-----------------|------------|
| Misalignment with coding practices | 42.0% |
| Invalid suggestions (total) | 58.0% |
| - False positives | 43.3% |
| - Out of scope | 10.0% |
| - Redundant | 4.7% |

Within misalignment with coding practices:
- Intended design trade-offs: 23.7%
- Personal coding preferences: remainder

**Trend Analysis:** Misalignment with coding practices has shown a statistically significant increase of **0.3% per month**, while invalid suggestions decreased by 7.5% since release, suggesting CodeRabbit is becoming more technically proficient but remains misaligned with developer practices.

### RQ2: Concern Types and Feedback

**Key Finding III:** Most CodeRabbit reviews focus on **functionality (75.9%)**, with a growing emphasis on evolvability-related feedback (17.5% since release). However:

| Concern Type | Rejection Rate | Primary Rejection Reason |
|--------------|---------------|-------------------------|
| Functional | 56.2% | Invalid (35.8% false positives) |
| Evolvability | 56.7% | Misalignment with coding practices (31.6%) |

Functional defects were the most common sub-type but yielded a **56.5% rejection rate**, with 35.8% of suggestions flagged as invalid by developers. Evolvability reviews were more likely to be technically valid but rejected due to misalignment with developer practices.

### RQ3: Early Prediction of Rejections

Experimental setup tested various approaches using the agentic code review as input, with and without associated code dif:

| Approach | F1 Score |
|----------|----------|
| Zero-shot prediction | Unviable |
| Lightweight learning-based methods | Up to **76%** |

Key findings:
- Zero-shot prediction of rejection at inference time is **unviable** when restricted to only the current review and code dif
- Prior developer feedback encodes **learnable rejection patterns** even within limited context
- Lightweight LLMs can address the precision bottleneck of direct-prompting approaches

## Theoretical and Practical Implications

### For Tool Builders
1. **Reduce fundamental program understanding errors** with specific lines of code under inspection
2. Develop **deeper understanding of system component integration** across repositories
3. Enable **runtime inspection of system behavior**
4. Better leverage **historical software development artifacts** to align with developer coding practices

### For Developers
- Findings highlight **strengths and weaknesses** of agentic code reviews in current form
- Helps align **expectations** of these tools
- The 36.2% of functional defects accepted and 7.3% triggering discussion serve as **valuable complements to human reviews**

### For Researchers
- Developer feedback provides **meaningful insights into code review quality**
- As agentic tools generate large volumes of interaction data, developer feedback could serve as an **additional evaluation signal**, complementing traditional metrics

## Conclusion

This paper presents the first in-depth empirical study of CodeRabbit's agentic code reviews, grounded in developer feedback and underlying reasons. Key takeaways:

1. **Mixed reception**: 36.4% acceptance, 56.3% rejection, 7.3% discussion-triggering
2. **Primary rejection drivers**: Misalignment with coding practices and invalid suggestions (false positives being most common)
3. **Functional focus with high rejection**: CodeRabbit increasingly focuses on functional defects but suffers from high invalidity rates
4. **Learnable patterns**: Lightweight LLMs can effectively predict rejections (up to 76% F1), suggesting viable gating mechanisms

### Future Directions
- Leverage developer feedback as an additional evaluation signal for code review quality
- Explore integration of predictive approaches as **quality gates** in code review workflows
- Further investigation into reducing non-helpful reviews to improve perceived reliability and foster long-term trust in agentic review tools

### Limitations
- Content of developer responses alone provides sufficient signal (no analysis of non-textual feedback)
- Results capture behaviors likely shared across similar tools given CodeRabbit's common design patterns
- Stratified sampling and filtering of minimal-interaction projects ensures representative distributional characteristics

---

_Markdown view of https://picx.dev/p/WDhHKB, served by PicX — AI-generated visual whiteboard summaries of research papers._