Key findings
- Noise is the most frequent complaint. 22 of the 31 accounts that discuss comment volume call it a problem.
- Accounts split almost evenly on whether the catches pay for the comments. 15 say they do. 13 removed or replaced CodeRabbit, limited it to on-demand review, or found it unhelpful.
- All 25 accounts about bug detection report real catches. They range from typos to an authorization bypass.
- The two teams that graded findings found most of them useful. FerroEHR measured about 60% true positives over 25 pull requests, and Lychee's maintainer rated 72% of 290 findings relevant.
- Every account that discusses human review keeps a person as the merge gate.
CodeRabbit catches real bugs, and its comment volume decides whether a team keeps it. Teams that kept it cut the volume with settings, such as a quieter profile, written path rules, and one review per ready pull request. Projects that left the defaults on every push switched it off.
Why review became the constraint
Teams added CodeRabbit when agent-written changes outgrew their reviewers. One Hacker News commenter's company adopted it in 2026 while trying to cut down large AI-written pull requests. Kubernetes SIG Release added two weeks to the release cycle to absorb more changes, and maintainers then asked for CodeRabbit on the main repository.
On repositories where agents push often, CodeRabbit's default review on every push used up hourly allowances and piled on comments. ShakaCode, OpenWISP, and Nathan Payne each changed when it reviews.
What it catches
Bug catches draw the most consistent praise. Lychee's maintainer lists a Zip Slip flaw (path traversal through archive extraction) and IDOR flaws (insecure direct object references) among 290 findings. Uri Harduf credits it with an authorization bypass and a privacy bug. Nix maintainer Jörg Thalheim credits it with catching use-after-free bugs before they merged.
Its most reliable catches are local mistakes, such as typos, swapped arguments, leftover debug code, missed fixes, and broken conventions. For Martin Zoeller, messy frontend state handling was the catch it made most reliably, and it found no larger bugs.
Where the noise comes from
The nitpicky profile produced 20 or more comments on one large pull request for Zoeller's client team. Review on every push added more. ShakaCode counted 103 top-level comments across 82 open pull requests. AsyncAPI turned off walkthrough extras such as poems and suggested labels. Nix turned off automatic reviews after the first review arrived with a poem. Kubernetes reviewers asked that the bot review only when a person calls it.
Wrong findings cost more than nitpicks, because someone has to disprove each one. Zoeller reports invented rules, and business-logic comments CodeRabbit lacked the context to judge. FerroEHR measured about 60% true positives, then replaced CodeRabbit with deterministic analysis.
What the teams that kept it changed
Teams that kept CodeRabbit changed its configuration first. Kubernetes pilots wrote their conventions down and added AGENTS.md. Nix chose opt-in reviews. ShakaCode collapsed walkthrough output and turned off per-push review. OpenWISP reviews only pull requests that pass validation. TorchSim kept on-demand review. Fedimint turned off only the poem and later removed CodeRabbit because the team still found it annoying.
Each of those changes is a setting in .coderabbit.yaml. The settings are the review profile, path filters, path instructions, automatic review controls, and the approval delay for rules CodeRabbit learns from review chat.
How to measure it
Choose the measure and the thresholds before turning CodeRabbit on. Write down what keeps automatic review on, what limits it to on-demand use, and what removes it.
CodeRabbit's dashboard reports the share of comments developers fully act on, by severity and category. Lychee graded each finding as relevant, nitpick, wrong, or useless. FerroEHR counted true positives over 25 pull requests before deciding whether the review should block merges.
- Acceptance rate by severity, from the CodeRabbit dashboard.
- A monthly hand grade of every finding, using Lychee's four grades.
- Time to first review and time to merge, measured before the pilot and during it.
Methodology
GILD read 78 public accounts from teams and maintainers who ran CodeRabbit, published from March 2025 to October 2026 and each opened on October 12, 2026. They include Hacker News threads, GitHub project decisions, engineering and personal blogs, independent reviews, security research, and recorded talks. Counts describe these 78 accounts and make no claim about CodeRabbit users at large. GILD checked product behavior against CodeRabbit's documentation on the same date.
Products in this piece
Sources
- Public source
- 15
- Official documentation
- 5
- Lychee blog: AI-assisted reviews, one month later (Sep 13, 2025)
- FerroEHR issue #2148: measure the AI reviewer's signal, then decide whether it gates (Aug 7, 2026)
- FerroEHR issue #2638: remove CodeRabbit, SonarQube Cloud replaces it (Aug 24, 2026)
- Martin Zoeller: CodeRabbit AI Review: Is It Worth It for Your Team? (Jul 30, 2026)
- ShakaCode agent-workflows issue #746: Collapse CodeRabbit output and turn off incremental re-review (Sep 5, 2026)
- OpenWISP issue #745: Opt in valid PRs to continuous CodeRabbit reviews (Aug 31, 2026)
- AsyncAPI Generator issue #1514: change coderabbit's config to be less annoying (Apr 24, 2025)
- Fedimint issue #4775: CodeRabbit AI PR reviews, removal comment (Apr 20, 2025)
- TorchSim issue #249: Turn off Coderabbit? (Sep 16, 2025)
- Nathan Payne, mergepath issue #826: rebalance the review topology (Jul 30, 2026)
- island-io/mila pull request #221: replace CodeRabbit with Cursor Bugbot (Aug 31, 2026)
- kubernetes/org issue #6517: REQUEST: Enable CodeRabbit integration for kubernetes (Jul 27, 2026)
- kubernetes-sigs/kueue issue #13078: Feedback on CodeRabbit (Jul 15, 2026)
- kubernetes-sigs/agent-sandbox issue #1231: Feedback for AI pre-reviewer (CodeRabbit / Copilot) (Jul 20, 2026)
- NixOS/org issue #185: CodeRabbit GitHub app for the Nix repo (Oct 27, 2025)
- CodeRabbit docs: Configuration reference
- CodeRabbit docs: Automatic review controls
- CodeRabbit docs: Path-based review instructions
- CodeRabbit docs: Review rate limits
- CodeRabbit docs: Git platform review metrics