Operator Insights

Reported by others3

1. 

   Benoît ViguierMaintainerLychee (LycheeOrg)

   Open source maintainers reviewing a month of pull requests. CodeRabbit open source plan.

   Lychee blog, published Sep 13, 2025Read the write-up (opens Lychee blog in a new tab)

   Observation

   Lychee's maintainer graded every CodeRabbit finding from a month of pull requests and rated 72% relevant.

   Takeaway

   Grade a month of CodeRabbit findings by hand before trusting an accuracy figure. Separate relevant findings from nitpicks and wrong comments to decide which settings to change.

   What happened

   CodeRabbit raised 290 findings across 28 pull requests with 32,784 lines added. The maintainer rated 35% quality improvements, 13% thoughtful, 21% nitpicks, 13% wrong assumptions, and 15% useless. Critical catches included a Zip Slip flaw and IDOR bugs. Hourly review limits led the maintainer to batch pushes.

   • Public engineering write-up
   • Team metric measured
   • No model named

   Deployment details

   Open source maintainers reviewing a month of pull requests. CodeRabbit open source plan.

   • Public engineering write-up
   • Team metric measured
   • No model named

   Team

   Lychee maintainers

   Duration

   One month

   Workload

   28 pull requests: 32,784 lines added, 4,768 removed, 693 files

   Product version

   CodeRabbit open source plan

   Measurement

   Each of 290 findings categorized by hand

1. KubernetesJul 27, 2026GitHub (kubernetes/org)Kubernetes maintainers called CodeRabbit excellent on Kueue, then kept it off kubernetes/kubernetes until it is opt-in and never load-bearing.

   

   Kevin HannonJobSet maintainer and Kueue reviewer (Red Hat)Kubernetes

   Pilots on three Kubernetes subprojects from October 2025. CodeRabbit's free open source plan. Request to extend it to kubernetes/kubernetes in July 2026.

   GitHub (kubernetes/org), published Jul 27, 2026Read the thread (opens GitHub (kubernetes/org) in a new tab)

   Observation

   Kubernetes maintainers called CodeRabbit excellent on Kueue, then kept it off kubernetes/kubernetes until it is opt-in and never load-bearing.

   Takeaway

   Pilot CodeRabbit on a few repositories and document the conventions it must check. Keep it optional so review capacity does not depend on the bot.

   What happened

   Kueue maintainers called CodeRabbit excellent and wanted to keep it, though it forgot lessons they taught it in comments. Agent Sandbox maintainers preferred its comment handling to Copilot's, though Copilot caught issues it missed. Hannon requested CodeRabbit for kubernetes/kubernetes in the 1.38 release cycle. SIG Architecture required opt-in reviews and public read access, with written posting rules. The bot could neither block review nor become essential to it. Hannon closed the request on August 14, 2026.

   • Public organization write-up (GitHub issue threads)
   • Maintainer feedback, no metrics
   • No model named

   Deployment details

   Pilots on three Kubernetes subprojects from October 2025. CodeRabbit's free open source plan. Request to extend it to kubernetes/kubernetes in July 2026.

   • Public organization write-up (GitHub issue threads)
   • Maintainer feedback, no metrics
   • No model named

   Team

   Maintainers of Kueue, JobSet, and Agent Sandbox

   Duration

   October 2025 to August 2026

   Workload

   Pull request review on Kubernetes subprojects; Kueue and Agent Sandbox have about 3,100 and 4,400 GitHub stars (as of Oct 28, 2026)

   Product version

   CodeRabbit GitHub app on the free open source plan

2. NixOSOct 27, 2025GitHub (NixOS/org)The Nix team installed CodeRabbit for opt-in reviews only, and a maintainer credits it with stopping use-after-free bugs.

   

   Jörg ThalheimNix team memberNixOS

   Nix package manager repository. Requested October 27, 2025. CodeRabbit's free open source plan with automatic reviews off.

   GitHub (NixOS/org), published Oct 27, 2025Read the thread (opens GitHub (NixOS/org) in a new tab)

   Observation

   The Nix team installed CodeRabbit for opt-in reviews only, and a maintainer credits it with stopping use-after-free bugs.

   Takeaway

   Turn off CodeRabbit's automatic reviews and walkthrough extras before the first pull request. Let reviewers decide when to call it.

   What happened

   Thalheim asked the NixOS organization to install CodeRabbit after it found bugs in other projects Thalheim maintains, with maintainers requesting a review per pull request. Organization owners approved it the same day. The first review included a generated poem. At Thalheim's request, the team turned off automatic reviews. Another contributor cited the code of conduct's rule on LLM and bot spam. Thalheim credits CodeRabbit with catching use-after-free bugs and twisted conditions before they merged.

   • Public organization write-up (GitHub issue thread)
   • No team metrics reported
   • No model named

   Deployment details

   Nix package manager repository. Requested October 27, 2025. CodeRabbit's free open source plan with automatic reviews off.

   • Public organization write-up (GitHub issue thread)
   • No team metrics reported
   • No model named

   Team

   Nix team maintainers

   Duration

   From October 2025

   Workload

   Pull request review on the Nix repository, requested per pull request

   Product version

   CodeRabbit GitHub app on the free open source plan