Key findings
- Generation speed and self-reported speedups do not show what shipped. In METR's trial, developers believed AI sped them up while it slowed them down.
- Hold review standards constant during the pilot so quality comparisons stay honest.
- Track spend per workflow and per repository, because organization-wide averages hide the spikes.
- Check security and data handling before you expand access.
Judge an AI coding agent on the changes that merge after normal review, tracked for at least six weeks beside quality, cost, and governance. Run the pilot on two or three representative teams with a control group.
Why most pilots mislead
Most pilots measure enthusiasm and output volume. Both rise fast and say little about whether more working software ships.
Developers misjudge their own speed. In METR's randomized trial of 16 experienced open-source developers on 246 real issues in early 2025, developers took 19% longer with AI tools allowed. Yet afterward the developers believed AI had sped them up by 20%. METR's 2026 follow-up, with 57 developers and more than 800 tasks, estimated speedups of 18% for returning developers and 4% for new ones. METR calls that data an unreliable signal of the current effect, because many developers declined to work without AI.
The four measures
A useful pilot tracks four things side by side for at least six weeks.
- Accepted delivery: changes merged after normal review, per engineer per week.
- Quality: reverted changes, escaped defects, and review rework.
- Cost: spend per seat, per workflow, and per repository.
- Governance: data exposure, permissions requested, and security exceptions.
Watch the review queue
Output grows before review capacity does. Spotify reports 76% more pull requests to review after broad adoption, and its platform leadership found that change volume outpaced verification controls before it adjusted them.
Vendor dashboards help with the first measure. Claude Code's Team and Enterprise analytics report accepted lines and, with the GitHub integration, pull requests shipped with its help. Anthropic labels those contribution metrics as public beta.
Running the pilot
Pick two or three representative teams and a control group. Agree review standards before the pilot starts and keep them unchanged.
Review results at the midpoint. Adjust the scope if you need to, and keep the review standards fixed.
Methodology
The protocol is GILD's recommendation, built on the published studies, engineering accounts, and vendor documentation linked below, each checked on the date shown.
Products in this piece
Sources
- Public source
- 4
- Official documentation
- 1
- METR: Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity (Jul 10, 2025)
- METR: We are Changing our Developer Productivity Experiment Design (Feb 24, 2026)
- Spotify Engineering: Coding Is No Longer the Constraint (Jun 3, 2026)
- Spotify Engineering: AI Changed How Spotify Builds (Sep 16, 2026)
- Claude Code docs: Analytics