Devin+39 Positive

Operator Insights

What are engineering leaders learning from deploying Devin?

10 accounts: 1 failure, incident or rollback, 1 switched, 1 run it beside another tool, 2 evaluated it, 1 adopted it and 4 reported by others

Failures, incidents and rollbacks1

Hamel HusainCo-author with Isaac Flath and Johno WhitakerAnswer.AI
Answer.AI team. About one month. 20 tasks across new projects, research, and existing code.
Its own announcement, published Jan 8, 2025Read the write-up (opens the announcement in a new tab)
Observation
Answer.AI gave Devin 20 tasks in a month. Three succeeded, and the team could not predict which would.
Takeaway
Before delegating at scale, check whether your team can predict which tasks the agent will finish, and weigh detailed accounts of shipped work over launch videos.
What happened
The team ran Devin through Slack and its web interface on 20 real tasks and counted 14 failures, 3 inconclusive results, and 3 successes. Simple tasks stretched to days, and Devin spent over a day trying to deploy several apps to one Railway deployment, which Railway does not support. The team went back to tools where developers drive each step.
“The results were sobering. Out of 20 tasks, we had 14 failures, 3 successes (including our 2 initial ones), and 3 inconclusive results.”
Hamel Husain
- Public engineering write-up
- Task outcomes counted by the authors
- No model named
Deployment details
Answer.AI team. About one month. 20 tasks across new projects, research, and existing code.
- Public engineering write-up
- Task outcomes counted by the authors
- No model named
Team
Three Answer.AI authors testing on their own work
Duration
About one month, published January 8, 2025
Workload
20 tasks: new projects, research, and analysis or changes to existing code
Measurement
Task outcomes counted by the authors: 3 successes, 3 inconclusive, 14 failures

Switched to or from it1

Thibaut LaBarreEngineering LeadAngelList
AngelList on-call incident response, August to October 2026.
Cognition, published Oct 13, 2026Read the write-up (opens Cognition in a new tab)
Observation
AngelList switched off its own incident agent after two weeks of Devin Oncall beside it.
Takeaway
Run a candidate agent beside the incumbent on the same incidents before switching; this result is the vendor's account.
What happened
AngelList ran Devin Oncall beside Sky, its homegrown incident agent, on the same incidents, then switched Sky off about two weeks later. Cognition's story reports root-cause correctness rising from about 20% with Sky to about 70% with Devin.
“About two weeks after the comparison, the team turned Sky off and made Devin Oncall the main incident responder. Everyone on the engineering team was on board with the switch.”
Thibaut LaBarre
- Vendor customer story (Cognition)
- Customer metric reported by the vendor
Deployment details
AngelList on-call incident response, August to October 2026.
- Vendor customer story (Cognition)
- Customer metric reported by the vendor
Team
AngelList engineering
Duration
About two weeks in parallel
Workload
On-call incident investigation
Product version
Devin Oncall
Measurement
Share of incidents with the correct root cause

Run it beside another tool1

Sahil LavingiaFounder and CEOGumroad
Gumroad and Flexile development, April 2025.
How I AI, published Apr 22, 2025Listen to the episode (opens How I AI in a new tab)
Observation
Gumroad's CEO prototypes in v0, hands the build to Devin, and opens Cursor when Devin stalls.
Takeaway
Pair Devin with an editor for the last mile, and make the environment easy for an agent to set up.
What happened
Lavingia's flow is v0 for the prototype, then Devin to implement it, with Cursor beside it when Devin does not finish. Gumroad also ran an internal competition with prize money for pull requests opened with Devin, where he opened 27.
“Yeah, so generally my flow is V0, Devin cursor, is probably how I would say it. Like generally V0 is my prototyping tool of choice. And once I have like a really good prototype that I'm happy with, then I go to Devin and if Devin sort of fails to completely finish then I open it up in cursor.”
Sahil Lavingia, at 8:20, from automatic captions
- Podcast episode
- Self-reported
Deployment details
Gumroad and Flexile development, April 2025.
- Podcast episode
- Self-reported
Team
Gumroad engineering
Workload
Flexile features
Product version
Devin beside v0 and Cursor
Measurement
Pull requests opened with Devin

Evaluated it2

Vikram Sreekanti and Joseph E. GonzalezJan 23, 2025Frontier AIRunLLM's founders trialled Devin for a month: small fixes worked, and 2 of 3 larger tasks gave nothing usable.
Vikram Sreekanti and Joseph E. GonzalezCo-foundersRunLLM
The RunLLM codebase, a month around the 2024 holidays.
Frontier AI, published Jan 23, 2025Read the write-up (opens Frontier AI in a new tab)
Observation
RunLLM's founders trialled Devin for a month: small fixes worked, and 2 of 3 larger tasks gave nothing usable.
Takeaway
Keep Devin on small, well-specified fixes until it proves itself on larger work in your codebase.
What happened
RunLLM's founders trialled Devin on their own codebase for a month. Small fixes cost 1 to 5 ACUs, US$2 to US$10 at US$500 for 250 ACUs; the three larger full-stack tasks averaged about 20 ACUs, and 2 of the 3 gave no usable result (Jan 23, 2025).
“The 3 bigger tasks we tried averaged about 20 ACUs and 2 of the 3 didn’t yield usable results.”
Vikram Sreekanti and Joseph E. Gonzalez
Public practitioner write-up
Self-reported, with ACU costs
Deployment details
The RunLLM codebase, a month around the 2024 holidays.
Public practitioner write-up
Self-reported, with ACU costs
Team
RunLLM founders
Duration
About a month
Workload
Full-stack product codebase
Product version
Devin
Measurement
ACUs per task, usable results
tlarkworthyJan 17, 2025Hacker NewsAfter a trial, one developer recommended cancelling Devin because its extra changes broke other code.
tlarkworthyDeveloper who trialled DevinTaktile
A developer's team, January 2025.
Hacker News, published Jan 17, 2025Read the write-up (opens Hacker News in a new tab)
Observation
After a trial, one developer recommended cancelling Devin because its extra changes broke other code.
Takeaway
Scope every task tightly and review the whole diff, since out-of-scope edits cost more than they save.
What happened
After trialling Devin, Hacker News user tlarkworthy wrote on Jan 17, 2025 that it kept adding changes beyond the task that broke other things, and that they would recommend cancelling it because it was not saving time and was expensive.
“…but it seems to always add extraneous changes beyond the task that can break other things.”
tlarkworthy
Public practitioner write-up
Self-reported
Deployment details
A developer's team, January 2025.
Public practitioner write-up
Self-reported
Team
One developer's team
Duration
A trial
Product version
Devin

Adopted it1

RoxOct 14, 2026RoxRox has Devin turn production failures into evals and open a weekly pull request.
Brian XuEngineer, Agents teamRox
Rox's Agents team, October 2026.
Its own announcement, published Oct 14, 2026Read the write-up (opens the announcement in a new tab)
Observation
Rox has Devin turn production failures into evals and open a weekly pull request.
Takeaway
A recurring, well-bounded job with a reviewable pull request is where an autonomous agent runs reliably.
What happened
Every week Devin reads about 350 production conversations from Rox's Braintrust traces, labels them, rebuilds failures as evals in per-branch environments and opens a pull request the team reviews, so production mistakes show up as failing evals (Oct 14, 2026).
“Every week, Devin, the coding agent from Cognition, reads real conversations on Rox from our production traces in Braintrust, turns them into evals, and opens a pull request that we review.”
Brian Xu
Public engineering write-up
Self-reported
Deployment details
Rox's Agents team, October 2026.
Public engineering write-up
Self-reported
Team
Rox Agents team
Duration
Weekly
Workload
Production conversation evals
Product version
Devin
Measurement
Production conversations read a week

Reported by others4

Eight SleepSep 4, 2025Eight SleepEight Sleep put Devin in a Slack channel as its data analyst and tripled the data work it ships each week.
Eight Sleep data teamCompany blog (Editorial byline)Eight Sleep
Small data team. About two months at publication. dbt, Snowflake, Looker, and Amplitude.
Its own announcement, published Sep 4, 2025Read the write-up (opens the announcement in a new tab)
Observation
Eight Sleep put Devin in a Slack channel as its data analyst and tripled the data work it ships each week.
Takeaway
Start Devin with the access a new analyst would get, add curated knowledge task by task, and settle what it may query before the whole company can tag it.
What happened
Eight Sleep's small data team gave Devin its dbt repository, Snowflake, Looker, and browser access to Amplitude, and opened a Slack channel where anyone can tag it. Setup took a couple of hours. When a new sales dashboard showed unusually high revenue, Devin traced the dashboard to its queries, ran checks in Snowflake, and found an email campaign that had outperformed. The team reports three times as many data features and investigations shipped per week, and an ad-hoc request queue near zero.
“Faster shipping. We’re shipping 3x as many data features and data investigations each week compared to before…”
Eight Sleep data team
Public organization write-up
Team metrics reported by the company
No model named
Deployment details
Small data team. About two months at publication. dbt, Snowflake, Looker, and Amplitude.
Public organization write-up
Team metrics reported by the company
No model named
Team
Eight Sleep's data team, answering requests from Product, Ops, Finance, Growth, and R&D
Duration
About two months at publication
Workload
Ad-hoc data requests, new dbt and semantic models, dashboards, and checks on numbers that look wrong
Product version
Devin in Slack, with access to the dbt repository, Snowflake, Looker, and Amplitude through its browser
Measurement
Data features and investigations shipped per week, before and after, as counted by the team
NubankNov 14, 2024CognitionNubank handed a monolith migration to Devin and kept its engineers on review.
Jose Carlos CastroSenior Product ManagerNubank
ETL monolith migration, 2023 to 2024.
Cognition, published Nov 14, 2024Read the case study (opens Cognition in a new tab)
Observation
Nubank handed a monolith migration to Devin and kept its engineers on review.
Takeaway
A migration made of thousands of similar sub-tasks suits delegation once the agent has learned one sub-task. Engineers then manage the project and approve the changes.
What happened
Nubank planned to split an ETL monolith by moving about 100,000 data class implementations. The job was planned for more than 1,000 engineers over 18 months. Nubank delegated it to a Devin fine-tuned on earlier manual migrations. Cognition reports an 8x to 12x engineering-hour efficiency gain and over 20x lower cost on the delegated scope.
“Delivering an 8-12x faster migration, lifting a burden from every engineer, and slashing migration costs by 20x.”
Cognition
Vendor case study
Organization metrics reported
No model named
Deployment details
ETL monolith migration, 2023 to 2024.
Vendor case study
Organization metrics reported
No model named
Team
More than 1,000 engineers in the migration scope
Duration
Weeks for business units such as Data and Risk, as Cognition reports, against 18 months planned for the manual migration
Workload
Monolithic ETL repository of over 6 million lines, about 100,000 data class implementations
Product version
A custom ETL migration Devin, fine-tuned on Nubank's earlier manual migrations
Measurement
Engineer hours per task against hours spent prompting and reviewing Devin; Devin's running cost against an engineer's hourly cost
David GriffithsJul 14, 2025American BankerCiti began rolling out Devin for simple, repeatable tasks beside Copilot; its developer organization numbers 40,000.
David GriffithsChief Technology OfficerCiti
Citi's developer organization of 40,000, as the rollout began in 2025.
American Banker, published Jul 14, 2025Read the report (opens American Banker in a new tab)
Observation
Citi began rolling out Devin for simple, repeatable tasks beside Copilot; its developer organization numbers 40,000.
Takeaway
Keep Devin on simple, repeatable work at scale and route every change through a developer.
What happened
American Banker reported on Jul 14, 2025 that Citi began giving developers Devin for library upgrades, rewrites, vendor patches, documentation and tests, beside GitHub Copilot. Griffiths states that agents produce artifacts for developers and deploy no code, and he would not yet ask one to build a complicated feature end to end.
“"It's not at the point where you would want to ask the agent to implement a very complicated feature from start to finish. We may get there with the technology in time, but today, I wouldn't recommend anyone to do that. We focus on the simple, repeatable tasks, which it can do at our scale.”
David Griffiths
Second-hand news report (American Banker)
Executive statement reported
Deployment details
Citi's developer organization of 40,000, as the rollout began in 2025.
Second-hand news report (American Banker)
Executive statement reported
Team
Citi's developer organization, 40,000 developers
Workload
Upgrades, rewrites, patches, documentation, tests
Product version
Devin beside GitHub Copilot
Marco ArgentiJul 25, 2025American BankerGoldman Sachs is giving developer teams Devin as a virtual colleague.
Marco ArgentiChief Information OfficerGoldman Sachs
Goldman Sachs' 12,000 developers, 2025.
American Banker, published Jul 25, 2025Read the report (opens American Banker in a new tab)
Observation
Goldman Sachs is giving developer teams Devin as a virtual colleague.
Takeaway
Devin enters as the delegated colleague beside the assistant every developer already has.
What happened
American Banker reported on Jul 25, 2025 that Goldman Sachs, which gave its 12,000 developers GitHub Copilot in 2024, is giving developer teams Devin to act as a team member that work can be delegated to.
“All the developers have Github Copilot. They are getting the Devin technology. There's a shift from the developer assistant that helps you auto complete your code or generate code for a specific task, so I now have a virtual colleague that I can delegate work to.”
Marco Argenti
Second-hand news report (American Banker)
Executive statement reported
Deployment details
Goldman Sachs' 12,000 developers, 2025.
Second-hand news report (American Banker)
Executive statement reported
Team
Goldman Sachs developer teams
Workload
Software development
Product version
Devin beside GitHub Copilot

Failures, incidents and rollbacks1

Answer.AI gave Devin 20 tasks in a month. Three succeeded, and the team could not predict which would.

Switched to or from it1

AngelList switched off its own incident agent after two weeks of Devin Oncall beside it.

Run it beside another tool1

Gumroad's CEO prototypes in v0, hands the build to Devin, and opens Cursor when Devin stalls.

Evaluated it2

RunLLM's founders trialled Devin for a month: small fixes worked, and 2 of 3 larger tasks gave nothing usable.

After a trial, one developer recommended cancelling Devin because its extra changes broke other code.

Adopted it1

Rox has Devin turn production failures into evals and open a weekly pull request.

Reported by others4

Eight Sleep put Devin in a Slack channel as its data analyst and tripled the data work it ships each week.

Nubank handed a monolith migration to Devin and kept its engineers on review.

Citi began rolling out Devin for simple, repeatable tasks beside Copilot; its developer organization numbers 40,000.

Goldman Sachs is giving developer teams Devin as a virtual colleague.