Operator Insights

Failures, incidents and rollbacks3

1. 

   Joe LeonAuthor of Truffle Security's post on Gemini and Google API keysTruffle Security

   Security research on Google API keys published in the November 2025 Common Crawl.

   Its own announcement, published Feb 25, 2026Read the report (opens the announcement in a new tab)

   Observation

   Truffle Security found 2,863 live Google API keys that had silently gained Gemini access when the Generative Language API was turned on in their projects.

   Takeaway

   Audit every existing API key in a project before enabling Gemini on it, restrict each key to named APIs, and set a project spend cap.

   What happened

   Google API keys created years earlier for Maps or Firebase were treated as public. When a project enabled the Generative Language API, those keys could call Gemini, read uploaded files and cached content, and bill usage to the owner. Truffle Security found 2,863 such live keys, including keys of financial institutions, security firms and Google itself (Truffle Security, Feb 25, 2026).

   “We identified 2,863 live Google API keys vulnerable to this privilege-escalation vector.”

   • Public organization write-up (independent security research)
   • Count of exposed keys measured

   Deployment details

   Security research on Google API keys published in the November 2025 Common Crawl.

   • Public organization write-up (independent security research)
   • Count of exposed keys measured

   Team

   Truffle Security research

   Workload

   Scan of the November 2025 Common Crawl for live Google API keys

   Product version

   Gemini API through the Generative Language API

   Measurement

   2,863 live keys exposed (Feb 25, 2026)

1. ClineOct 13, 2026Cline on GitHubCline made Gemini 4 Argon its default model on October 16, 2026, then rolled the default back to Claude Sonnet 5.5 within days.

   

   Cline maintainersRelease notesCline

   The Cline coding agent's own model provider, whose default model serves users who choose none.

   Cline on GitHub, published Oct 13, 2026Read the write-up (opens Cline on GitHub in a new tab)

   Observation

   Cline made Gemini 4 Argon its default model on October 16, 2026, then rolled the default back to Claude Sonnet 5.5 within days.

   Takeaway

   Treat a new Gemini flagship as a candidate for a week of your own traffic before it becomes a default; Cline's default changed twice in four days.

   What happened

   Cline set Claude Sonnet 5.5 as its provider default in v4.1.24 on October 13, 2026. In v4.1.25 on October 16 it switched the default to Argon, and from October 17 to 26 it rolled back to Sonnet 5.5 at the top of its recommended list (Cline release notes on GitHub, October 2026). The notes give no reason.

   “With no model chosen, the Cline provider now uses Gemini 4 Argon, the new first model in Cline…”

   • Public organization write-up (Cline release notes on GitHub)
   • No metrics reported

   Deployment details

   The Cline coding agent's own model provider, whose default model serves users who choose none.

   • Public organization write-up (Cline release notes on GitHub)
   • No metrics reported

   Team

   Cline maintainers

   Duration

   Argon default from October 16 to October 17, 2026

   Workload

   Default model of a coding agent's own provider

   Product version

   Cline v4.1.24 to v4.1.26 and CLI 3.0.73

   Model

   Gemini 4 Argon, rolled back to Claude Sonnet 5.5

2. GitHubMay 20, 2026GitHub ChangelogGitHub rolled back every Gemini model from Copilot Chat on the web on May 20, 2026, limiting web chat's model list to keep its responses reliable.

   

   GitHub Copilot teamProduct team (GitHub Changelog)GitHub

   GitHub Copilot's model picker across Pro, Pro+, Business and Enterprise plans.

   GitHub Changelog, published May 20, 2026Read the write-up (opens GitHub Changelog in a new tab)

   Observation

   GitHub rolled back every Gemini model from Copilot Chat on the web on May 20, 2026, limiting web chat's model list to keep its responses reliable.

   Takeaway

   Before a team standardizes on a Gemini model inside a third-party tool, check that every surface it works in carries that model; GitHub pulled every Gemini model from web chat in one change.

   What happened

   On May 20, 2026 GitHub removed all Gemini models, with some OpenAI models such as GPT-5.2 Codex, from Copilot Chat on the web, saying it was limiting the list so it could ensure reliable responses. OpenAI and Claude models stayed available across Copilot plans (GitHub Changelog, May 20, 2026).

   “All Gemini models and several other models (e.g., GPT-5.2 Codex and GPT-5.4 nano) have been removed from Copilot Chat on the web.”

   • Public organization write-up (GitHub Changelog)
   • No metrics reported

   Deployment details

   GitHub Copilot's model picker across Pro, Pro+, Business and Enterprise plans.

   • Public organization write-up (GitHub Changelog)
   • No metrics reported

   Team

   GitHub Copilot

   Duration

   Gemini 3.5 Flash on Copilot Chat on the web for one day, May 19 to 20, 2026

   Workload

   Models in the Copilot model picker

   Model

   Gemini 3.5 Flash, later Gemini 3.8 Flash and Gemini 4 Argon

Left it or turned it down1

1. 

   Bruno ŠkvorcStaff Software EngineerLindy

   Lindy, an AI assistant company running managed agents for email, calendar and meeting work.

   Lindy engineering blog, published Jun 24, 2026Read the write-up (opens Lindy engineering blog in a new tab)

   Observation

   Lindy moved off Gemini and Claude Sonnet for most of its managed-agent traffic, to DeepSeek v4 Flash, and inference cost on those routes fell by about 90%.

   Takeaway

   Price every Gemini route against an open model on your own evals; for high-volume assistant traffic Lindy found one that held quality for about a tenth of the cost.

   What happened

   In May and June 2026 Lindy moved most managed-agent traffic from its Gemini and Claude Sonnet paths to DeepSeek v4 Flash on Atlas Cloud, ramping to 100% of target traffic only while offline evals, online evals and retention held. Inference cost on those routes fell by about 90% (Lindy engineering blog, Jun 24, 2026).

   “We moved most managed model traffic from Claude Sonnet and Gemini paths to DeepSeek v4 Flash on Atlas Cloud. Inference costs on those routes fell by about 90%.”

   • Public engineering write-up
   • Cost change reported

   Deployment details

   Lindy, an AI assistant company running managed agents for email, calendar and meeting work.

   • Public engineering write-up
   • Cost change reported

   Team

   Lindy engineering

   Duration

   Ramp over May and June 2026

   Workload

   Most managed-agent model traffic

   Model

   DeepSeek v4 Flash in place of Gemini and Claude Sonnet

   Measurement

   About 90% lower inference cost on the moved routes (Jun 24, 2026)

Switched to or from it1

1. 

   Scott SykoraAuthor of PartForge's post on its default modelPartForge

   PartForge, a chat product that builds CAD parts, with Gemini 3.8 Flash as the default for new chats since September 2026.

   PartForge blog, published Oct 20, 2026Read the write-up (opens PartForge blog in a new tab)

   Observation

   PartForge replaced Gemini 3.8 Flash as its default model with Claude Sonnet 5.5, which finished a typical CAD task in 48 seconds against Flash's 143.

   Takeaway

   Measure time and cost per completed task, retries and tool turns included, before you pick a default model; in PartForge's trial Sonnet 5.5 was faster and cheaper per task, US$0.13 against Flash's US$0.20.

   What happened

   On October 20, 2026 PartForge moved new chats from Gemini 3.8 Flash to Claude Sonnet 5.5 as a trial of a couple of weeks. In its own tests Sonnet 5.5 finished a typical hard CAD task in 48 seconds and Gemini 3.8 Flash in 143 seconds, about three times as long (PartForge blog, Oct 20, 2026).

   “New chats in PartForge now run on Claude Sonnet 5.5, replacing Gemini 3.8 Flash, which has been our default since September.”

   • Public practitioner write-up
   • Task time measured by the team

   Deployment details

   PartForge, a chat product that builds CAD parts, with Gemini 3.8 Flash as the default for new chats since September 2026.

   • Public practitioner write-up
   • Task time measured by the team

   Team

   PartForge (Pixite Inc.)

   Duration

   Gemini 3.8 Flash as default from September to October 20, 2026

   Workload

   Chat that designs CAD parts

   Model

   Gemini 3.8 Flash, replaced by Claude Sonnet 5.5

   Measurement

   Typical hard task: 143 seconds on Gemini 3.8 Flash, 48 seconds on Sonnet 5.5 (Oct 20, 2026)

Run it beside another tool2

1. DoorDashOct 3, 2026InfoQDoorDash runs Gemini beside OpenAI and Anthropic models behind one gateway, and its platform lead names Gemini's one-year model lifetime as a migration cost across hundreds of use cases.

   

   Swaroop ChitlurLead, Generative AI PlatformDoorDash

   DoorDash's GenAI platform team, serving product teams across the company through one LLM Gateway.

   InfoQ, published Oct 3, 2026Watch the talk (opens InfoQ in a new tab)

   Observation

   DoorDash runs Gemini beside OpenAI and Anthropic models behind one gateway, and its platform lead names Gemini's one-year model lifetime as a migration cost across hundreds of use cases.

   Takeaway

   Budget a migration per Gemini model per year, and route Gemini through a gateway where one change moves every caller.

   What happened

   DoorDash started vendor-first, putting OpenAI, Anthropic and Gemini models behind one API so product teams could change models with one line. At DoorDash's scale cost became an issue, and so did what Chitlur calls the deprecation treadmill: each Gemini model shuts down about a year after release, so the platform team has to move hundreds of use cases to a newer model whether or not it helps them (InfoQ talk, Oct 3, 2026). The same gateway gives the Claude API an Amazon Bedrock fallback and OpenAI an Azure one.

   “Similarly, on the model side, we went vendor first. Let's use OpenAI, Anthropic, Gemini models. Folks started getting productive. Folks learned how to do prompt engineering. They were able to scale up. At DoorDash scale, at some point, cost becomes an issue. There were a few other cracks that started to appear. Like the one is the deprecation treadmill. How many of you know that Gemini has a one-year lifetime for models? In one year, the model gets shut down, and this applies to every new model. At the end of that one year, you have to switch to a newer model, whether or not it meets your business needs. I could probably accept that if I was a single product team and I was working on my product, but I as a platform team, I have to think about hundreds of use cases and how I'm going to drive hundreds of teams to migrate to the latest model without necessarily giving them a business argument of the benefits.”

   • Conference talk (InfoQ)
   • No metrics reported

   Deployment details

   DoorDash's GenAI platform team, serving product teams across the company through one LLM Gateway.

   • Conference talk (InfoQ)
   • No metrics reported

   Team

   DoorDash GenAI Platform

   Workload

   Hundreds of product use cases behind one LLM Gateway

   Model

   Gemini, OpenAI and Anthropic models

2. WealthsimpleFeb 6, 2025InfoQWealthsimple added Gemini to its internal LLM gateway beside OpenAI and Cohere for its 1 million token context, then added Amazon Bedrock.

   

   Mandy GuAuthor of InfoQ's article on Wealthsimple's generative AI toolsWealthsimple

   Wealthsimple's internal LLM gateway, built in 2023 for employee productivity tools.

   InfoQ, published Feb 6, 2025Read the write-up (opens InfoQ in a new tab)

   Observation

   Wealthsimple added Gemini to its internal LLM gateway beside OpenAI and Cohere for its 1 million token context, then added Amazon Bedrock.

   Takeaway

   Add Gemini where its context window or price solves a problem the other providers do not, behind an OpenAI-style gateway so callers never change.

   What happened

   Wealthsimple's gateway began with OpenAI and Cohere. Gemini became the next provider because its 1 million token context models promised a way around the context limits the team kept hitting; Amazon Bedrock came after multimodal support. In September 2024 the team shipped v2 of the gateway API to mirror OpenAI's API specification (InfoQ, Feb 6, 2025).

   “Following that, we started expanding the LLM providers that we supported. The catalyst for this was Gemini. Around that time, Gemini had launched their 1-million-token context window models, and we were really interested to see how this could circumvent a lot of our previous challenges with the context window limitations.”

   • Public engineering write-up (InfoQ)
   • No metrics reported

   Deployment details

   Wealthsimple's internal LLM gateway, built in 2023 for employee productivity tools.

   • Public engineering write-up (InfoQ)
   • No metrics reported

   Team

   Wealthsimple machine learning and platform engineering

   Duration

   Gateway since 2023; API v2 in September 2024

   Workload

   Employee productivity tools behind one LLM gateway

   Product version

   Gateway API v2, mirroring OpenAI's specification

   Model

   Gemini 1 million token context models, beside OpenAI, Cohere and Bedrock models

Evaluated it2

1. Major League HackingOct 8, 2026dev.toAt Major League Hacking, a private eval kept Gemini Flash-Lite in production over a hyped rival.

   

   Mike SwiftCEO and co-founderMajor League Hacking

   Product team. benchspec eval runner. Router classifying thousands of checks.

   dev.to, published Oct 8, 2026Read the write-up (opens dev.to in a new tab)

   Observation

   At Major League Hacking, a private eval kept Gemini Flash-Lite in production over a hyped rival.

   Takeaway

   Test a hyped model against the one you run on your own data before switching, and count cost per decision. Per-million-token cost is the wrong measure.

   What happened

   MLH's open-source benchspec runner routes coding-agent checks to code checkers or an LLM judge. MLH compared Flash-Lite with a new decision model over a few thousand runs. Flash-Lite made no dangerous routing calls in the first round, while the rival made three. On harder checks written later, Flash-Lite erred about one run in five. Flash-Lite handles 99% of Swift's own monthly LLM calls. Pairing both models cut measured cost from US$0.52 to US$0.10 per 1,000 checks.

   • Public engineering write-up (also given as an AI Tinkerers NYC talk)
   • Accuracy and cost measured
   • Model stated

   Deployment details

   Product team. benchspec eval runner. Router classifying thousands of checks.

   • Public engineering write-up (also given as an AI Tinkerers NYC talk)
   • Accuracy and cost measured
   • Model stated

   Team

   Major League Hacking

   Workload

   Routing eval checks to code checkers or an LLM judge

   Product version

   Gemini Flash-Lite through the API; test runs billed through OpenRouter

   Model

   Gemini 3.5 Flash-Lite

   Measurement

   Dangerous routing calls, label accuracy, and cost per 1,000 checks

2. PydanticFeb 6, 2025Latent SpacePydantic tests live models on every commit to main, and found Google's Generative Language API failed more often than Vertex AI serving the same models.

   

   Samuel ColvinFounderPydantic

   Pydantic AI, an open-source agent framework that supports every major model provider.

   Latent Space, published Feb 6, 2025Listen to the episode (opens Latent Space in a new tab)

   Observation

   Pydantic tests live models on every commit to main, and found Google's Generative Language API failed more often than Vertex AI serving the same models.

   Takeaway

   Run a few live calls per route in CI; Pydantic tested both Google routes and found the Google Cloud route the more reliable one.

   What happened

   Pydantic runs a handful of tests against live models on every commit to main. Colvin stated that all providers fail more often than expected, that the Generative Language API, the route behind an AI Studio key, was particularly likely to fail, and that Vertex AI, the same API on Google Cloud, was much more reliable (Latent Space podcast, Feb 6, 2025).

   “So like all of the models fail more often than you might expect, but like that one seems to be particularly likely to fail. But Vertex is the same API, but much more reliable.”

   • Podcast episode (Latent Space)
   • No metrics reported

   Deployment details

   Pydantic AI, an open-source agent framework that supports every major model provider.

   • Podcast episode (Latent Space)
   • No metrics reported

   Team

   Pydantic AI maintainers

   Workload

   Live model tests on every commit to main

   Product version

   Generative Language API against Vertex AI

Adopted it2

1. Home AssistantOct 31, 2026Home Assistant AnalyticsHome Assistant's Google Gemini integration runs in 36,533 reporting installations, more than its OpenAI integration's 23,896.

   

   Home AssistantProject analytics (Open Home Foundation)Home Assistant

   The open-source home automation platform, whose users bring their own Gemini API key.

   Home Assistant Analytics, published Oct 31, 2026Read the report (opens Home Assistant Analytics in a new tab)

   Observation

   Home Assistant's Google Gemini integration runs in 36,533 reporting installations, more than its OpenAI integration's 23,896.

   Takeaway

   If your product lets users bring their own model key, support a Gemini key beside an OpenAI one; more Home Assistant installations configure Gemini than OpenAI.

   What happened

   Home Assistant's Google Gemini integration uses a Gemini API key to add a conversation agent, speech-to-text and text-to-speech. Its opt-in analytics, read on October 31, 2026, counted 36,533 installations with the integration configured, against 23,896 for its OpenAI integration, out of 566,955 installations reporting integrations (Home Assistant analytics, read Oct 31, 2026).

   • Public organization write-up (Home Assistant opt-in analytics)
   • Installation count measured

   Deployment details

   The open-source home automation platform, whose users bring their own Gemini API key.

   • Public organization write-up (Home Assistant opt-in analytics)
   • Installation count measured

   Team

   Home Assistant users, each with their own Gemini API key

   Workload

   Voice and chat assistant for home automation

   Product version

   Google Gemini integration (google_generative_ai_conversation)

   Measurement

   36,533 configured installations of 566,955 reporting (read Oct 31, 2026)

2. DEV (Forem)Jan 22, 2026DEV CommunityAt DEV, Gemini handles the moderation calls that need judgment, after cheaper rules run first.

   

   Ben HalpernAuthor of DEV's moderation write-upDEV (Forem)

   Product team. DEV community platform, which uses Gemini for content moderation in production.

   DEV Community, published Jan 22, 2026Read the write-up (opens DEV Community in a new tab)

   Observation

   At DEV, Gemini handles the moderation calls that need judgment, after cheaper rules run first.

   Takeaway

   Put rules in front of the model for the obvious cases, and give the model the context a human moderator would check, such as account history.

   What happened

   DEV's algorithms remove clear spam patterns before Gemini 3 reviews individual posts. The prompt includes account age, badges, and post history and favors avoiding false positives. Labels trigger automated workflows before human moderation. Labeled spam peaked in summer 2025, before full integration, and fell afterward.

   • Public engineering write-up
   • Model stated
   • Trend reported, no rates

   Deployment details

   Product team. DEV community platform, which uses Gemini for content moderation in production.

   • Public engineering write-up
   • Model stated
   • Trend reported, no rates

   Team

   The DEV Team

   Workload

   Labeling posts for spam and quality before human moderation

   Product version

   ContentModerationLabeler service calling Gemini, open source in the Forem repository

   Model

   Gemini 3

   Measurement

   Labeled spam over time