LiteLLM+8 Mixed

Operator Insights

What are teams learning from building on LiteLLM and running it in production?

8 accounts: 4 failures, incidents or rollbacks, 1 left it or turned it down and 3 adopted it

Failures, incidents and rollbacks4

Jack WildmanSenior Software EngineerFutureSearch
FutureSearch's research-agent platform, which uses LiteLLM for routing, fallback, retry and accounting and bills users from LiteLLM's spend logs.
FutureSearch blog, published Jun 10, 2026Read the write-up (opens FutureSearch blog in a new tab)
Observation
At FutureSearch, a routine LiteLLM upgrade stopped usage billing for 6.5 days, and no alert fired because spend fell to zero.
Takeaway
Alert on billed spend falling as well as spiking, and read every LiteLLM upgrade's release notes for new defaults that change what reaches your spend logs.
What happened
On June 4, 2026 FutureSearch upgraded LiteLLM from v1.80.5.rc.2 to v1.83.14-stable.patch.3. A new allow_client_tags security setting, on by default, dropped the request tags its billing read, so usage billing stopped without an error (FutureSearch blog, Jun 10, 2026). The incident ran 6.5 days before anyone noticed: the team had alerts for runaway spend and none for spend falling to zero. It added that alert and stayed on LiteLLM at v1.86.4.
“A routine LiteLLM version bump turned off our usage billing for six and a half days. We had many alerts for runaway spend, but not for spend dropping to zero.”
Jack Wildman
- Public engineering write-up
- Incident length reported
Deployment details
FutureSearch's research-agent platform, which uses LiteLLM for routing, fallback, retry and accounting and bills users from LiteLLM's spend logs.
- Public engineering write-up
- Incident length reported
Team
FutureSearch engineering
Duration
6.5 days without billed usage, from June 4, 2026
Workload
Routing, fallback, retry and accounting for a research-agent platform
Product version
LiteLLM v1.80.5.rc.2 to v1.83.14-stable.patch.3, later v1.86.4
Measurement
6.5 days of usage not billed (Jun 10, 2026)

CometMar 24, 2026Comet blogComet audited its repositories and every developer machine within hours of the LiteLLM compromise, and found exposure limited to two CI workflows.
Nimrod LahavAuthor of Comet's incident response write-upComet
Comet, maker of the Opik LLM observability product, which depends on LiteLLM
Comet blog, published Mar 24, 2026Read the write-up (opens Comet blog in a new tab)
Observation
Comet audited its repositories and every developer machine within hours of the LiteLLM compromise, and found exposure limited to two CI workflows.
Takeaway
Know where LiteLLM enters your builds as a transitive dependency, and run production from pinned images so a poisoned release reaches CI at most.
What happened
When litellm 1.82.7 and 1.82.8 shipped credential-stealing malware on March 24, 2026, Comet treated it as a critical incident. It read the pip download logs of GitHub Actions runs across more than 50 repositories and found two CI workflows that installed a compromised version. Only CI test credentials were exposed. It rotated them within hours. Every engineer, product manager, and solutions engineer scanned their Python environments. None was compromised, and the highest version found was 1.82.4. Production and Opik Cloud ran pinned, containerized images and were never exposed.
“We rotated all potentially exposed CI credentials within hours of discovery, without waiting to complete the full investigation.”
Nimrod Lahav
Public engineering write-up
Exposure counts reported
Deployment details
Comet, maker of the Opik LLM observability product, which depends on LiteLLM
Public engineering write-up
Exposure counts reported
Team
Comet engineering, product, and solutions teams
Duration
Same-day response on March 24, 2026
Workload
CI pipelines and developer machines across 50+ repositories
Product version
LiteLLM as a Python dependency of Opik; 1.82.7 and 1.82.8 compromised
Measurement
2 CI workflows exposed, 0 developer environments compromised
PfizerSep 11, 2026LiteLLM blogPfizer's AI platform team load-tests every LiteLLM upgrade in CI, and caught a release that halved gateway throughput with zero HTTP errors.
Aleksandr LiadovAI Platform EngineeringPfizer
Self-hosted LiteLLM gateway serving every team, application, and agent at Pfizer. Five automated test gates per change.
LiteLLM blog, published Sep 11, 2026Read the write-up (opens LiteLLM blog in a new tab)
Observation
Pfizer's AI platform team load-tests every LiteLLM upgrade in CI, and caught a release that halved gateway throughput with zero HTTP errors.
Takeaway
Treat each gateway upgrade as a change to every AI product at once: load-test it against fixed baselines, because the worst regressions return HTTP 200.
What happened
Pfizer runs LiteLLM as one OpenAI-compatible entry point to external and self-hosted models, so every LiteLLM bump runs through load and regression tests in CI first. Upgrading to v1.89.2 dropped CI throughput from about 300 to 156 requests per second with an unchanged error rate. Bisecting by configuration isolated a long-standing Redis bug: an ssl: false setting still opened TLS connections that stalled until timeout. After the fix, median CI latency fell from 5,000 ms to 1,200 ms, and Pfizer shared its load-test configuration for LiteLLM's own CI.
“A LiteLLM version bump exposed a long-standing Redis configuration bug that cut Pfizer's gateway throughput by ~48%, with zero HTTP errors in the application logs.”
Ishaan Jaffer, Krrish Dholakia (LiteLLM) with Pfizer AI Platform Engineering
Vendor customer story (LiteLLM blog, co-written with Pfizer AI Platform Engineering)
Team metrics reported
Deployment details
Self-hosted LiteLLM gateway serving every team, application, and agent at Pfizer. Five automated test gates per change.
Vendor customer story (LiteLLM blog, co-written with Pfizer AI Platform Engineering)
Team metrics reported
Team
Pfizer AI Platform Engineering
Workload
Company-wide gateway for chat sessions and agent loops
Product version
LiteLLM v1.85.0 to v1.92.0 with Redis caching; fix shipped in v1.93.0
Measurement
CI throughput about 300 to 156 RPS on v1.89.2; median CI latency 5,000 ms to 1,200 ms after the fix
FutureSearchMar 24, 2026FutureSearch blogAt FutureSearch, the compromised litellm 1.82.8 arrived through an MCP plugin inside Cursor and crashed an engineer's machine, which is how the team found the attack.
Callum McMahonResearch ScientistFutureSearch
A FutureSearch engineer's laptop, where an MCP plugin running inside Cursor pulled litellm in as a transitive dependency.
FutureSearch blog, published Mar 24, 2026Read the write-up (opens FutureSearch blog in a new tab)
Observation
At FutureSearch, the compromised litellm 1.82.8 arrived through an MCP plugin inside Cursor and crashed an engineer's machine, which is how the team found the attack.
Takeaway
Inventory where LiteLLM arrives as a dependency of other tools, editor plugins included, since the payload runs without any of your code importing it.
What happened
On March 24, 2026, litellm 1.82.8 reached the machine as a transitive dependency of an MCP plugin running in Cursor. Its .pth launcher ran at every Python start and spawned processes until the machine crashed, which led FutureSearch to find and first report the PyPI compromise (FutureSearch blog, Mar 24, 2026).
“We discovered it when the package was pulled in as a transitive dependency by an MCP plugin running inside Cursor.”
Callum McMahon
Public engineering write-up
No metrics reported
Deployment details
A FutureSearch engineer's laptop, where an MCP plugin running inside Cursor pulled litellm in as a transitive dependency.
Public engineering write-up
No metrics reported
Team
FutureSearch engineering
Duration
Found the same day, March 24, 2026
Workload
Developer laptop running Cursor with an MCP plugin
Product version
litellm 1.82.8 from PyPI, a transitive dependency

Left it or turned it down1

meffmaddMember of the TU Vienna team, author of the Aqueduct gatewayTU Wien
A small TU Vienna team providing an OpenAI-compatible API to university researchers.
Hacker News, published Jul 1, 2025Read the write-up (opens Hacker News in a new tab)
Observation
A TU Vienna team rejected LiteLLM's proxy, Portkey and Kong over enterprise-gated SSO and observability, and built its own gateway on LiteLLM's router library.
Takeaway
Write down the features you need without an enterprise license, SSO and per-organization management first, before a pilot; a small team may find building on the open library cheaper.
What happened
Looking for a gateway for researchers, the team found that every option it evaluated, LiteLLM and Portkey among them, locked SSO, organization management, observability or self-hosting behind enterprise plans (Hacker News, Jul 1, 2025). It rejected them and built Aqueduct, an open-source gateway on the LiteLLM Router SDK.
“Every AI gateway we looked at (LiteLLM, Portkey, Kong, etc.) locked critical features like SSO, organization management, observability, or self-hosting behind enterprise plans.”
meffmadd
- Public practitioner write-up (Show HN post)
- No metrics reported
Deployment details
A small TU Vienna team providing an OpenAI-compatible API to university researchers.
- Public practitioner write-up (Show HN post)
- No metrics reported
Team
Small TU Vienna team serving university researchers
Workload
OpenAI-compatible API for researchers
Product version
LiteLLM proxy evaluated; Aqueduct built on the LiteLLM Router SDK

Adopted it3

Bartosz OcytkoExecutive Principal EngineerZalando
Zalando's company LLM proxy, run by its ML platform team in front of OpenAI, AWS Bedrock and Google Vertex models.
Zalando Engineering Blog, published Aug 14, 2026Read the write-up (opens Zalando Engineering Blog in a new tab)
Observation
Zalando's ML platform team has run a LiteLLM proxy as the company's model gateway since January 2024, with call hooks for cost tracking and client upgrades.
Takeaway
Use the proxy's call hooks for the controls only your company needs, and schedule restarts until memory growth is fixed.
What happened
Zalando's ML platform team deployed a LiteLLM-based proxy in January 2024 that now serves OpenAI, AWS Bedrock and Google Vertex models (Zalando Engineering Blog, Aug 14, 2026). Post-call hooks track cost anonymously, and pre-call hooks block outdated clients by their User-Agent header. The team values that extensibility and reports stability problems and memory leaks that force a restart every 20,000 requests, with the proxy serving about 2,000 monthly users on six small pods.
“We use post-call hooks for anonymized cost tracking and pre-call hooks for enforcing client version upgrades by restricting access to the proxy based on the User-Agent header.”
Bartosz Ocytko
- Public engineering write-up
- Usage reported
Deployment details
Zalando's company LLM proxy, run by its ML platform team in front of OpenAI, AWS Bedrock and Google Vertex models.
- Public engineering write-up
- Usage reported
Team
Zalando ML platform team
Duration
Since January 2024
Workload
Company LLM gateway for about 2,000 monthly users
Product version
LiteLLM proxy with post-call and pre-call hooks
Model
OpenAI, AWS Bedrock and Google Vertex models
Measurement
A restart every 20,000 requests; six small pods (Aug 14, 2026)

OktaFeb 26, 2026Okta blogOkta runs LiteLLM in its private cloud as the single entry point for every employee and developer AI call, and logs every interaction for its security team.
Dennis HenryProductivity ArchitectOkta
Okta's company-wide AI gateway, deployed in its private cloud for developers calling model APIs and employees using an internal web UI.
Okta blog, published Feb 26, 2026Read the write-up (opens Okta blog in a new tab)
Observation
Okta runs LiteLLM in its private cloud as the single entry point for every employee and developer AI call, and logs every interaction for its security team.
Takeaway
Put one self-hosted gateway in front of every model call before AI use spreads, so security reviews a single log that covers every team.
What happened
Okta built its AI gateway on LiteLLM inside its private cloud as a secure proxy and single point of entry for every AI interaction, from code-generation API calls to its internal chat UI (Okta blog, Feb 26, 2026). Routing all model traffic through it gives Okta a full view of AI use: every interaction is logged and sent to its Defensive Cyber Operations team for continuous auditing.
“By centralizing all LLM traffic, we are able to gain 100% visibility into AI usage. Every single interaction meticulously logged and routed to our Defensive Cyber Operations (DCO) team for continuous auditing.”
Dennis Henry
Public organization write-up
No metrics reported
Deployment details
Okta's company-wide AI gateway, deployed in its private cloud for developers calling model APIs and employees using an internal web UI.
Public organization write-up
No metrics reported
Team
Okta, company-wide
Workload
Every employee and developer AI interaction, from code-generation APIs to an internal web UI
Product version
LiteLLM gateway in Okta's private cloud
raphikiSep 9, 2026DEV Community (Technology at Worldline)Worldline uses LiteLLM in production, and a Worldline technologist chose it as the governance and routing proxy for a local-first coding agent.
raphikiDevRel and architect at Worldline by his dev.to profile, writing for Technology at WorldlineWorldline
A local-first coding-agent setup built with open tools and routed through LiteLLM.
DEV Community (Technology at Worldline), published Sep 9, 2026Read the write-up (opens DEV Community (Technology at Worldline) in a new tab)
Observation
Worldline uses LiteLLM in production, and a Worldline technologist chose it as the governance and routing proxy for a local-first coding agent.
Takeaway
A gateway already approved in production is the cheapest place to route new agent tools; reuse it before adopting another.
What happened
Writing on Worldline's technology publication on DEV Community (Sep 9, 2026), the author chose LiteLLM as the proxy for governance and adaptive routing in a local-first coding-agent build, and says Worldline also uses LiteLLM in production at work. The post states no scale or team.
“I selected LiteLLM, which we also use in production at work.”
raphiki
Public practitioner write-up
No metrics reported
Deployment details
A local-first coding-agent setup built with open tools and routed through LiteLLM.
Public practitioner write-up
No metrics reported
Workload
Governance and routing proxy for a local-first coding agent
Product version
LiteLLM proxy

Failures, incidents and rollbacks4

At FutureSearch, a routine LiteLLM upgrade stopped usage billing for 6.5 days, and no alert fired because spend fell to zero.

Comet audited its repositories and every developer machine within hours of the LiteLLM compromise, and found exposure limited to two CI workflows.

Pfizer's AI platform team load-tests every LiteLLM upgrade in CI, and caught a release that halved gateway throughput with zero HTTP errors.

At FutureSearch, the compromised litellm 1.82.8 arrived through an MCP plugin inside Cursor and crashed an engineer's machine, which is how the team found the attack.

Left it or turned it down1

A TU Vienna team rejected LiteLLM's proxy, Portkey and Kong over enterprise-gated SSO and observability, and built its own gateway on LiteLLM's router library.

Adopted it3

Zalando's ML platform team has run a LiteLLM proxy as the company's model gateway since January 2024, with call hooks for cost tracking and client upgrades.

Okta runs LiteLLM in its private cloud as the single entry point for every employee and developer AI call, and logs every interaction for its security team.

Worldline uses LiteLLM in production, and a Worldline technologist chose it as the governance and routing proxy for a local-first coding agent.