GILD verdict
Adopt LiteLLM when your platform team needs one self-hosted door to many model providers, as Okta and Zalando run it, with keys and budgets per person, team, and app at no license cost.
It fits only teams ready to run Postgres and Redis, upgrade weekly and patch quickly, and to alert when spend drops to zero, since one upgrade stopped FutureSearch's billing for 6.5 days.
LiteLLM
AI Gateway by BerriAI
+8 MixedFit and limits
Who it fits
- You want one company gateway that logs every model call for security and cost tracking, as Okta runs in its private cloud and Zalando's ML platform team has run since January 2024.
- Prompts and provider keys must stay in your own network, and legal requires model use through your own provider contracts.
- Many engineers and apps need model access, and you want each one on its own virtual key with a budget and rate limit, so nobody shares a provider key.
- Coding tools such as Claude Code, Codex, OpenCode, Cursor, and Aider should reach every approved model through one endpoint whose spend you can see.
- You call several providers, including Bedrock, Vertex AI, Azure, and self-hosted vLLM, and want fallbacks and load balancing across them without changing app code.
Where it falls short
- You operate it: Postgres for keys and spend, Redis across instances, sizing at 1 vCPU and 4 GiB per pod, and weekly releases, with patches only for the four latest minor lines.
- Practitioners report regressions after upgrades, memory growth, and latency. BerriAI's own July 2026 benchmark measured 257.7 ms of added p99 latency on the Python proxy against 0.7 ms on its Rust beta. Agentgateway, a competing gateway, measured LiteLLM 1.98.0 in Rust mode at 71.451 ms p99 and 2.15 GiB of memory under maximum load in August 2026. Rust mode did not serve the OpenAI API path in that test.
- Upgrades change defaults without an error. A new security setting in a June 2026 release stopped FutureSearch's usage billing for 6.5 days, Pfizer caught a release that halved throughput, and Zalando restarts its proxy every 20,000 requests over memory leaks.
- SSO beyond 5 users, SCIM, audit logs, tag budgets, and several guardrail integrations need an Enterprise license, which BerriAI quotes by annual gateway request capacity, deployment architecture, and support needs. Standard support runs 9am to 9pm PST on weekdays with no guaranteed response time; round-the-clock SLAs cost extra.
- GitHub lists 44 reviewed advisories for the package as of October 28, 2026, 7 of them critical, and CISA lists three LiteLLM CVEs as exploited. Two PyPI releases carried malware on March 24, 2026. On October 21, 2026, the 1.107.0 release on PyPI and four official Docker tags carried a credential stealer. BerriAI's own repository published 6 advisories in April 2026 and 2 a month in August and September.
Implementation reality
Install a pinned image by digest
BerriAI states the official Docker image was not affected by the March 2026 PyPI compromise because it pins dependencies. On October 21, 2026, its own pipeline signed and published malicious 1.107.0 images under the v1.107.0, latest, main-latest, and main-stable tags, so a cosign check alone passes them. Pin 1.106.5 or another release with the GHSA-w4c7 fix by digest, and use no floating tag. Check that the image provenance names BerriAI/litellm as its source repository. Set a long, random master key and salt key before first start.
Give budgets a database and make them fail closed
LiteLLM enforces budgets against spend in Postgres. Without a database, it skips global max_budget and keeps accepting requests. Keep budget reservation on, and set fail_closed_budget_enforcement and fail_closed_rate_limit_enforcement so a Redis outage rejects requests and doesn't multiply limits by the number of instances.
Issue keys by team and person
Create organizations and teams, give each person or app a virtual key with a max_budget and a budget_duration, and restrict the models each key may call. Point coding tools at the gateway with these keys, and keep the master key out of every client.
Gate upgrades on a load test
LiteLLM ships a stable minor every week and supports the last four. Pin a line, take patches, and load-test each move to a new line against a fixed baseline before production: Pfizer's CI caught a release that halved throughput with no HTTP errors.
What to measure in your pilot
- Mirror a week of production traffic through Rust mode and compare p99 latency and memory with the Python proxy; switch once it serves every API path you call within your latency budget.
- Log response time on every support ticket for the first quarter; if a partial production outage waits past 6 hours, buy the 24/7 SLA.
- Track new GitHub advisories and your own patch lag monthly; if a critical fix takes you longer than 72 hours to deploy, move to a hosted gateway.
Sources
- Official documentation
- 18
- Public source
- 3
- LiteLLM pricing
- LiteLLM docs: Enterprise
- LiteLLM docs: Support and SLA
- LiteLLM docs: Version support
- LiteLLM docs: Release cycle
- LiteLLM docs: Production best practices
- LiteLLM docs: Budgets and rate limits
- LiteLLM docs: Spend tracking
- LiteLLM docs: SSO for the Admin UI
- LiteLLM docs: Data privacy and security
- LiteLLM docs: Compliance and SOC 2 Type II
- LiteLLM docs: Docker image security
- LiteLLM blog: Security update, suspected supply chain incident (Mar 24, 2026)
- LiteLLM blog: LiteLLM release v1.107.0 compromised (Oct 21, 2026)
- LiteLLM blog: The attack chain behind the v1.107.0 LiteLLM release (Oct 23, 2026)
- BerriAI/litellm advisory GHSA-w4c7-34wj-hqm8: unauthenticated file read leaks master key on Enterprise proxy, patched in 1.106.5 (Oct 27, 2026)
- LiteLLM blog: Benchmarking the LiteLLM Rust AI Gateway (Jul 22, 2026)
- BerriAI/litellm LICENSE
- LiteLLM AI Gateway home page (providers, models, adoption figures)
- Lin Sun, agentgateway blog: Benchmarking Agentgateway vs LiteLLM's Rust Mode (Aug 17, 2026)
- GitHub: BerriAI/litellm repository security advisories (Oct 1, 2026)