Operator Insights

Failures, incidents and rollbacks3

1. 

   Jian GuoKION GROUP ITKION GROUP

   KION GROUP IT's production Open WebUI on Kubernetes with Helm on GKE, Redis Sentinel and websockets, for about 40,000 users.

   GitHub (open-webui/open-webui), published Jul 20, 2026Read the issue (opens GitHub (open-webui/open-webui) in a new tab)

   Observation

   KION GROUP's 40,000-user Open WebUI deployment lost pods when Redis Sentinel lookups slowed: health checks timed out and Kubernetes killed the containers with no chat load.

   Takeaway

   At company scale, load-test the Redis Sentinel path and watch connection churn, because a slow lookup can take down healthy pods through the health check.

   What happened

   Open WebUI v0.10.2 opened a new Redis connection for each operation through Sentinel, about 6.9 a second or some 25,000 an hour. When Sentinel resolution slowed, the timeouts blocked the event loop, the /health endpoint timed out, and Kubernetes killed the pod with exit code 137, an outage while Redis itself stayed healthy with no restarts or rejected connections. The report proposes caching the master client and reusing it across calls.

   • Public engineering write-up (a GitHub issue filed from KION GROUP IT)
   • Connection rates measured

   Deployment details

   KION GROUP IT's production Open WebUI on Kubernetes with Helm on GKE, Redis Sentinel and websockets, for about 40,000 users.

   • Public engineering write-up (a GitHub issue filed from KION GROUP IT)
   • Connection rates measured

   Team

   KION GROUP IT

   Workload

   Company-wide chat for about 40,000 users, websockets enabled

   Product version

   Open WebUI v0.10.2 with redis-py 8.0.0

   Measurement

   About 6.9 new Redis connections a second, some 25,000 an hour (Jul 20, 2026)

1. MikrotikJul 9, 2026GitHub (open-webui/open-webui)Mikrotik hit three OAuth configuration bugs in production while setting up Keycloak SSO for Open WebUI, and the sign-in button never appeared.

   

   Dāvis PazarsAI EngineerMikrotik

   Mikrotik's Open WebUI v0.10.2 deployment, signing users in through Keycloak over OAuth.

   GitHub (open-webui/open-webui), published Jul 9, 2026Read the issue (opens GitHub (open-webui/open-webui) in a new tab)

   Observation

   Mikrotik hit three OAuth configuration bugs in production while setting up Keycloak SSO for Open WebUI, and the sign-in button never appeared.

   Takeaway

   Configure SSO through environment variables only, and test sign-in after a restart before you announce it, because admin-panel and stored OAuth settings can silently disagree.

   What happened

   OAuth settings saved in the admin panel were lost on restart with persistent configuration off. With ENABLE_OAUTH_PERSISTENT_CONFIG on, they persisted but never registered a provider, so no SSO button appeared. Turning the flag on later let stale stored OAuth rows override live environment variables, and OAuth sign-up failed. The engineer reports hitting all three in production while setting up Keycloak SSO.

   • Public engineering write-up (a GitHub issue by Mikrotik's AI engineer)

   Deployment details

   Mikrotik's Open WebUI v0.10.2 deployment, signing users in through Keycloak over OAuth.

   • Public engineering write-up (a GitHub issue by Mikrotik's AI engineer)

   Team

   Mikrotik

   Workload

   Single sign-on to Open WebUI through Keycloak

   Product version

   Open WebUI v0.10.2

2. UMBCApr 24, 2026GitHub (open-webui/open-webui)At UMBC, Open WebUI's title, tag, and follow-up tasks failed for every chat on a user's own direct connection, so the configured task model never ran.

   

   Tim ChampUMBCUMBC

   UMBC's Open WebUI v0.9.2 on AWS ECS Fargate, with Ollama off and users bringing their own model keys through direct connections.

   GitHub (open-webui/open-webui), published Apr 24, 2026Read the issue (opens GitHub (open-webui/open-webui) in a new tab)

   Observation

   At UMBC, Open WebUI's title, tag, and follow-up tasks failed for every chat on a user's own direct connection, so the configured task model never ran.

   Takeaway

   Test background tasks such as titles and tags on every connection type you allow, because a task model on the server does not reach chats on users' direct connections.

   What happened

   UMBC set Amazon Nova Micro on an Amazon Bedrock endpoint as the server's task model and let users chat on their own keys through direct connections. For those chats, the title, tag, follow-up, and other task endpoints returned 404 MODEL_NOT_FOUND, so the task model never ran and the features failed.

   • Public engineering write-up (a GitHub issue filed from UMBC)

   Deployment details

   UMBC's Open WebUI v0.9.2 on AWS ECS Fargate, with Ollama off and users bringing their own model keys through direct connections.

   • Public engineering write-up (a GitHub issue filed from UMBC)

   Team

   UMBC

   Workload

   Campus chat with users' own model keys through direct connections

   Product version

   Open WebUI v0.9.2 on Docker, AWS ECS Fargate, ARM64

   Model

   Amazon Nova Micro on Amazon Bedrock as the task model

Adopted it1

1. 

   Marios AdamidisPostgraduate researcherFORTH-IESL

   Professional team. Laser and physics research lab in Greece. Repository created June 25, 2025 and shown on Hacker News February 22, 2026.

   GitHub, published Jun 25, 2025Read the write-up (opens GitHub in a new tab)

   Observation

   A physics lab at FORTH-IESL runs its research platform on Open WebUI with custom MCP tool servers for 15 researchers.

   Takeaway

   Self-host Open WebUI where data must stay in the lab, and extend it with MCP tool servers for the instruments and files researchers use.

   What happened

   Adamidis built FORTHought, a locally hosted research platform that runs on Open WebUI with custom MCP tool servers and serves 15 active researchers across spectroscopy, electron microscopy and other groups in the lab.

   “It runs on Open WebUI and serves 15 active researchers across spectroscopy, electron microscopy…”

   • Public engineering write-up (the researcher's GitHub repository)
   • User count stated by the author

   Deployment details

   Professional team. Laser and physics research lab in Greece. Repository created June 25, 2025 and shown on Hacker News February 22, 2026.

   • Public engineering write-up (the researcher's GitHub repository)
   • User count stated by the author

   Team

   15 active researchers

   Workload

   Research assistance with lab tools and documents

   Product version

   Open WebUI with custom MCP tool servers, self-hosted

   Measurement

   Active researchers (15)