Mo MoustafaFounderOlly
Olly, an AI assistant in iMessage that has carried over 18 million messages, about a third on open models through OpenRouter.
mmoustafa.com, published Read the write-up (opens mmoustafa.com in a new tab)
Observation
OpenRouter's per-host benchmarks, reported by Olly's founder, scored the same DeepSeek weights 90% on GPQA at DeepSeek and 75% at another host, and Olly went down after pinning three hosts with fallbacks off.
- Takeaway
- Choose hosts by the benchmark closest to your workload and keep fallbacks on, because a hard filter also shrinks the pool OpenRouter can fall back to.
- What happened
- Olly routes open models such as DeepSeek V4 Flash through OpenRouter. OpenRouter's per-host board showed first-party DeepSeek at 90% GPQA and 81% TAU-Bench Airline, against 75% and 58% for DigitalOcean on the same weights, and in July one host carried about 20% of Olly's DeepSeek traffic and 92% of its empty completions. Filtering hosts by declared precision did not predict quality. Pinning the model to three hosts with fallbacks off took Olly down within two weeks as those hosts rate limited or stopped serving it.
- Public engineering write-up
- Measured results reported
Deployment details
Olly, an AI assistant in iMessage that has carried over 18 million messages, about a third on open models through OpenRouter.
- Public engineering write-up
- Measured results reported
- Team
- Olly, run by its founder
- Workload
- Over 18 million assistant messages, about a third on open models through OpenRouter
- Model
- DeepSeek V4 Flash 0731 and other open models
- Measurement
- GPQA 90% and TAU 81% first-party against 75% and 58% at DigitalOcean (Sep 7, 2026); 92% of empty completions from one host carrying about 20% of traffic (July 2026)




