OpenAI API+3 Mixed

Operator Insights

What are teams learning from building on OpenAI API and running it in production?

9 accounts: 3 switched, 2 run it beside another tool, 1 evaluated it and 3 adopted it

Switched to or from it3

Lorenzo GentileAuthor of Ploy's engineering write-upPloy
Product engineering. Website-building agent. Default model moved from Claude Opus 4.8 to GPT-5.6 Sol.
Ploy blog, published Jul 9, 2026Read the write-up (opens Ploy blog in a new tab)
Observation
At Ploy, GPT-5.6 Sol beat Claude Opus in production evals once the team rewrote its tool schemas and cache keys.
Takeaway
Fix the eval harness and the cache configuration for the new provider before comparing models on cost. Ploy's first run showed GPT-5.6 about 50% more expensive than Opus, and the whole gap was cache keys.
What happened
Ploy's agent plans, builds, and edits marketing websites, and the team runs every frontier release against hundreds of fixture cases. On its redesign suite, a completed build on GPT-5.6 Sol averaged US$2.22 and 3 minutes 42 seconds, against US$3.06 and 8 minutes on Claude Opus 4.8. Its visual score was 0.970 against 0.936. Three fixes came first. About a third of the first run's failures came from an eval harness sized for Opus. GPT-5.6 filled all 25 optional tool parameters with invented values, which emptied 52% to 64% of its file reads until the team made optional properties required but nullable. A per-conversation cache key cached none of the shared 29K-token prefix; a workspace-scoped prompt_cache_key raised first-call cache hits to 83.7%.
“As of today, Ploy’s agent runs on GPT-5.6 Sol, the flagship tier of the model family OpenAI released this morning. For months, no model met our bar for replacing Claude Opus. GPT-5.6 Sol did. After a head-to-head evaluation, we made it the default model for every Ploy workspace.”
Lorenzo Gentile
- Public engineering write-up
- Cost, build time, and visual score reported per completed build
- Measured on 11 Opus builds and 10 GPT-5.6 builds
Deployment details
Product engineering. Website-building agent. Default model moved from Claude Opus 4.8 to GPT-5.6 Sol.
- Public engineering write-up
- Cost, build time, and visual score reported per completed build
- Measured on 11 Opus builds and 10 GPT-5.6 builds
Team
Ploy design and engineering
Workload
Building and editing production marketing websites
Product version
Responses API through Vercel's AI SDK, with a workspace-scoped prompt_cache_key
Model
GPT-5.6 Sol, replacing Claude Opus 4.8
Measurement
Mean per completed build: US$2.22 against US$3.06, 3m 42s against 8m 00s

AmpSep 28, 2026Amp NewsAmp replaced GPT-5.6 Sol with Claude Opus 5.5 in its default medium mode after its evals, and kept GPT-6 Astra behind its High mode.
Amp teamAmp product newsAmp
Amp, a coding agent that routes each of its modes to a model from a different lab.
Amp News, published Sep 28, 2026Read the write-up (opens Amp News in a new tab)
Observation
Amp replaced GPT-5.6 Sol with Claude Opus 5.5 in its default medium mode after its evals, and kept GPT-6 Astra behind its High mode.
Takeaway
Assign each mode or task to whichever model wins your own evals that month, and keep OpenAI as one swappable provider among several.
What happened
On Sep 28, 2026 Amp made Claude Opus 5.5 the model behind its default medium mode, replacing GPT-5.6 Sol: in Amp's internal evals Opus 5.5 solved 65% of tasks against 61% for GPT-5.6 Sol, at 10% lower cost than GPT-5.6 Sol (Amp-reported). A day later Amp's news post said GPT-6 Astra powers its default High mode, with a faster Plaid speed tier for modes that use it (Sep 29, 2026). The posts do not say whether Amp reaches OpenAI's models through OpenAI's own API.
“Claude Opus 5.5 now powers Amp's medium mode by default, replacing GPT-5.6 Sol.”
Amp team
Public organization write-up
Internal eval results reported by Amp
Deployment details
Amp, a coding agent that routes each of its modes to a model from a different lab.
Public organization write-up
Internal eval results reported by Amp
Team
Amp
Workload
Coding agent modes: medium mode and High mode
Model
Claude Opus 5.5 in place of GPT-5.6 Sol for medium mode; GPT-6 Astra for High mode
Measurement
65% of internal eval tasks solved against 61% for GPT-5.6 Sol (Sep 28, 2026)
Simon WillisonSep 22, 2026Simon Willison's WeblogSimon Willison switched the public Datasette Agent to GPT-6 Luna on its launch day and found it fast and competent at SQL.
Simon WillisonIndependent developer, creator of DatasetteDatasette
The public Datasette Agent at agent.datasette.io, which writes SQL and builds Datasette Apps.
Simon Willison's Weblog, published Sep 22, 2026Read the write-up (opens Simon Willison's Weblog in a new tab)
Observation
Simon Willison switched the public Datasette Agent to GPT-6 Luna on its launch day and found it fast and competent at SQL.
Takeaway
Try the smallest current model on scoped tasks such as SQL generation before paying for a larger one.
What happened
On Sep 22, 2026 Willison moved the public Datasette Agent to GPT-6 Luna and reported it fast and competent at SQL queries and at building HTML and JavaScript for Datasette Apps (Simon Willison's Weblog, Sep 22, 2026). Luna lists at US$0.10 input and US$0.50 output per million tokens on OpenAI's pricing page (checked Oct 28, 2026).
Public practitioner write-up
No metrics reported
Deployment details
The public Datasette Agent at agent.datasette.io, which writes SQL and builds Datasette Apps.
Public practitioner write-up
No metrics reported
Team
Datasette (one maintainer)
Duration
Since Sep 22, 2026
Workload
SQL queries and HTML and JavaScript for Datasette Apps
Model
GPT-6 Luna

Run it beside another tool2

Siddharth KodwaniSoftware Engineer, GenAI PlatformDoorDash
DoorDash's GenAI platform team, serving product teams across the company through one LLM Gateway.
InfoQ, published Oct 3, 2026Watch the talk (opens InfoQ in a new tab)
Observation
DoorDash runs the OpenAI API with Azure OpenAI as its fallback, behind one gateway that attributes usage costs to each department.
Takeaway
Set up a second route to the same OpenAI models before you need it, and put cost attribution in your gateway from the first team onward.
What happened
DoorDash put OpenAI, Anthropic and Gemini models behind one API and SDK so a product team can try a new model with one change. For reliability each provider got a fallback: OpenAI's endpoint falls back to Azure, and the Claude API to Amazon Bedrock (InfoQ, Oct 3, 2026). As usage grew, finance asked who was spending what on shared keys, so the gateway added workspaces that attribute cost to each department. The platform lead also names rising cost at scale and model deprecations that force migrations across hundreds of use cases.
“We actually had to go understand what the fallback should be, what kind of routing should be. At one point, I remember we had OpenAI endpoint, but we also had a fallback on Azure endpoint. Or, we had Claude model API endpoint, then we had a fallback on Bedrock, so that the services can rely on us from the reliability point of view. We went in that direction. We quickly realized that as the usage is going up, then we need a way to track, because now finance got involved. Like, you guys have one API key for multiple teams, how do I track who is using what? We quickly realized we need workspaces where we have cost attribution. We can actually drill down on who is using how much. We can attribute cost back to those departments. That's what basically we did with LLM Gateway.”
Siddharth Kodwani
- Conference talk (InfoQ)
- No metrics reported
Deployment details
DoorDash's GenAI platform team, serving product teams across the company through one LLM Gateway.
- Conference talk (InfoQ)
- No metrics reported
Team
DoorDash GenAI Platform
Workload
Hundreds of product use cases behind one LLM Gateway
Product version
OpenAI API with an Azure OpenAI fallback; Claude API with an Amazon Bedrock fallback
Model
OpenAI, Anthropic and Gemini models

WealthsimpleFeb 6, 2025InfoQWealthsimple's LLM gateway started with OpenAI and Cohere, added Gemini and Amazon Bedrock beside them, then rebuilt its API to mirror OpenAI's specification.
Mandy GuAuthor of the InfoQ article on Wealthsimple's platformWealthsimple
Wealthsimple's internal LLM gateway, used by teams across the company.
InfoQ, published Feb 6, 2025Read the write-up (opens InfoQ in a new tab)
Observation
Wealthsimple's LLM gateway started with OpenAI and Cohere, added Gemini and Amazon Bedrock beside them, then rebuilt its API to mirror OpenAI's specification.
Takeaway
Code your gateway against OpenAI's request shape so teams can swap providers behind it, and add providers as specific limits, such as context length, call for them.
What happened
Wealthsimple's gateway began with OpenAI and Cohere. It added Gemini, whose 1-million-token context window promised to get past earlier context limits, and then Amazon Bedrock. In September 2024 the team shipped v2 of the gateway API, which mirrors OpenAI's API specification, after integration friction with its own interface (InfoQ, Feb 6, 2025).
“We took some time in September of 2024 to ship v2 of our API, which did mirror OpenAI's API specs. We learned that as the GenAI industry matures, it's important to think about what the right standards and integrations are.”
Mandy Gu
Public engineering write-up (InfoQ)
No metrics reported
Deployment details
Wealthsimple's internal LLM gateway, used by teams across the company.
Public engineering write-up (InfoQ)
No metrics reported
Team
Wealthsimple engineering
Duration
2023 to September 2024, as described
Workload
Company LLM gateway for internal tools
Product version
Gateway v2 mirroring OpenAI's API specification (September 2024)
Model
OpenAI and Cohere, then Gemini and Amazon Bedrock models

Evaluated it1

Daniel FurmanReplit engineering, co-author of the write-upReplit
Replit Agent in production, one week per model across August and September 2026.
Replit blog, published Sep 29, 2026Read the write-up (opens Replit blog in a new tab)
Observation
Replit tested GPT-6 Astra as Replit Agent's core loop for a production week and found it delegates to cheaper workers on its own.
Takeaway
Trial a new model on real production traffic for a fixed window and read the traces, since delegation habits change cost more than list price does.
What happened
Replit ran Replit Agent's production traffic on one model per week: Fable 5 in August 2026, then Fable 5.1 and GPT-6 Astra in September. In its Astra week the agent dispatched subagents on 36% of turns and handed work to general workers on 20% of turns, by Replit's own production traces (Replit blog, Sep 29, 2026).
“Astra is the first model we’ve seen routinely delegate to general workers without being told to…”
Daniel Furman
- Public engineering write-up
- Production traces, one week per model
Deployment details
Replit Agent in production, one week per model across August and September 2026.
- Public engineering write-up
- Production traces, one week per model
Team
Replit Agent engineering
Duration
One production week per model, August and September 2026
Workload
Replit Agent's core agent loop
Model
GPT-6 Astra, against Fable 5 and Fable 5.1
Measurement
Subagents dispatched on 36% of turns; work handed to general workers on 20% (Sep 2026)

Adopted it3

AsanaOct 8, 2026Inside AsanaAt Asana, keeping a browser agent's cached history intact made each run far cheaper on GPT-6.1 Sol.
Frank HidalgoCTO of StackAIAsana
Product engineering. StackAI browser agent at Asana, tested across four models.
Inside Asana, published Oct 8, 2026Read the write-up (opens Inside Asana in a new tab)
Observation
At Asana, keeping a browser agent's cached history intact made each run far cheaper on GPT-6.1 Sol.
Takeaway
A prompt cache pays only when the start of each request stays the same, so stop editing history on every call, and test caching policies on each candidate model before choosing one.
What happened
Asana's StackAI browser agent cached its tools and system prompt but not its growing history. It also trimmed text and dropped a screenshot on almost every call, so the cache broke on nearly every request. Hidalgo had GPT-6 Astra in Codex run a 144-run study of six caching and history policies on GPT-6.1 Sol and three other frontier models. The fix cached the history, let up to 20 screenshots build up before pruning, and raised the history budget from 120,000 to 480,000 characters. On GPT-6.1 Sol the agent then cost about US$0.47 per run, 76x less and 5x faster than the original production setup on another lab's model. It read 89% of input from cache. Every best-condition run found all 192 facts it was asked for.
“On GPT-6.1 Sol, the same agent cost 76x less and ran 5x faster, reading 89% of its input from cache.”
Frank Hidalgo
Public engineering write-up
Costs read from each provider's token counters
Three runs per condition
Deployment details
Product engineering. StackAI browser agent at Asana, tested across four models.
Public engineering write-up
Costs read from each provider's token counters
Three runs per condition
Team
Asana StackAI engineering, with GPT-6 Astra in Codex running the study
Duration
About a week for the study
Workload
Browser agent collecting six fields for each of 32 books from a public catalog
Product version
GPT-6.1 Sol through the API, with a cache marker on the latest tool result
Model
GPT-6.1 Sol, against three models from another lab
Measurement
US$0.47 per run, 76x cheaper and 5x faster than the original setup
incident.ioJan 17, 2024incident.io blogincident.io built suggested incident summaries on OpenAI's models and shipped them to every customer; 63% of actioned suggestions were accepted as written.
Milly LeadleyEngineerincident.io
incident.io's incident management product, its first AI feature.
incident.io blog, published Jan 17, 2024Read the write-up (opens incident.io blog in a new tab)
Observation
incident.io built suggested incident summaries on OpenAI's models and shipped them to every customer; 63% of actioned suggestions were accepted as written.
Takeaway
Ship a first AI feature as a suggestion with accept and edit buttons, and count acceptances as the quality measure.
What happened
incident.io used OpenAI's models with JSON mode to suggest incident summaries and launched the feature to all customers in November 2023. Users acted on suggestions 20% of the time; of those, 63% were accepted as written and 26% edited, by incident.io's in-product feedback (incident.io blog, Jan 17, 2024).
“We launched it to all our customers in November, and the reaction was better than we could have hoped for.”
Milly Leadley
Public engineering write-up
Self-reported product telemetry
Deployment details
incident.io's incident management product, its first AI feature.
Public engineering write-up
Self-reported product telemetry
Team
incident.io product engineering
Duration
Since November 2023
Workload
Suggested incident summaries
Product version
OpenAI API with JSON mode
Measurement
63% of actioned suggestions accepted, 26% edited (Jan 17, 2024)
Lowe'sSep 25, 2026OpenAI customer story (vendor case study)At Lowe's, OpenAI models answer store associates' project questions, nearly half of them by voice.
Seemantini GodboleEVP, Chief Information and AI OfficerLowe's
Organization-wide. Mylow on Lowes.com and Mylow Companion in 1,700 stores. Since spring 2025.
OpenAI customer story (vendor case study), published Sep 25, 2026Read the case study (opens OpenAI customer story (vendor case study) in a new tab)
Observation
At Lowe's, OpenAI models answer store associates' project questions, nearly half of them by voice.
Takeaway
Measure an assistant by the questions people bring to it and by a business outcome downstream, such as conversion, and offer voice where employees work on their feet.
What happened
Lowe's built two assistants on OpenAI models: Mylow, a project advisor on Lowes.com, and Mylow Companion, which gives store associates answers across departments. Both launched in spring 2025. Mylow Companion is available to about 300,000 associates across 1,700 stores, and nearly half of its interactions happen by voice. Lowe's counts the questions associates ask as its adoption measure, and on its Q2 2026 earnings call reported that Mylow users convert at three times the rate of other online shoppers. The two assistants have answered millions of questions since launch.
Vendor customer story
Published by OpenAI with Lowe's executives quoted
Conversion figure from Lowe's Q2 2026 earnings call
Deployment details
Organization-wide. Mylow on Lowes.com and Mylow Companion in 1,700 stores. Since spring 2025.
Vendor customer story
Published by OpenAI with Lowe's executives quoted
Conversion figure from Lowe's Q2 2026 earnings call
Team
Lowe's digital commerce and store operations
Duration
Since spring 2025
Workload
Project guidance for customers online and associates in stores
Product version
Mylow and Mylow Companion, built on OpenAI models
Measurement
Millions of questions answered; Mylow users convert at 3x other online shoppers

Switched to or from it3

At Ploy, GPT-5.6 Sol beat Claude Opus in production evals once the team rewrote its tool schemas and cache keys.

Amp replaced GPT-5.6 Sol with Claude Opus 5.5 in its default medium mode after its evals, and kept GPT-6 Astra behind its High mode.

Simon Willison switched the public Datasette Agent to GPT-6 Luna on its launch day and found it fast and competent at SQL.

Run it beside another tool2

DoorDash runs the OpenAI API with Azure OpenAI as its fallback, behind one gateway that attributes usage costs to each department.

Wealthsimple's LLM gateway started with OpenAI and Cohere, added Gemini and Amazon Bedrock beside them, then rebuilt its API to mirror OpenAI's specification.

Evaluated it1

Replit tested GPT-6 Astra as Replit Agent's core loop for a production week and found it delegates to cheaper workers on its own.

Adopted it3

At Asana, keeping a browser agent's cached history intact made each run far cheaper on GPT-6.1 Sol.

incident.io built suggested incident summaries on OpenAI's models and shipped them to every customer; 63% of actioned suggestions were accepted as written.

At Lowe's, OpenAI models answer store associates' project questions, nearly half of them by voice.