Choose each model from evaluations you run yourself, and keep the test set in tooling you control, because OpenAI's Evals platform shuts down on November 30, 2026. Then put stable content at the front of every prompt so it caches. Set spend limits and retention terms before the workload reaches production, the order OpenAI's documentation and the public accounts of teams running its models agree on.
When you need this
- A feature or agent built on OpenAI models is moving from prototype to production traffic.
- The monthly bill grows faster than usage, and nobody can say which workload drives it.
- A model you depend on is scheduled for shutdown, or a new model ships and the team has to decide whether to switch.
Evaluation criteria
Setup: one project per workload
Can you see and cap spend and rate limits for each product or team?
Look forGive each workload its own project, API keys, and hard spend limit. Rate and spend limits apply at organization and project level. Set a model allowlist through the Admin APIs so each workload calls only approved models.
Model choice: evaluations per task
Which model passes your own test set at the lowest cost per completed task?
Look forA held-out test set run on at least two of GPT-6 Astra, GPT-6.1 Sol, and GPT-6 Luna, scored on cost per task. One builder measured US$1.12 per task on GPT-5.5 against nearly US$23 on GPT-5.5 Pro. A team running agent evals found GPT-5.6 Sol twice as costly as a rival model for the same result. The Evals platform shuts down on November 30, 2026, so keep the test set in your own tooling.
Cost control: build for the cache
Does the stable part of every prompt sit before anything that changes per request?
Look forPut tool definitions and instructions before content that changes. Track cache hits with cached_tokens and cache_write_tokens in each response. GPT-5.6 cached none of Ploy's shared 29K-token prefix for new conversations until the team set prompt_cache_key per customer workspace. First-call hits then reached 83.7%.
Rollout: batch what can wait
Which workloads can accept results within 24 hours, or slower responses?
Look forSend evaluations, backfills, and bulk classification to Batch at half price. Each batch supports up to 50,000 requests in a separate rate-limit pool. Flex handles lower-priority requests at the same rate. One builder backfilled about 60,000 job postings through Batch for US$10. For classification and routing, the Decisions API on GPT-6 Luna, generally available since October 22, 2026, bills input at US$0.10 per million tokens and charges nothing for output. It supports up to 400 questions per request.
Capacity: ramp gradually
Will launch traffic stay within your tier's rate limits and ramp rate?
Look forAfter traffic passes 1 million tokens per minute, increase it by no more than 50% every 15 minutes. Follow Retry-After on 429 slow_down and 503 server_is_overloaded. Move to Scale Tier or Reserved Tier when traffic routinely hits ramp limits.
Governance: retention before launch
Does your retention arrangement cover every endpoint you call?
Look forObtain Zero Data Retention or Modified Abuse Monitoring approval where required. Set store to false on Responses calls. Identify endpoints outside Zero Data Retention, including Batch, Files, vector stores, Conversations, and the Agents API.
Review practice: plan for model changes
What happens to your feature when a model you use is retired?
Look forKeep model IDs in one configuration file and rerun evaluations before each switch. Track shutdown dates. OpenAI gives at least 6 months' notice for generally available models and 3 months for specialized variants. Early-release models get as little as 2 weeks.
Questions to ask vendors
- Which usage tier is our organization on, and what monthly usage limit and per-model rate limits come with it?
- Which of the endpoints we call are eligible for Zero Data Retention, and which keep application state?
- Is any model or endpoint we use deprecated, and what is the recommended replacement and shutdown date?
- Do we need Scale Tier, Reserved Tier, or regional processing before launch, and what lead time do they need?
Red flags
- One API key and one project shared by every product, so nobody can trace spend or rate limits to a workload.
- No evaluation set, so benchmark headlines or list price decide a model switch.
- A timestamp, user ID, or tool list that changes per request placed at the top of the prompt.
- Retry logic that resends immediately on 429 or 503 and ignores Retry-After and backoff.
- No organization or project hard spend limit below the tier's monthly usage limit.
Products in this piece
Sources
- Official documentation
- 12
- Public source
- 4
- OpenAI API docs: Models
- OpenAI API docs: Rate limits
- OpenAI API docs: Spend limits
- OpenAI API docs: Admin APIs
- OpenAI API docs: Evaluation best practices
- OpenAI API docs: Prompt caching
- OpenAI API docs: Batch API
- OpenAI API docs: Flex processing
- OpenAI API docs: Decisions API
- OpenAI API docs: Changelog
- OpenAI API docs: Data controls in the OpenAI platform
- OpenAI API docs: Deprecations
- Hacker News comment on cost per task for GPT-5.5 and GPT-5.5 Pro (Jul 6, 2026)
- Hacker News comment on agent evals of GPT-5.6 Sol, Terra, and Luna (Jul 10, 2026)
- Ploy: Migrating a production AI agent to GPT-5.6 (Jul 9, 2026)
- Dumky de Wilde: LLMs Are Cheap Enough to Change How You Work With Data (Dec 21, 2025)