Together AI−49 Negative

Operator Insights

What are teams learning from building on Together AI and running it in production?

3 accounts: 1 left it or turned it down, 1 evaluated it and 1 adopted it

Left it or turned it down1

Swain MolsterStaff Full Stack EngineerGoodParty.org
GoodParty.org, a public benefit corporation whose product ran streaming chat on Anthropic and non-streaming completions on Together.
GitHub (thegoodparty/omni), published Jun 26, 2026Read the pull request (opens GitHub (thegoodparty/omni) in a new tab)
Observation
GoodParty.org moved its non-streaming LLM calls off Together AI onto Anthropic in June 2026 to run one provider after using two.
Takeaway
If you keep Together as a second provider, pin only models it serves serverless, alert on a 4xx model error as a configuration fault, and fund credits ahead of usage.
What happened
GoodParty.org's non-streaming completions called Together directly while its streaming chat already ran on Anthropic. PR #435, merged June 26, 2026, migrated the LLM layer fully off Together AI onto Anthropic, naming Together as the same dependency behind the recurring credit-exhaustion and non-serverless-model fallback incidents (MiniMax/Qwen). Two days earlier PR #367 had traced a production alert that fired and resolved about every 30 minutes to a fallback model Together does not serve serverless, and said the fallback was never functional.
- Public engineering write-up (pull requests in GoodParty.org's own repository)
Deployment details
GoodParty.org, a public benefit corporation whose product ran streaming chat on Anthropic and non-streaming completions on Together.
- Public engineering write-up (pull requests in GoodParty.org's own repository)
Team
GoodParty.org engineering
Duration
Until June 26, 2026
Workload
Non-streaming chat, JSON and tool completions for queued background work
Model
A Qwen primary with a MiniMax-M2.5 fallback that Together does not serve serverless
Measurement
A production alert firing and resolving about every 30 minutes (PR #367, Jun 24, 2026)

Evaluated it1

William PietriWorks for MLCommons on tools to measure AI safety, as his own site describes himMLCommons
MLCommons' modelbench, the evaluation code behind its AILuminate benchmark, with an ensemble of annotator models served by Together.
GitHub (mlcommons/modelbench), published May 20, 2025Read the issue (opens GitHub (mlcommons/modelbench) in a new tab)
Observation
MLCommons scores its AILuminate safety benchmark with annotator models it calls on Together AI, and two changes in what Together serves serverless broke that evaluation pipeline in ten months.
Takeaway
An evaluation pipeline pinned to exact open models needs those models on dedicated endpoints, or a monitor that catches a serverless change before a benchmark run.
What happened
In May 2025 an MLCommons engineer filed that Together no longer supported Mixtral-8x22B-Instruct-v0.1, one of the annotators in the ensemble. In March 2026 the default evaluator broke when Together returned that LlamaGuard-2-8b was a non-serverless model that needs a dedicated endpoint. Later MLCommons issues are titled to swap the Mixtral annotator, create scorers as Together dedicated endpoints and monitor them, and their bodies are empty.
- Public engineering write-up (issues in MLCommons' own repository)
Deployment details
MLCommons' modelbench, the evaluation code behind its AILuminate benchmark, with an ensemble of annotator models served by Together.
- Public engineering write-up (issues in MLCommons' own repository)
Team
MLCommons' AILuminate benchmark engineers
Duration
May 2025 to March 2026
Workload
Safety benchmark scoring with an ensemble of annotator models
Model
Mixtral-8x22B-Instruct-v0.1 and LlamaGuard-2-8b annotators
Measurement
Two serverless changes broke the pipeline in ten months (May 2025 and March 2026)

Adopted it1

Taylor BlantonLead Software Engineer, as his GitHub profile describes himselfAi2
Ai2's olmo-api, the backend of the playground for its own open OLMo models.
GitHub (allenai/olmo-api), published Apr 16, 2024Read the pull request (opens GitHub (allenai/olmo-api) in a new tab)
Observation
Ai2 switched its OLMo Playground API to a TogetherAI engine in April 2024 as a replacement for its InferD serving, and removed the integration in November 2024.
Takeaway
A managed host can stand in for in-house serving quickly. Check that it returns what your app reads from the model, such as per-token logprobs while streaming, before you switch.
What happened
In April 2024 Ai2 engineers added a TogetherAI engine to the OLMo Playground API and switched to it as a replacement for InferD. The pull request notes that Together returned the logprob only for the chosen token while streaming, which did not map to Ai2's expected response. A follow-up by Taylor Blanton added max_tokens handling and the correct finish reason for streams. In November 2024 Ai2 removed the TogetherAI integration, and the pull request gives no reason.
- Public engineering write-up (pull requests in Ai2's own repository)
Deployment details
Ai2's olmo-api, the backend of the playground for its own open OLMo models.
- Public engineering write-up (pull requests in Ai2's own repository)
Team
Ai2 engineers on the OLMo Playground API
Duration
April 2024 to November 2024
Workload
The OLMo Playground API, replacing InferD

Left it or turned it down1

GoodParty.org moved its non-streaming LLM calls off Together AI onto Anthropic in June 2026 to run one provider after using two.

Evaluated it1

MLCommons scores its AILuminate safety benchmark with annotator models it calls on Together AI, and two changes in what Together serves serverless broke that evaluation pipeline in ten months.

Adopted it1

Ai2 switched its OLMo Playground API to a TogetherAI engine in April 2024 as a replacement for its InferD serving, and removed the integration in November 2024.