Skip to main content

Inference Hosting

Together AI

Together AI serves open models through a serverless per-token API compatible with the OpenAI SDKs, provisioned throughput with a monthly success commitment, dedicated endpoints billed by the minute, fine-tuning and multi-node GPU clusters. Accounts are fully prepaid, with a minimum US$15 credit purchase.

The 30-Second Take

Pick Together AI if you want open models on an OpenAI-compatible API without running GPUs.

You start on prepaid per-token serverless and move steady traffic to reserved capacity. MLCommons scores its AILuminate safety benchmark with annotator models it calls on Together AI.

Best fit
Teams that want open-weight models behind an OpenAI-compatible API, billed per token, with a path to reserved capacity.
Main constraint
Serverless performance is best-effort, and serverless models are deprecated on notice.
First move
Run your own tool-calling and long-prompt tests on each model you plan to pin, from a prepaid balance.

Why

Watch the models you pin. Two serverless changes broke MLCommons' benchmark scoring.

Practitioner sentiment, all practitioners−49 Negative

35 practitioners, 46 sources reviewed, Updated Nov 1, 2026How this is measured

Read the GILD Review
Save a product your organization uses.Keep up with Together AI without adding it to your stack.Visit Website
Operating stack
AI Application Platform

Comparisons

How does Together AI compare with the alternatives?

  1. Together AI vs Fireworks AI

    • Inference Hosting

    Both list gpt-oss-120b at US$0.15 input and US$0.60 output per 1M tokens. Pick Together AI for the lower listed on-demand H100 rate for a dedicated deployment. Pick Fireworks AI when prompts must not be stored by default, since it states Zero Data Retention by default.

  2. Together AI vs OpenRouter

    • Inference Hosting
    • AI Gateways

    By GILD's calculation, at 1,000,000 gpt-oss-120b requests a month of 2,000 input and 500 output tokens, Together costs US$600, and OpenRouter pinned to Together's endpoint adds a US$33 credit fee. Pick Together AI to hold the host and buy dedicated GPUs. Pick OpenRouter to route the same model across hosts.

  3. Together AI vs Groq

    • Inference Hosting

    Both list gpt-oss-120b at US$0.15 input and US$0.60 output per 1M tokens and a 50% batch discount on the models their batch pages list. Groq does not retain inference data by default, while Together stores prompts until an admin turns storage off.

Operator Insights

What are teams learning from building on Together AI and running it in production?

See all 3 operator accounts

Buying and deployment

What a team building on it should verify before production traffic grows.

Sourced from Together AI documentation

How you buy it

US$15 minimum

Fully prepaid. API access stops at a zero balance, and provisioned throughput goes through sales.

Source 1, docs.together.ai, Billing and credits: Billing and creditsSource 2, Together AI, Provisioned throughput: Provisioned throughput

Details

Together AI is fully prepaid: you need a positive credit balance to use the platform, and access requires a minimum US$15 credit purchase. If your balance reaches zero, API access is suspended until you add credits. Customers with an active Scale or Enterprise contract continue to be billed under their existing contract terms. Serverless and dedicated endpoints are self-serve. Provisioned throughput is currently unavailable for self-service and is requested through sales.

Cost drivers

US$1.40 in, US$4.40 out

Per 1M tokens for GLM-5.3 on serverless. Dedicated GPUs bill by the minute, an H100 at US$5.49 an hour.

Source 9, docs.together.ai, Serverless overview: Serverless overviewSource 10, together.ai, GLM-5.3 model page: GLM-5.3 model pageSource 11, docs.together.ai, Dedicated endpoints pricing: Dedicated endpoints pricingSource 2, Together AI, Provisioned throughput: Provisioned throughputSource 12, docs.together.ai, Billing spend controls: Billing spend controls

Details

Serverless chat models bill per input and output token. GLM-5.3 costs US$1.40 per 1M input tokens and US$4.40 per 1M output tokens. Dedicated model inference bills by the minute for each running replica. An H100 80GB costs US$5.49 per hour. Provisioned throughput is priced at a flat US$0.05 per PTU per minute, on a one-month minimum term. Traffic above purchased PTU capacity may fall back to the shared serverless pool on a best-effort basis, billed at standard serverless rates.

Where it runs

US-only at 1.25x

us-api.together.ai keeps processing in the US for select models. EU needs a VPC or dedicated endpoint.

Source 13, Together AI, Privacy and security: Privacy and securitySource 14, docs.together.ai, Geo-specific inference: Geo-specific inferenceSource 9, docs.together.ai, Serverless overview: Serverless overviewSource 15, docs.together.ai, List placement profiles: List placement profilesSource 16, docs.together.ai, Dedicated endpoint configurations: Dedicated endpoint configurations

Details

For select serverless models, requests to us-api.together.ai keep inference processing in the United States at 1.25× the standard serverless rate. The guarantee covers only where those requests are processed and makes no promise about where data is stored, logged, or routed. For other data-residency, regulatory, or compliance requirements, such as GDPR, Together supports private networking and virtual private cloud (VPC) deployments, including in EU regions. Use a dedicated endpoint, or contact sales to discuss the right setup for your workload.

What to verify

SSO on Scale, Enterprise

The trust center lists SOC 2 and ISO 27001:2022. SCIM is listed under What's coming.

Source 18, docs.together.ai, Single sign-on: Single sign-onSource 17, trust.together.ai, Trust Center: Trust Center

Details

Together AI's trust center lists SOC 2 and ISO 27001:2022 under Compliance. SSO is available for Scale and Enterprise accounts. SCIM provisioning is listed under What's coming. Together's API lists an endpoint's audit and lifecycle events, newest first. That scope is per endpoint, and an organization-wide audit log of user or admin actions is not part of it.

Signals

What is the current evidence saying about Together AI?

A current read of what developers and teams report publicly about running open models on Together AI, from Hacker News, Reddit, GitHub, G2 and two practitioners' own blogs, with each finding linked to its sources.

Whose sources count

Practitioner sentiment, all practitioners

−49 Negative net

of 40 sources that take a side, each author counted once.

How this figure is made

Teams and leaders have 1 accounts; GILD prints their sentiment from 20.

Current sentiment

Positive, mixed and negative sources

The score counts them as 5 positive, 8 mixed and 22 negative votes, one per author.

The read leans negative. Developers like pay-per-token access to open models on a US host and its speed against some hosts, and report tool-calling errors, retired serverless models, outages and credit problems.

Run a pilot on your own prompts at expected concurrency to measure time to first token, output speed, error rate and cost per million tokens.

Evidence field. All 46 sources, one mark each.

Arrange by

Each mark is one source: positive, mixed, negative, no opinion stated, or documentation (not counted).

By sentiment, All practitioners. Every source GILD read, grouped as the field draws it.
GroupSourcesPositiveMixedNegativeNo opinion stated and not counted in the scoreDocumentation, checked and not counted
Positive770000
Mixed707000
Negative26002600
Not counted100010
Documentation500005
Every source GILD read, by the quarter it was published, as the field's time axis counts it. Documentation is dated by the day it was checked.
QuarterSources
Q1 20253
Q2 20253
Q3 20257
Q4 20252
Q1 20264
Q2 20265
Q3 202610
Q4 20267
Documentation, checked5

By sentiment

5 documentation pages check facts and are not counted.

Hover a mark for a quick look, click it or press Enter to open its source.Tap a mark to open its source, then step to its neighbors with Previous and Next.

What practitioners praise

Each mark is a source on it; filled marks praise it.

For all practitioners, no capability reaches 5 sources from 3 authors.

    Most common concerns

    Top concern of 3. Each mark is a source on it; filled marks raise it.

    1. Tool Calling & API CompatibilityAll 9 sources on it raise it as a concern
    All 46 sources behind these findings
    1. Third-party integrationsTogether AI (Official), Product documentation, checked Nov 1, 2026
    2. OpenAI compatibilityTogether AI (Official), Product documentation, checked Nov 1, 2026
    3. Privacy and securityTogether AI (Official), Product documentation, checked Nov 1, 2026
    4. Serverless rate limitsTogether AI (Official), Product documentation, checked Nov 1, 2026
    5. Provisioned throughputTogether AI (Official), Product documentation, checked Nov 1, 2026
    6. Provider question (GLM 5.3 Flash)Opposite-Cup2850, Reddit (r/SillyTavernAI) (self-described: Choosing a host for GLM-5.3-Flash in SillyTavern), Reddit post, r/SillyTavernAI, published Oct 28, 2026
    7. Same model, two services: gpt-oss-120b on Cerebras and Together (speed)Sam Mirazi, data scientist and founder of BrAIN (self-described: His own test of gpt-oss-120b on two paid API accounts, 20 runs per service per test), Personal blog post, advisee.substack.com, published Oct 25, 2026
    8. Same model, two services: gpt-oss-120b on Cerebras and Together (cost)Sam Mirazi, data scientist and founder of BrAIN (self-described: The same test, priced at each service's published rates on October 24, 2026), Personal blog post, advisee.substack.com, published Oct 25, 2026
    9. Hacker News comment recommending DeepSeek V4.1 Flash from together.aiJosh Heitzman (joshheitzman), software engineer and creator of the FluxImp agent harness (self-described: Choosing a host for DeepSeek V4.1 Flash as a coding model, after open-sourcing their harness), Hacker News comment, published Oct 23, 2026
    10. G2 review of Together AI on AWS Marketplace by karan R.karan R., G2 reviewer (self-described: Builds voice agents for businesses on their own platform), G2 review on AWS Marketplace, published Oct 21, 2026
    11. G2 review of Together AI on AWS Marketplace by Muhammad Basit J.Muhammad Basit J., G2 reviewer (self-described: A developer testing the platform's playground and API), G2 review on AWS Marketplace, published Oct 21, 2026
    12. Hacker News comment on GLM-5.3 on together.ai as a second reviewerzzzeek, creator of SQLAlchemy, Alembic Migrations, dogpile.cache and Mako Templates (self-described: An open-source maintainer on a Claude Max subscription who uses open models when tokens matter), Hacker News comment, published Oct 4, 2026
    13. Hacker News comment on running DeepSeek-V4-Flash-0731 through Kilo Code and his own harnessJosh Heitzman (joshheitzman), software engineer and creator of the FluxImp agent harness (self-described: Benchmarking open-weight models across hosts with Kilo Code and an agent harness of their own), Hacker News comment, published Sep 17, 2026
    14. OpenRouter keeps routing calls to failing providerGT95, Reddit (r/openrouter) (self-described: A system administration task in the Hermes agent, routed by OpenRouter), Reddit post, r/openrouter, published Sep 13, 2026
    15. Hacker News comment on noisy hosted results in gpt-oss red-teamingNils Durner (ndr_), technologist in generative AI, security and privacy, author of arXiv:2510.01259 (self-described: Red-teaming research for OpenAI's gpt-oss competition, published as arXiv:2510.01259), Hacker News comment, published Sep 12, 2026
    16. So you want to use OpenRouter? (hollow completions)Mo Moustafa, AI Product Engineer for Startups, runs Olly (self-described: Olly, an iMessage assistant on open models through OpenRouter), Engineering blog post, mmoustafa.com, published Sep 7, 2026
    17. So you want to use OpenRouter? (image input by host)Mo Moustafa, AI Product Engineer for Startups, runs Olly (self-described: Olly, an iMessage assistant on open models through OpenRouter, testing image input per host), Engineering blog post, mmoustafa.com, published Sep 7, 2026
    18. Hacker News comment on GLM-5.3-Flash via together.ai against DeepSeek V4 FlashPeter Sugihara (rgbrgb), co-founder of Open Listings (self-described: A personal ad-detection task over written transcripts, ten tasks per run), Hacker News comment, published Aug 28, 2026
    19. Hacker News comment on a mixture of open-weight models through Together.ai and Fireworks.aidkersten, Hacker News (self-described: Their own coding work, after leaving a closed-model service they call unreliable), Hacker News comment, published Aug 24, 2026
    20. Hacker News comment on running new Chinese models on US providersjoecot, Hacker News (self-described: Choosing where to run newly released open-weight models), Hacker News comment, published Jul 20, 2026
    21. Reply on OpenRouter billing compared with together.ai refundsEvening-Guarantee-84, Reddit (r/openrouter) (self-described: Prepaid credit left unused on Together), Reddit comment, r/openrouter, published Jul 19, 2026
    22. Mark 6 removed Together.ai serverless models as deprecatedMax Collier (mcwalrus), software engineer, entain-group on his GitHub profile (self-described: Checking the models.dev catalog against Together's serverless endpoint), GitHub issue, anomalyco/models.dev, published Jul 18, 2026
    23. Together.ai models to be deprecated July 10Max Collier (mcwalrus), software engineer, entain-group on his GitHub profile (self-described: Keeping the pi coding agent's Together model list current), GitHub issue, earendil-works/pi, published Jun 28, 2026
    24. Together.ai MiniMaxAI/MiniMax-M2.5 model doesn't work with serverless inferenceEric Henry (awakenalive), TrueVektor Development (self-described: Selecting a Together model in the pi coding agent on pay-per-token serverless), GitHub issue, earendil-works/pi, published Jun 8, 2026
    25. bug: max_output_tokens setting doesn't work with Together.aiBranko Radovanović (brankoradovanovic-mcom), Multicom d.o.o. (self-described: The Jan desktop client with Together.ai as its remote provider), GitHub issue, janhq/jan, published Jun 3, 2026
    26. Hacker News comment on together.ai outages and latencyHDBaseT, Hacker News (self-described: An aside in a thread about open-model hosts), Hacker News comment, published May 19, 2026
    27. Hacker News comment on Together.ai's price against DeepSeek's own APIpants2, Hacker News (self-described: Weighing US hosts against DeepSeek's first-party pricing), Hacker News comment, published May 11, 2026
    28. Release a langchain-together 1.0 without dependency conflictsSantiago Martinez Bragado (smartinezbragado), Traza (self-described: Traza's workspace on LangChain 1.x), GitHub issue, langchain-ai/langchain-together, published Mar 24, 2026
    29. Ephemeral chat. Private, secure, chat with together.ai open source models.slippery, builder of Ephemeral, an open-source chat app built on Together (self-described: Ephemeral, their open-source web chat app on Together's Kimi-K2.5), Reddit post, r/LocalLLaMA, published Feb 7, 2026
    30. Change model qwen/qwen3-coder-next from together.ai to openrouterJuan Pedro Michelini Jorge (juanmichelini), MTS @ Openhands: Evals, benchmarks, leaderboards, harness (self-described: The OpenHands agent SDK's evaluation runs, calling models through LiteLLM), GitHub issue, OpenHands/software-agent-sdk, published Feb 4, 2026
    31. seed must not be set when best_of is greater than 1Roman Vashurin (rvashurin), research engineer at MBZUAI (self-described: Uncertainty-quantification research that needs several seeded completions per prompt), GitHub issue, togethercomputer/together-python, published Jan 16, 2026
    32. Having a problem with the together.ai apiInfamous_Sector_6411, Reddit (r/LocalLLaMA) (self-described: Personal projects on AnythingLLM, with €15 of Together credits), Reddit post, r/LocalLLaMA, published Oct 19, 2025
    33. Reply on truncated answers from the together.ai apiabnormal_human, Reddit (r/LocalLLaMA) (self-described: Replying in the same thread from their own usage across many use cases), Reddit comment, r/LocalLLaMA, published Oct 19, 2025
    34. Fix access to Llama-3.3-70B-Instruct modelwleonie, collaborator on debatelab/evidence-seeker (self-described: A multi-step evidence-checking pipeline that uses grammar-constrained output), GitHub issue, debatelab/evidence-seeker, published Sep 18, 2025
    35. Agent seems unable to do anything using together.aiDavid Else (David-Else), software developer, Else Web Development (self-described: The opencode coding agent on Together-hosted open models, with no proxy), GitHub issue comment, anomalyco/opencode, published Aug 22, 2025
    36. Cannot use GPT-OSS-120b with ChatTogether and tools for create_react_agentSagar (sagar-catomz), GitHub (self-described: A LangGraph ReAct agent on ChatTogether), GitHub issue, langchain-ai/langchain-together, published Aug 20, 2025
    37. Hacker News comment recommending together.ai for usage-based billingjoecot, Hacker News (self-described: Occasional use of open-source models), Hacker News comment, published Aug 5, 2025
    38. Use alternative of TogetherAIAmritanshu Barpanda (ABarpanda), aspiring AI engineer, Advanced Engineering Hire intern at Accenture Pune from May to July 2026 (self-described: Animegen, his image-generation project, on Together's free FLUX model), GitHub issue, ABarpanda/Animegen, published Jul 19, 2025
    39. Got Out of Credits Email from Together AI While Only Using Free Model and Still Have $1 in BalanceWild_King_1035, Reddit (r/LocalLLaMA) (self-described: An app calling the free llama-3-70b-instruct-turbo-free model through the Together API for about a month), Reddit post, r/LocalLLaMA, published Jul 16, 2025
    40. Hacker News comment on Gemma E4B on together.aiimpure, Hacker News (self-described: Trying a small open model on together.ai against Gemini 2.5 Flash Lite), Hacker News comment, published Jul 3, 2025
    41. Llama 4 Scout on together_ai returns tool calls in the message textJarett (JarettForzano), Vals AI (self-described: A tool-using agent calling Together through LiteLLM), GitHub issue, BerriAI/litellm, published Jun 5, 2025
    42. The together library adds over 100MB of dependenciesMatthias Geihs (matthiasgeihs), cryptography engineer and systems architect at Consensys (self-described: Calling Together from small shared test VMs), GitHub issue, togethercomputer/together-python, published Jun 3, 2025
    43. Unclosed client session errors from the async client after a few batch callsVenkat (venkatasg), assistant professor of computer science at Ithaca College (self-described: Research batches of DeepSeek-V3 calls on Build tier 3), GitHub issue, togethercomputer/together-python, published Apr 17, 2025
    44. Remove the pyarrow dependency from the together Python SDKSteve Cosman (scosman), building Kiln (kiln.tech), formerly Apple Photos AI (self-described: Shipping Kiln, an app that bundles model-provider SDKs), GitHub issue, togethercomputer/together-python, published Mar 19, 2025
    45. together fine-tuning list crashes with a validation errorCarl Silva (silvacarl2), Sphinx Medical Technologies (self-described: Managing fine-tuning jobs from Together's command-line client), GitHub issue, togethercomputer/together-python, published Feb 2, 2025
    46. JSON mode on Llama 3.1 and 3.3 70B Turbo returns invalid JSONShivanshu Gupta (Shivanshu-Gupta), NLP and AI researcher, Adobe on his GitHub profile (self-described: Generating structured agent traces on Together's serverless Llama 70B Turbo models), GitHub issue, togethercomputer/together-python, published Jan 8, 2025

    Jan 2025 to Oct 2026.

    41 sources from practitioners, by month and sentiment.

    The full readExplore all Together AI Signals6 praise themes, 7 concerns, 0 points of agreement and 0 disagreements across all three groups, with all 46 sources open to inspect.

    Where It Fits

    Where does Together AI belong in the stack?

    Open Where It Fits: the Stack Field and the full map

    Together AI is Core across Operate, supports Model, and touches Build, Data, and Evaluate.

    Business area
    Product & Engineering
    Function
    Software Development
    Operating stack
    AI Application Platform
    Primary role
    Open-Model Inference & GPU Capacity
    Together AI beside the rest of its stackThe Stack Field, every product on every layer
    FootprintTogether AI across the AI Application Platform stack. Bar height is its role on each layer.

    1. Model

      1. Model APIs: Supports

        Model APIs, supports. Together serves open models through a serverless per-token API that is compatible with the OpenAI REST API and SDKs, so an app switches by changing the API key and base URL. Its serverless catalog also lists closed image and video models such as openai/gpt-image-2 and google/veo-3.1.

      2. Model gateway: Touches

        Model gateway, touches. LiteLLM 1.100.0 and later ships official Together AI support, so a gateway can sit in front of Together. Together Link, in beta, routes a coding agent's requests through an auto router that picks a Together model for each one.

    2. Build

      1. Orchestration: Touches

        Orchestration, touches. LangChain's langchain-together package and the Vercel AI SDK's @ai-sdk/togetherai provider call Together models. The agent's steps and control flow stay in those frameworks.

      2. Tool use: Touches

        Tool use, touches. Together serves the models an agent calls through OpenAI-compatible requests. The tools themselves run in the agent or its framework.

    3. Data

      1. Retrieval: Touches

        Retrieval, touches. Together's API serves embeddings, and langchain-together provides them to LangChain apps. The index and search stay with a vector database.

    4. Evaluate

      1. Evaluation: Touches

        Evaluation, touches. The batch API runs many independent inference requests from one uploaded JSONL file, billing only successful responses. Datasets and scoring stay with an evaluation tool.

    5. Operate

      1. Observability: Touches

        Observability, touches. For dedicated endpoints, Together lists each endpoint's audit and lifecycle events, newest first. Request tracing across an application stays with an observability tool.

      2. Cloud deployment: Core

        Cloud deployment, core. This is Together's primary role: serverless inference with no minimum cost, provisioned throughput with a 99% monthly success commitment on five models, dedicated endpoints on reserved hardware billed by the minute, and multi-node GPU clusters on Kubernetes.

      3. Guardrails: Touches

        Guardrails, touches. Organization admins can turn prompt storage off for zero data retention and control passthrough models with a separate toggle. Prompt and output filtering stay with a guardrails product.

    Market mapThe layers Together AI covers, and who else owns each one.

    1. Cloud deployment & capacity

      CoreOperate

    2. Model APIs

      SupportsModel

    3. Model gateways & routing

      TouchesModel

    4. Orchestration & agent frameworks

      TouchesBuild

    5. Tool use & integrations

      TouchesBuild

    4 more layers on the full map. Core, Supports and Touches describe coverage. They say nothing about quality.

    Models

    Which models run in Together AI?

    The current releases GILD tracks that public sources record running in Together AI. Each model's page charts its public benchmark scores.

    1. DeepSeek V4.1 FlashDeepSeek's smallest model in its V4.1 architecture family, with native visual understanding, served on the DeepSeek API as deepseek-flash. Since October 17, 2026 that name serves the DeepSeek-V4.1-Flash-1017 build. Weights under the MIT license.DeepSeek V4.1Benchmarks
    2. GLM-5.3Z.ai's flagship open-weight model for coding and agentic work, with a 1M-token context window, on the Z.ai API and also served by Mistral AI and OpenRouter. Weights under the GLM-5.3 License.GLMBenchmarks
    3. GLM-5.3-FlashZ.ai's first native multimodal model in the GLM-5 series, for efficient coding and long-horizon agent tasks, with a 1M-token context and a faster FlashX variant. 320B parameters with 18B active; weights under the MIT license.GLMBenchmarks
    4. gpt-oss-120bOpenAI's largest open-weight model, 117B parameters with 5.1B active, with a 131,072-token context window. Weights under the Apache 2.0 license; OpenAI lists no API price for it, so it runs on your own hardware or a host's.gpt-ossBenchmarks
    5. MiniMax M3.1 FlashMiniMax's multimodal reasoning model for agentic work, with a 1M-token context window, on the MiniMax API, M Plan and MiniMax Code, and also on OpenRouter and OpenCode Go.MiniMax M3Benchmarks

    More on Together AI

    1. GILD ReviewGILD's evaluation, with the buying facts
    2. Research1 piece
    3. Guides1 guide
    4. Operator Insights3 operator accounts
    5. Where It FitsOn the AI Application Platform stack
    6. Related Products6 products