Where It Fits
Where does Braintrust belong in the stack?
Braintrust's primary role: AI Evaluation & Observability. It is Core across Evaluate and Operate, supports Model and Build, and touches Data.
Model
- Model APIs: Touches
- Model gateway: Supports
Build
- Orchestration: Touches
- Tool use: Supports
Data
- Retrieval: Touches
Evaluate
- Evaluation: Core
Operate
- Observability: Core
- Cloud deployment: Touches
- Guardrails: Touches
Braintrust beside the rest of its stack
Every product we cover on AI Application Platform, on the layers it owns, placed by what its practitioners report. Pick any of them to compare.
Signals from
Show products that
Activating a product opens its read here. Activating the selected product opens its hub.
Each product sits on the layers it owns, placed left to right by how far its positive sources lead its negative ones. A disc's area is its sources in this view and its tone is its band. Products with fewer than 20 sources from 15 authors have no net and sit at the right, under Unscored. The teal line joins Braintrust across its layers.
Read the field as a table
| Product | Core on | Supports | Overall Signal | Net | Sources | Positive | Mixed | Negative |
|---|---|---|---|---|---|---|---|---|
| Claude Agent SDK | Orchestration | Tool use, Cloud deployment, Guardrails | Strong positive | +52 | 22 | 13 | 8 | 1 |
| Braintrust | Evaluation, Observability | Model gateway, Tool use | Strong positive | +50 | 34 | 20 | 11 | 3 |
| Langfuse | Evaluation, Observability | None | Positive | +46 | 26 | 12 | 14 | 0 |
| Jev | Supporting only | Model APIs, Model gateway, Tool use, Retrieval, Evaluation, Guardrails | Positive | +40 | 25 | 12 | 11 | 2 |
| Datadog Agent Observability | Evaluation, Observability | Guardrails | Positive | +32 | 26 | 9 | 16 | 1 |
| Arize | Evaluation, Observability | Guardrails | Positive | +26 | 27 | 11 | 12 | 4 |
| Claude API | Model APIs, Tool use | Orchestration, Retrieval, Cloud deployment | Positive | +25 | 52 | 22 | 21 | 9 |
| Amazon Bedrock | Model APIs, Cloud deployment | Model gateway, Orchestration, Tool use, Retrieval, Evaluation, Observability, Guardrails | Mixed | +9 | 57 | 25 | 12 | 20 |
| LiteLLM | Model gateway | Tool use, Observability, Guardrails | Mixed | +8 | 37 | 10 | 21 | 6 |
| OpenAI API | Model APIs, Tool use | Orchestration, Retrieval, Observability, Cloud deployment, Guardrails | Mixed | +3 | 65 | 22 | 23 | 20 |
| Gemini API | Model APIs, Tool use | Orchestration, Retrieval, Cloud deployment, Guardrails | Mixed | 0 | 74 | 24 | 25 | 25 |
| pgvector | Retrieval | Cloud deployment | Negative | −29 | 24 | 5 | 8 | 11 |
| Mastra | Orchestration | Model gateway, Tool use, Retrieval, Evaluation, Observability, Cloud deployment, Guardrails | Negative | −36 | 25 | 4 | 8 | 13 |
| Qdrant | Retrieval | Cloud deployment | Negative | −40 | 41 | 8 | 10 | 23 |
| Pinecone | Retrieval | Tool use, Cloud deployment | Strong negative | −63 | 33 | 3 | 6 | 24 |
| LangSmith | Evaluation, Observability | Model gateway, Orchestration, Tool use, Cloud deployment | Below minimum | none | 19 | 9 | 7 | 3 |
| AI SDK | Orchestration, Tool use | Model gateway, Retrieval, Observability | Below minimum | none | 18 | 8 | 9 | 1 |
| LangGraph | Orchestration | Tool use, Cloud deployment | Below minimum | none | 18 | 10 | 5 | 3 |
| Portkey | Model gateway | Observability, Guardrails | Below minimum | none | 18 | 11 | 4 | 3 |
| Fireworks AI | Cloud deployment | Model APIs, Model gateway, Retrieval | Below minimum | none | 16 | 6 | 5 | 5 |
| Agent Platform | Model APIs, Cloud deployment | Model gateway, Orchestration, Tool use, Retrieval, Evaluation, Observability, Guardrails | Below minimum | none | 13 | 7 | 1 | 5 |
| OpenRouter | Model gateway | Model APIs, Observability, Guardrails | Below minimum | none | 11 | 4 | 4 | 3 |
| Microsoft Foundry | Model APIs, Cloud deployment | Model gateway, Orchestration, Tool use, Retrieval, Evaluation, Observability, Guardrails | Below minimum | none | 8 | 1 | 2 | 5 |
| OpenAI Agents SDK | Orchestration | Tool use, Observability, Guardrails | Below minimum | none | 7 | 3 | 2 | 2 |
| CrewAI | Orchestration | Tool use, Retrieval, Observability, Cloud deployment, Guardrails | Below minimum | none | 4 | 1 | 3 | 0 |
| Together AI | Cloud deployment | Model APIs | Below minimum | none | 1 | 0 | 0 | 1 |
| DeepSeek API | Model APIs | Tool use, Retrieval | Below minimum | none | 0 | 0 | 0 | 0 |
Each author counts once.
Stack Footprint
What is Braintrust used for, and by whom?
AI Observability, also listed under AI engineering, testing, and developer tools.
Model
Model APIs: Touches
Model APIs, touches. Braintrust's built-in models serve playgrounds and scorers, plus Topics and Loop, using monthly model credits. Applications access production models through OpenAI, Anthropic, Google, and the other providers Braintrust records.
Model gateway: Supports
Model gateway, supports. Braintrust Gateway exposes providers through one OpenAI-compatible API with caching and logging. Self-hosted data planes can run it. The hosted Gateway has no SLA and is not generally available, so production routing remains with a dedicated gateway.
Build
Orchestration: Touches
Orchestration, touches. Braintrust traces agents built with LangGraph, the Vercel AI SDK, the OpenAI Agents SDK, the Claude Agent SDK, CrewAI, and other frameworks. It has no agent framework of its own.
Tool use: Supports
Tool use, supports. Braintrust records tool calls and hosts tool and scorer functions that models invoke. Coding agents query logs and experiments through its MCP server. The application's framework owns tool execution.
Data
Retrieval: Touches
Retrieval, touches. Braintrust traces and scores retrieval, including whether an answer uses the retrieved context. The application embeds and indexes its documents and stores them elsewhere.
Evaluate
Evaluation: Core
Evaluation, core. Braintrust tests changes through datasets and experiments in code, the UI, or CI. Teams compare runs side by side using code and LLM scorers, plus human review and playgrounds.
Operate
Observability: Core
Observability, core. SDKs in six languages and OpenTelemetry send traces to Brainstore. Teams search logs and score production traffic there. Topics groups the logs, and SQL thresholds trigger alerts.
Cloud deployment: Touches
Cloud deployment, touches. Braintrust versions and serves prompts and runs its own functions. Enterprise data planes can run in your cloud. Braintrust supplies no application hosting or model capacity.
Guardrails: Touches
Guardrails, touches. Braintrust online scoring runs scorers automatically on production logs for quality and safety analysis. It evaluates recorded conversations and does not sit in the request path to block them.
Used for AI feature development, agent development, testing, and debugging, by AI engineers, product teams, and platform teams.
- Business Area
- Product & Engineering
- Function
- Software Development
- Operating Stack
- AI Application Platform
- Primary Role
- AI Evaluation & Observability
How to read this map. Each layer lists the products that structurally own it, so a product that only touches a layer isn't listed among its owners. Braintrust is underlined in teal where it is Core, and its rows sit a shade lighter; where it supports or touches a layer, that row names it first.
Model
Braintrust: supports on 1 of 2 layers; touches on 1 of 2 layers
Model APIs
Hosted models an application calls for generation, reasoning, and vision, under the provider's rate limits, pricing, and data terms.
- Braintrust touchesAdjacentBraintrust's built-in models serve playgrounds and scorers, plus Topics and Loop, using monthly model credits. Applications access production models through OpenAI, Anthropic, Google, and the other providers Braintrust records.
- Amazon BedrockAI Model APIAmazon BedrockAmazon Web Services+9 MixedAWS service for building on models from Amazon, Anthropic, OpenAI, DeepSeek, and others. AgentCore, Knowledge Bases, and Guardrails add agents, retrieval, and filters. Usage bills per token on AWS.Owns this layerModel APIsOpen product page
- Claude APIAI Model APIClaude APIAnthropic+25 PositiveAnthropic's developer platform for building AI features and agents on Claude models: the Messages API, tool use, prompt caching, batch processing, and the Agent SDK.Owns this layerModel APIsOpen product page
- DeepSeek APIAI Model APIDeepSeek APIDeepSeek+57 Strong positiveDeepSeek's first-party API serves deepseek-flash (DeepSeek-V4.1-Flash-1017) and deepseek-v4-pro with a 1M-token context, in OpenAI- and Anthropic-compatible formats. It bills per million tokens from a prepaid balance, with cheap cache hits and half-price off-peak hours, and stores data in the People's Republic of China.Owns this layerModel APIsOpen product page
- Gemini APIAI Model APIGemini APIGoogle0 MixedGoogle's API for Gemini features and agents, reached with an AI Studio key or the Gemini Enterprise Agent Platform. It supports function calling, built-in tools, context caching, batch, Flex, and Priority inference.Owns this layerModel APIsOpen product page
- Gemini Enterprise Agent PlatformAI Model APIGemini Enterprise Agent PlatformGoogle−27 NegativeGoogle Cloud platform, formerly Vertex AI, for building on Gemini, Claude and open models from Model Garden. The Agent Development Kit, Agent Runtime and RAG Engine add agents and retrieval. Usage bills per token through Google Cloud.Owns this layerModel APIsOpen product page
- Microsoft FoundryAI Model APIMicrosoft FoundryMicrosoft−26 NegativeAzure platform, formerly Azure AI Foundry, for building on OpenAI, Anthropic, Meta, DeepSeek and Microsoft models. Foundry Agent Service, evaluations and model router add agents and routing. Usage bills per token or per provisioned unit on Azure.Owns this layerModel APIsOpen product page
- OpenAI APIAI Model APIOpenAI APIOpenAI+3 MixedOpenAI's platform for teams building AI features and agents on GPT models. The Responses API supports function calling and remote MCP servers. Prompt caching, Batch, and Flex reduce usage costs, and the Agents SDK runs agents.Owns this layerModel APIsOpen product page
Model gateways & routing
One interface in front of several models or providers that routes requests, fails over, and enforces spend and rate limits.
- Braintrust supportsSupportingBraintrust Gateway exposes providers through one OpenAI-compatible API with caching and logging. Self-hosted data planes can run it. The hosted Gateway has no SLA and is not generally available, so production routing remains with a dedicated gateway.
- LiteLLMAI GatewayLiteLLMBerriAI+8 MixedBerriAI's self-hosted gateway and Python SDK expose 100+ model providers through one OpenAI-format API. The MIT-licensed core includes virtual keys, budgets, rate limits, fallbacks, and spend tracking. Enterprise adds SSO and audit logs, plus support.Owns this layerModel gateways & routingOpen product page
- OpenRouterAI GatewayOpenRouterOpenRouter+10 MixedOpenRouter's hosted gateway routes one OpenAI-compatible API to 500+ models from 80+ providers, with provider fallback, per-key credit limits, workspaces, guardrails, and data-policy routing. Tokens bill at provider list price from prepaid credits, with a fee on credit purchases. Stripe agreed to acquire OpenRouter on August 19, 2026.Owns this layerModel gateways & routingOpen product page
- PortkeyAI GatewayPortkeyPalo Alto Networks0 MixedPortkey's gateway puts one OpenAI-compatible API, fallbacks, load balancing, request logs, and guardrails in front of a team's own model API keys, as a hosted service, a hybrid data plane in your VPC, or an MIT-licensed open-source gateway. Palo Alto Networks completed its purchase on May 29, 2026 and sells it as Prisma AIRS AI Gateway.Owns this layerModel gateways & routingOpen product page
- Vercel AI Gateway
Build
Braintrust: supports on 1 of 2 layers; touches on 1 of 2 layers
Orchestration & agent frameworks
Code that composes model calls, tools, memory, and control flow into multi-step workflows and agents.
- Braintrust touchesAdjacentBraintrust traces agents built with LangGraph, the Vercel AI SDK, the OpenAI Agents SDK, the Claude Agent SDK, CrewAI, and other frameworks. It has no agent framework of its own.
- Claude Agent SDKAgent FrameworkClaude Agent SDKAnthropic+52 Strong positiveAnthropic's SDK, formerly the Claude Code SDK, runs the Claude Code binary inside your own service, with built-in tools, subagents, hooks, MCP, permissions, and sessions. It calls Claude models only and is billed at Claude API token prices or against Max and Team plan credits.Owns this layerOrchestration & agent frameworksOpen product page
- CrewAIAgent FrameworkCrewAICrewAI−7 MixedCrewAI's MIT-licensed Python framework builds crews of role-based agents inside event-driven flows, on models from any vendor. CrewAI AMP deploys, traces, and governs them, free for 50 executions a month and custom-priced on Enterprise.Owns this layerOrchestration & agent frameworksOpen product page
- LangGraphAgent FrameworkLangGraphLangChain+25 PositiveLangChain's MIT-licensed runtime builds agents as graphs with shared state in Python and JavaScript, with or without LangChain. It adds checkpoints, human interrupts, and streaming. LangSmith Deployment offers paid hosting.Owns this layerOrchestration & agent frameworksOpen product page
- MastraAgent FrameworkMastraMastra−36 NegativeMastra is a TypeScript framework for agents, typed workflows, memory, and evals, Apache 2.0 outside its ee/ directories. Mastra Platform hosts and traces them: Starter is free with monthly allowances, Teams is US$250 a month, and Enterprise is custom.Owns this layerOrchestration & agent frameworksOpen product page
- OpenAI Agents SDKAgent FrameworkOpenAI Agents SDKOpenAI−17 MixedOpenAI's MIT-licensed SDK runs agents with tools, handoffs, guardrails, sessions, approvals, MCP tools, and tracing inside your own service. It calls OpenAI models through the Responses API by default, reaches other models through beta adapters, and bills at OpenAI API token prices.Owns this layerOrchestration & agent frameworksOpen product page
- AI SDKAgent FrameworkAI SDKVercel+36 PositiveVercel's Apache 2.0 TypeScript toolkit ships on npm as ai. One API covers text, structured output, tools, and agents across 46 first-party model integrations. React, Vue, Svelte, and Angular hooks stream chat. Vercel AI Gateway is the default model route.Owns this layerOrchestration & agent frameworksOpen product page
Tool use & integrations
Letting a model call functions, APIs, and MCP servers, and act on what they return.
- Braintrust supportsSupportingBraintrust records tool calls and hosts tool and scorer functions that models invoke. Coding agents query logs and experiments through its MCP server. The application's framework owns tool execution.
- Claude APIAI Model APIClaude APIAnthropic+25 PositiveAnthropic's developer platform for building AI features and agents on Claude models: the Messages API, tool use, prompt caching, batch processing, and the Agent SDK.Owns this layerTool use & integrationsOpen product page
- Gemini APIAI Model APIGemini APIGoogle0 MixedGoogle's API for Gemini features and agents, reached with an AI Studio key or the Gemini Enterprise Agent Platform. It supports function calling, built-in tools, context caching, batch, Flex, and Priority inference.Owns this layerTool use & integrationsOpen product page
- OpenAI APIAI Model APIOpenAI APIOpenAI+3 MixedOpenAI's platform for teams building AI features and agents on GPT models. The Responses API supports function calling and remote MCP servers. Prompt caching, Batch, and Flex reduce usage costs, and the Agents SDK runs agents.Owns this layerTool use & integrationsOpen product page
- AI SDKAgent FrameworkAI SDKVercel+36 PositiveVercel's Apache 2.0 TypeScript toolkit ships on npm as ai. One API covers text, structured output, tools, and agents across 46 first-party model integrations. React, Vue, Svelte, and Angular hooks stream chat. Vercel AI Gateway is the default model route.Owns this layerTool use & integrationsOpen product page
Data
Braintrust: touches on 1 of 1 layer
Retrieval & grounding
Embedding, indexing, and retrieving the documents and records an application grounds its answers in.
- Braintrust touchesAdjacentBraintrust traces and scores retrieval, including whether an answer uses the retrieved context. The application embeds and indexes its documents and stores them elsewhere.
- pgvectorVector DatabasepgvectorThe pgvector project−29 NegativePostgres extension that stores vectors with the rest of your data and searches them exactly or with HNSW and IVFFlat indexes. Free under the PostgreSQL License, self-installed or on managed hosts such as Neon, Supabase, Cloud SQL, Amazon RDS and Azure.Owns this layerRetrieval & groundingOpen product page
- PineconeVector DatabasePineconePinecone Systems−63 Strong negativeManaged vector database serving full-text, semantic, sparse and hybrid search from one index, with hosted embedding and reranking and namespaces per tenant. Runs on AWS, GCP and Azure, or in your own cloud with Bring Your Own Cloud on Enterprise.Owns this layerRetrieval & groundingOpen product page
- QdrantVector DatabaseQdrantQdrant Solutions GmbH−40 NegativeVector database for dense, sparse and multivector search with JSON payload filters, hybrid search and built-in quantization. Self-hosted under the Apache License 2.0, or Qdrant Cloud on AWS, GCP and Azure with a free tier and usage-based Standard billing.Owns this layerRetrieval & groundingOpen product page
- Voyage AI
Evaluate
Braintrust: core on 1 of 1 layer
Evaluation & testing
Scoring model and agent output against datasets and graders, so a prompt or model change is measured before it ships.
- Braintrust
- ArizeAI ObservabilityArizeArize AI+26 PositiveArize AX and Phoenix trace and evaluate LLM apps and agents over OpenTelemetry. Managed AX adds online evals and monitors, plus Signal. Phoenix is free to self-host under the Elastic License 2.0. Arize has been part of Dynatrace since October 2026.Owns this layerEvaluation & testingOpen product page
- Datadog Agent ObservabilityAI ObservabilityDatadog Agent ObservabilityDatadog+32 PositiveDatadog Agent Observability, sold as LLM Observability until 2026, traces LLM apps and agents and evaluates their output. It shares one trace ID with Datadog APM, infrastructure, and user monitoring. It bills only LLM spans, one per model call, and can be bought without other Datadog products.Owns this layerEvaluation & testingOpen product page
- LangfuseAI ObservabilityLangfuseClickHouse+46 PositiveLangfuse traces and evaluates LLM apps and agents. It versions prompts on Langfuse Cloud or self-hosted under the MIT license. ClickHouse acquired it in January 2026.Owns this layerEvaluation & testingOpen product page
- LangSmithAI ObservabilityLangSmithLangChain+17 MixedLangChain's platform for teams tracing and evaluating LLM applications and agents. It also deploys agents on LangChain's cloud or your infrastructure, with or without the LangChain frameworks.Owns this layerEvaluation & testingOpen product page
Operate
Braintrust: core on 1 of 3 layers; touches on 2 of 3 layers
Observability & tracing
Tracing production requests across model calls and agent steps, with the cost, latency, and errors of each.
- Braintrust
- ArizeAI ObservabilityArizeArize AI+26 PositiveArize AX and Phoenix trace and evaluate LLM apps and agents over OpenTelemetry. Managed AX adds online evals and monitors, plus Signal. Phoenix is free to self-host under the Elastic License 2.0. Arize has been part of Dynatrace since October 2026.Owns this layerObservability & tracingOpen product page
- Datadog Agent ObservabilityAI ObservabilityDatadog Agent ObservabilityDatadog+32 PositiveDatadog Agent Observability, sold as LLM Observability until 2026, traces LLM apps and agents and evaluates their output. It shares one trace ID with Datadog APM, infrastructure, and user monitoring. It bills only LLM spans, one per model call, and can be bought without other Datadog products.Owns this layerObservability & tracingOpen product page
- LangfuseAI ObservabilityLangfuseClickHouse+46 PositiveLangfuse traces and evaluates LLM apps and agents. It versions prompts on Langfuse Cloud or self-hosted under the MIT license. ClickHouse acquired it in January 2026.Owns this layerObservability & tracingOpen product page
- LangSmithAI ObservabilityLangSmithLangChain+17 MixedLangChain's platform for teams tracing and evaluating LLM applications and agents. It also deploys agents on LangChain's cloud or your infrastructure, with or without the LangChain frameworks.Owns this layerObservability & tracingOpen product page
Cloud deployment & capacity
Running model workloads inside a cloud account or region with the throughput, data residency, and billing the business requires.
- Braintrust touchesAdjacentBraintrust versions and serves prompts and runs its own functions. Enterprise data planes can run in your cloud. Braintrust supplies no application hosting or model capacity.
- Amazon BedrockAI Model APIAmazon BedrockAmazon Web Services+9 MixedAWS service for building on models from Amazon, Anthropic, OpenAI, DeepSeek, and others. AgentCore, Knowledge Bases, and Guardrails add agents, retrieval, and filters. Usage bills per token on AWS.Owns this layerCloud deployment & capacityOpen product page
- Fireworks AIInference HostingFireworks AIFireworks AI−17 MixedFireworks AI serves open models through Serverless Inference, multi-tenant inference on Fireworks-managed infrastructure billed per token, and through on-demand deployments on dedicated GPUs billed per GPU second. It offers supervised and reinforcement fine-tuning and serves fine-tuned models for the same price as base models.Owns this layerCloud deployment & capacityOpen product page
- Gemini Enterprise Agent PlatformAI Model APIGemini Enterprise Agent PlatformGoogle−27 NegativeGoogle Cloud platform, formerly Vertex AI, for building on Gemini, Claude and open models from Model Garden. The Agent Development Kit, Agent Runtime and RAG Engine add agents and retrieval. Usage bills per token through Google Cloud.Owns this layerCloud deployment & capacityOpen product page
- Microsoft FoundryAI Model APIMicrosoft FoundryMicrosoft−26 NegativeAzure platform, formerly Azure AI Foundry, for building on OpenAI, Anthropic, Meta, DeepSeek and Microsoft models. Foundry Agent Service, evaluations and model router add agents and routing. Usage bills per token or per provisioned unit on Azure.Owns this layerCloud deployment & capacityOpen product page
- Together AIInference HostingTogether AITogether AI−49 NegativeTogether AI serves open models through a serverless per-token API compatible with the OpenAI SDKs, provisioned throughput with a monthly success commitment, dedicated endpoints billed by the minute, fine-tuning and multi-node GPU clusters. Accounts are fully prepaid, with a minimum US$15 credit purchase.Owns this layerCloud deployment & capacityOpen product page
- Microsoft Foundry
- Groq
Guardrails & AI security
Screening prompts, responses, and agent actions for prompt injection, sensitive data, and policy violations, and blocking or logging what fails.
- Braintrust touchesAdjacentBraintrust online scoring runs scorers automatically on production logs for quality and safety analysis. It evaluates recorded conversations and does not sit in the request path to block them.
- Prisma AIRS
- NeMo Guardrails
