Skip to main content

Decision brief

Braintrust Review: Evals and Tracing for Teams Shipping AI Features

Choose Braintrust when evals gate every release, as Dropbox and Fintool run it, and product people grade outputs too. Set keys, region, and spend alerts before production.

Reading time
3 min read
Sources
20 cited
In this piece
Braintrust+50 Strong positive

GILD verdict

Choose Braintrust when evals decide what ships and product managers or specialists grade the same traces and experiments as engineers, as at Dropbox, Fintool and Notion.

Before production, mask sensitive fields, connect providers without stored keys where you can, pick the data plane region, and set spend alerts.

Braintrust

AI Observability by Braintrust

+50 Strong positive
Product page

Fit and limits

Who it fits

  • Your team ships prompt and model changes, plus code, often. Each change needs dataset scores in CI and online scoring after release.
  • Product managers and designers, or domain specialists, need direct trace access. They must label outputs and curate datasets without engineering log exports.
  • Many people need access: every plan includes unlimited users, so cost tracks data and scores whatever the headcount.
  • The application spans languages or frameworks: SDKs cover Python, TypeScript, Go, Java, Ruby, and C#, and OpenTelemetry traces are accepted.
  • An eval agent must surface as many failures as it can: in Parlance Labs' July 2026 test on 100 hand-labelled production traces, Braintrust Loop had the highest recall of three platforms, ahead of Arize Alyx and LangSmith.

Where it falls short

  • Braintrust counts processed data at ingestion and has no hard spend cap. Verbose agent traces incur charges even if you delete them later.
  • BYOC and a self-hosted data plane need Enterprise, and even then the control plane, login, and project metadata stay with Braintrust. An early-stage team on Reddit ruled Braintrust out for that reason in October 2026.
  • Starter and Pro sign in with Google only; SAML or OIDC SSO, custom permission groups, audit logs, export automations, and a BAA need Enterprise.
  • Provider keys stored in Braintrust were exposed in a May 2026 incident, and 30 customers reported suspicious use before Braintrust's May 29 report closed the investigation. The short-lived credentials added since cover OpenAI, Anthropic, Vertex AI, Azure AI Foundry, and Bedrock, and only for Braintrust-hosted organizations.

Implementation reality

  1. Instrument one feature first

    Choose the US or EU data plane when creating the organization. You cannot change it later. Set BRAINTRUST_API_KEY and initialize a project per application. Wrap provider clients or enable automatic instrumentation. Existing OpenTelemetry pipelines can use an OTLP exporter or the Braintrust span processor.

  2. Keep sensitive data out

    Mask secrets and personal data in the SDK or span customizers before upload. Ingestion redaction runs after upload and is available to limited customers. It sends selected text to a Baseten-hosted model outside the EU, even for EU organizations. Use application-side masking for data that must never leave.

  3. Turn failures into a gate

    Add production failures to a dataset. Use code scorers for structure and LLM judges for criteria. Run experiments on every pull request with the GitHub Action or bt eval. Agree on scorer thresholds first and review a share of judge scores by hand.

  4. Score production and watch spend

    Set a sampling rate for online scoring and alerts for errors and quality. Configure Billing spend alerts through email or Slack. Braintrust has no usage cap that stops charges. From September 1, 2026, Billing's usage analytics splits logs, scores, and Topics spend by project.

What to measure in your pilot

  • Hand-grade 100 traces per use case and compare them with the judge; Zheng et al. measured over 80% agreement for GPT-4 judges, so a judge below that should not gate releases.
  • GILD found no customer bill in Braintrust's docs or public posts that splits data from scores. Export a week of usage by project, and cut the scoring rate if scores pass half.
  • Log every model provider your scorers call for a month; each one without keyless access keeps a stored key, so check its scope and rotation before production traffic starts.
  • Teams leave without saying why: CREATE SOMETHING's July 2026 move to Langfuse names no reason. Keep scorers and suites in your own repository, so a platform change means rerunning the same suites.

Sources

Official documentation
19
Public source
1
  1. Braintrust pricingOfficial documentationbraintrust.dev
  2. Braintrust docs: Plans and limitsOfficial documentationbraintrust.dev
  3. Braintrust docs: Billing FAQOfficial documentationbraintrust.dev
  4. Braintrust docs: Monitor usage and spend alertsOfficial documentationbraintrust.dev
  5. Braintrust docs: Configure data retentionOfficial documentationbraintrust.dev
  6. Braintrust docs: Deployment options (SaaS, BYOC, self-hosted)Official documentationbraintrust.dev
  7. Braintrust docs: Architecture (control plane and data plane)Official documentationbraintrust.dev
  8. Braintrust docs: Organizations and data plane regionOfficial documentationbraintrust.dev
  9. Braintrust docs: SecurityOfficial documentationbraintrust.dev
  10. Braintrust docs: Protect sensitive dataOfficial documentationbraintrust.dev
  11. Braintrust docs: Authentication, API keys, and provider credentialsOfficial documentationbraintrust.dev
  12. Braintrust docs: ExperimentsOfficial documentationbraintrust.dev
  13. Braintrust docs: Run experiments in CI/CDOfficial documentationbraintrust.dev
  14. Braintrust docs: Score production tracesOfficial documentationbraintrust.dev
  15. Braintrust docs: Trace LLM callsOfficial documentationbraintrust.dev
  16. Braintrust docs: Send OpenTelemetry traces and logsOfficial documentationbraintrust.dev
  17. Braintrust Master Subscription AgreementOfficial documentationbraintrust.dev
  18. Braintrust Trust Center: Updates, including the May 2026 security incident reportsOfficial documentationtrust.braintrust.dev
  19. Braintrust docs: Product updates (workload identity federation May and June 2026, key-rotation warnings May 2026, usage analytics by project Sep 2026, organization API key listing Oct 2026)Official documentationbraintrust.dev
  20. arXiv: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Dec 24, 2023)Public sourcearxiv.org

Keep reading

  1. Buyer guideBraintrust Evaluation & Tracing PlaybookHow to instrument an LLM feature in Braintrust and gate each release on evals, with the practices Notion and Zapier have published.
  2. Buyer guideBraintrust Usage, Spend & Retention ChecklistBraintrust has no hard spending limit, so control the bill with spend alerts and scoring sampling rates. Size each charge from real traffic.
  3. Buyer guideBraintrust Data Plane, Hosting & Security ChecklistChoose a US or EU region, BYOC, or self-hosting before launch. Then move model API keys to short-lived credentials and set access and redaction.
Read the research