Key findings
- 22 of 24 sources that discuss evaluation praise it, including 19 of 20 team and leadership accounts.
- All 9 sources that discuss SDK and ingestion reliability report problems, the most frequent concern. Braintrust closed all 11 such issues on its own repositories as fixed, a median of 7 days after each report.
- Of 21 platform decisions, 16 chose or kept Braintrust. Dropbox adopted it once spreadsheets broke down. PostHog moved off DeepEval, and Carmenta replaced about 2,000 lines of its own eval code. The other 5 chose Langfuse, local tooling, or custom scripts.
- 5 sources raise data security and control concerns, including keys written into span metadata and a team that keeps Braintrust off until it has a DPA and retention terms.
- The largest public accounts come from single product teams. They include Dropbox Dash on its own engineering blog, Vercel's v0 and Zapier Agents in their own conference talks, and Notion and Trainline in talks co-presented with Braintrust. Ask Braintrust for a reference customer that runs it across several product groups at your scale.
In public accounts from April 2025 to October 2026, Braintrust's evals catch regressions before release, and most teams that weigh it against other tools keep it. The recurring cost is SDK reliability, so pin the SDK and load-test every upgrade before it reaches production. Teams that left cited exhausted credits, account requirements, or a wish to keep traces and prompts together in Langfuse.
What the evals caught
Team accounts describe regressions caught before release. In Zapier's November 2024 talk, which predates the other accounts, scores fell below 80% with 22 regressions when Zapier's Copilot moved from GPT-4 Turbo to GPT-4o. Removing prompt instructions written for the old model recovered the scores. Dropbox Dash runs about 150 judged queries on every pull request in under 10 minutes and blocks the merge on any red-line miss. Notion writes rule-based judges for each example. Its thumbs-down traces showed that research-mode users often wanted a drafted report.
Smaller teams use evals to test prompts. A Hacker News commenter checks prompts for LLM text classification in Braintrust. Mura's co-founder picked it over Vellum, W&B Weave, and Patronus for creating and visualizing experiments. PostHog moved its Max AI evals from DeepEval and added judges that check whether a generated plan answers the question. Supabase links each experiment to its CI run and pull request.
How teams organize the work
People outside engineering work in Braintrust directly. Zapier product managers wrote eval criteria. Notion data specialists curate datasets while product managers and designers read traces. Trainline product managers inspect traces without asking engineers for logs.
Engineers build their own tools on top. Notion built authoring tools on Braintrust's SDK. Mura's CLI owns the domain logic and hands comparison to Braintrust. Public pull requests at Sefaria add dataset-tagging jobs, and one at HolmesGPT exposes about 53 LLM tests as remote evals that run from the playground.
Where it hurts
Most complaints are issues filed on Braintrust's own repositories. A TypeScript upgrade slowed evals from 1.5 to 5 minutes. Other reports describe ingestion 500 errors, per-turn logging that crashed a Cloudflare Workers agent, a Ruby runner that added 10 to 25 seconds per case, and a broken npm bundle. Braintrust fixed nearly all of them within weeks.
Two issues reported keys in logs. Logging a span object exposed a Braintrust key, and the Agno integration wrote a provider key into tool-span metadata. Self-hosters reported unexpected traffic to braintrust.dev and vulnerable dependencies in the published images. Separately, Braintrust's May 2026 incident exposed provider keys stored on the platform.
Why some teams leave, and why most stay
Teams that left cited cost, control, or a wish to run fewer tools. One developer ran out of write credits and moved to a local store and runner. A fuzzing project removed Braintrust so its evals run without an account. Two projects moved to Langfuse, one to keep traces and evals next to its prompts. A June 2025 Hacker News poster stayed on custom scripts after finding the docs sparse and the product young.
Teams that stayed value Braintrust's experiment pages and having less code to maintain. A homelab operator tail-samples OpenTelemetry traces into Braintrust. A startup proposed replacing its in-house tracing SDK with it. GStack replaced its internal benchmark scoring with Braintrust Eval. One commenter who liked it said they would now vibe-code the same features themselves.
What to settle before choosing
Answer these with your own traffic before the pilot ends. In these accounts, those decisions settled the outcome more often than feature lists did.
- Who besides engineers will write criteria, curate datasets, and read traces?
- How many gigabytes and scores does a week of production traffic produce?
- Must traces stay in your own cloud, and can you buy Enterprise for BYOC or self-hosting?
- How will provider keys be held: workload identity, assume-role, or stored keys with provider-side limits?
- Which SDK version is pinned, and is every upgrade load-tested?
Methodology
GILD reviewed 50 public Braintrust sources, 41 on October 12, 2026, 5 more on October 25, 2026, and 4 engineering accounts on October 26, 2026, all published in 2025 or later. They include GitHub issues and pull requests, Hacker News comments and posts, engineering and personal blogs, and talks. The rest are a Reddit post, a Product Hunt review, and an independent tool evaluation. Braintrust staff co-presented two of the customer talks at AI Engineer events. Zapier's 2024 talk adds context and sits outside the count. Every count comes from these records. Issue-resolution times come from GitHub creation and close dates. GILD searched Hacker News through Algolia, Reddit through Arctic Shift, and GitHub, and kept 30 GitHub issue and pull-request accounts. GILD checked product behavior against Braintrust's docs on October 12, 2026. On October 26, 2026, GILD searched company engineering blogs and conference talks and found team-level accounts. In a pilot, test shared dataset ownership and access controls across teams.
Products in this piece
Sources
- Public source
- 29
- Official documentation
- 4
- Olmo Maldonado (Zapier) with Ankur Goyal (Braintrust): How Zapier Builds AI Products and Features with the Help of Braintrust, AI Engineer World's Fair 2024 (Nov 7, 2024)
- Dropbox Tech blog: A practical blueprint for evaluating conversational AI at scale (Oct 2, 2025)
- Sarah Sachs (Notion) with Carlos Esteban and Doug Guthrie (Braintrust): How to build world-class AI products, AI Engineer World's Fair 2025 (Jun 27, 2025)
- Oussama Hafferssas and Mayank Soni (Trainline) with Giran Moodley (Braintrust): Shipping complex AI applications, AI Engineer Europe 2026 (May 1, 2026)
- James Mackey (Mura): BOLT, How Mura wrote an in-house LLM Eval Framework (Oct 21, 2025)
- PostHog pull request #31964: migrate Max AI insight evals from DeepEval to Braintrust (May 6, 2025)
- supabase/evals pull request #384: link Braintrust experiments to their CI run and pull request (Oct 8, 2026)
- Carmenta pull request #145: replace custom eval code with Braintrust Eval() (Dec 11, 2025)
- Hacker News comment on rebuilding Braintrust's eval loop in house (Mar 17, 2026)
- Ask HN: What tools are you using for AI evals? Everything feels half-baked (Jun 5, 2025)
- Sefaria pull request #3076: Braintrust log backup and dataset tagging jobs (Feb 4, 2026)
- HolmesGPT pull request #1312: run HolmesGPT LLM tests as remote Braintrust evals (Jan 3, 2026)
- braintrust-sdk-javascript issue #1394: eval runs slowed from 1.5 to 5 minutes after the 1.0 upgrade (Feb 19, 2026)
- braintrust-sdk-javascript issue #1414: regular 500 responses from the log ingestion endpoint (Feb 27, 2026)
- braintrust-sdk-javascript issue #2353: per-turn conversation logging exhausts Cloudflare Workers memory (Aug 11, 2026)
- braintrust-sdk-ruby issue #210: per-case flush adds 10 to 25 seconds to Ruby evals (Aug 19, 2026)
- braintrust-sdk-javascript issue #912: logging a span object writes the API key into the log (Sep 7, 2025)
- braintrust-sdk-python issue #733: Agno tool spans leak the provider API key into span metadata (Sep 3, 2026)
- bt issue #384: self-hosted daemon contacts braintrust.dev on cold start (Sep 25, 2026)
- helm issue #86: self-hosted data plane images ship a vulnerable starlette (May 28, 2026)
- julianken, bird-sight-system issue #1094: replace Braintrust after exhausting write credits (Jun 12, 2026)
- Ultrafuzz issue #1133: remove Braintrust reporting so evaluations run without an account (Sep 22, 2026)
- Adam Belton, pull request #26: migrate ThoughtForm observability from Braintrust to Langfuse (Aug 7, 2026)
- Create Something pull request #783: retire Braintrust for Langfuse (Jul 6, 2026)
- Jerred Shepherd, pull request #2827: tail-sample LLM traces into Braintrust over OpenTelemetry (Sep 10, 2026)
- GigaFlow pull request #11: trace agent simulations to Braintrust instead of an in-house SDK (Sep 17, 2026)
- GStack pull request #13: back the model benchmark with Braintrust and drop in-house scoring (Jul 21, 2026)
- Hamel Husain: AI Evals, Everything You Need to Know (Sep 18, 2026)
- Howie Liu (Airtable) on Product Hunt: Review of Braintrust (Apr 9, 2025)
- Braintrust docs: Experiments
- Braintrust docs: Run experiments in CI/CD
- Braintrust docs: Protect sensitive data
- Braintrust Trust Center: Updates, including the May 2026 security incident reports