Braintrust+50 Strong positive

Operator Insights

What are teams learning from building on Braintrust and running it in production?

12 accounts: 2 left it or turned it down, 1 switched, 1 evaluated it, 7 adopted it and 1 reported by others

Left it or turned it down2

createsomethingtodayMaintainer, create-something-monorepoCREATE SOMETHING
CREATE SOMETHING's monorepo of MCP services, Dify agents, evals and telemetry.
GitHub, createsomethingtoday/create-something-monorepo, published Jul 6, 2026Read the write-up (opens GitHub, createsomethingtoday/create-something-monorepo in a new tab)
Observation
CREATE SOMETHING left Braintrust for Langfuse across its evals and telemetry, then reran its eval suites on Langfuse with none failing.
Takeaway
Keep scorers and suites in your own repository, so a platform change is a rerun of the same suites on the new backend.
What happened
A pull request merged Jul 6, 2026 replaced every active Braintrust eval, telemetry, package, script, environment and workflow surface with Langfuse equivalents, and archived the Braintrust artifacts as internal history. Its validation ran the MCP and Dify eval sets on Langfuse, 9 suites each with 0 failed. The pull request does not say why the team left.
- Public engineering write-up (GitHub pull request)
- Team metrics reported
Deployment details
CREATE SOMETHING's monorepo of MCP services, Dify agents, evals and telemetry.
- Public engineering write-up (GitHub pull request)
- Team metrics reported
Team
CREATE SOMETHING maintainers
Workload
Evals and telemetry for MCP services and Dify agents
Product version
Braintrust evals and SDK telemetry, replaced by Langfuse
Measurement
9 eval suites rerun on Langfuse with 0 failed, for each of two eval sets

MonadSep 22, 2026GitHub, monad-developers/ultrafuzzUltrafuzz stopped using Braintrust for reporting and judging after cutting paid CI, so its evaluations run locally without a Braintrust account.
aviggianoCollaborator, UltrafuzzMonad
Ultrafuzz, a fuzzing project whose benchmark evaluations reported to Braintrust and judged through its gateway.
GitHub, monad-developers/ultrafuzz, published Sep 22, 2026Read the write-up (opens GitHub, monad-developers/ultrafuzz in a new tab)
Observation
Ultrafuzz stopped using Braintrust for reporting and judging after cutting paid CI, so its evaluations run locally without a Braintrust account.
Takeaway
Keep evals runnable without the platform account, so a budget cut removes the upload path and leaves scoring in place.
What happened
An issue opened and closed on Sep 22, 2026 removed the Braintrust reporter, credential forwarding and the default gateway.braintrust.dev judge endpoint, after a paid-CI cleanup took BRAINTRUST_API_KEY out of the repository's Actions secrets. Local evaluation, reports and scoring stayed, and optional LLM judging now needs an endpoint the team configures itself.
Public engineering write-up (GitHub issue)
No team metrics reported
Deployment details
Ultrafuzz, a fuzzing project whose benchmark evaluations reported to Braintrust and judged through its gateway.
Public engineering write-up (GitHub issue)
No team metrics reported
Team
Ultrafuzz maintainers
Workload
Benchmark evaluation reports and optional LLM judging
Product version
Braintrust reporting and the Braintrust gateway as judge endpoint

Switched to or from it1

Michael MatlokaEngineerPostHog
PostHog Max AI, which turns a user's question into a plan and then an analytics insight query.
GitHub, PostHog/posthog, published May 6, 2025Read the write-up (opens GitHub, PostHog/posthog in a new tab)
Observation
PostHog migrated the evals for its Max AI insight queries from DeepEval to Braintrust and added LLM classifiers that check the plan and the query.
Takeaway
Score the plan and the query separately, so a regression shows which step of the agent broke.
What happened
A pull request opened May 6, 2025 and merged May 26, 2025 moved the SQL and trends insight evals from DeepEval to Braintrust. It added two classifiers: one judges whether the generated plan answers the question, against an expected plan, and one rates how well the final query follows that plan. A second part moved the remaining insight types.
- Public engineering write-up (GitHub pull request)
- No team metrics reported
Deployment details
PostHog Max AI, which turns a user's question into a plan and then an analytics insight query.
- Public engineering write-up (GitHub pull request)
- No team metrics reported
Team
PostHog Max AI engineers
Duration
Opened May 6, 2025; merged May 26, 2025
Workload
Evals for generated SQL and trends insights
Product version
Braintrust evals with LLM classifiers, replacing DeepEval

Evaluated it1

Antaripa Saha and Hamel HusainEvaluation practitioners and co-authorsParlance Labs
An evaluation of three platforms' built-in agents on 100 traces from a production apartment-leasing assistant, labelled by hand with a domain expert and then masked.
Parlance Labs blog, published Jul 11, 2026Read the write-up (opens Parlance Labs blog in a new tab)
Observation
Parlance Labs ran Braintrust Loop, Arize Alyx and LangSmith on the same 100 hand-labelled production traces. On this dataset Loop found the most failures, and the authors found it easy to use from upload to export.
Takeaway
Run Loop against your own labelled traces before relying on it, since one 100-trace dataset cannot rank the vendors, and keep a person reviewing for the failures that need business context.
What happened
Parlance Labs masked its hand labels and asked each platform's agent to find the failures on its own (write-up of Jul 11, 2026). Braintrust Loop had the highest recall of the three, and the authors uploaded traces, ran the review and exported the findings in one pass. LangSmith Engine surfaced too few issues for the 100-trace comparison, so the authors scored LangSmith's chat agent instead. Every system missed failures that needed business context, and Claude Code, Codex and Factory Droid, given the same prompt and traces, were competitive with the platforms.
“First, we prepared a dataset of 100 traces from a production apartment-leasing AI, working with a domain expert to label failures by hand. Then we masked those labels and asked each system to review the traces and find the errors on its own.”
Antaripa Saha and Hamel Husain
- Public practitioner write-up
- Team metrics reported
Deployment details
An evaluation of three platforms' built-in agents on 100 traces from a production apartment-leasing assistant, labelled by hand with a domain expert and then masked.
- Public practitioner write-up
- Team metrics reported
Team
Two Parlance Labs authors and a domain expert
Workload
Finding the failures in 100 masked production traces
Product version
Braintrust Loop, Arize Alyx, and LangSmith Engine and chat agent
Measurement
Recall and precision against the hand labels; Loop had the highest recall of the three platforms

Adopted it7

RoxOct 14, 2026Rox engineering blogRox has Devin turn about 350 production conversations a week from Braintrust traces into evals. The first sweep covered 67 behaviors, where 142 hand-written evals had covered 34.
Brian XuEngineer, Agents teamRox
Rox's Agents team, which runs its production agent traces through Braintrust and reviews a weekly pull request of new evals. Rox published the post in partnership with Cognition and Braintrust.
Rox engineering blog, published Oct 14, 2026Read the write-up (opens Rox engineering blog in a new tab)
Observation
Rox has Devin turn about 350 production conversations a week from Braintrust traces into evals. The first sweep covered 67 behaviors, where 142 hand-written evals had covered 34.
Takeaway
Build the eval suite from production traces, and read every failing verdict before you trust it, because agent runs vary.
What happened
Each week Devin pulls about 350 conversations from Braintrust, labels them, builds evals with synthetic data in per-branch environments and opens a pull request the team reviews (Rox, Oct 14, 2026). Before the first sweep Rox had 142 hand-written evals covering 34 behaviors; the first sweep's suite covered 67 behaviors and 72% of conversations in each of the next two weeks. About 1 in 8 evals passes some runs and fails others with no code change, and in 36 of 40 failing verdicts the team reviewed, the judge applied its criterion correctly. Production bugs now show up as failing evals in that week's pull request.
“Every week, Devin, the coding agent from Cognition, reads real conversations on Rox from our production traces in Braintrust, turns them into evals, and opens a pull request that we review.”
Brian Xu
Public engineering write-up
Team metrics reported
Deployment details
Rox's Agents team, which runs its production agent traces through Braintrust and reviews a weekly pull request of new evals. Rox published the post in partnership with Cognition and Braintrust.
Public engineering write-up
Team metrics reported
Team
Rox Agents team
Duration
Weekly sweeps, with two weeks measured after the first
Workload
Evals for Rox's production agent, written by Devin from Braintrust traces
Product version
Braintrust tracing, read by a Devin skill
Measurement
67 behaviors covered against 34; 72% of conversations covered; 36 of 40 failing verdicts applied correctly
DropboxOct 2, 2025Dropbox Tech blogDropbox's Dash team adopted Braintrust once its scattered eval artifacts became unmanageable, and now runs about 150 judged queries on every pull request in under 10 minutes.
Ranjitha Gurunath KulkarniDropbox Dash engineer, co-author with Ameya Bhatawdekar and Gonzalo GarciaDropbox
Dropbox Dash, a conversational AI search product, evaluated on datasets built partly from employees' dogfooding logs.
Dropbox Tech blog, published Oct 2, 2025Read the write-up (opens Dropbox Tech blog in a new tab)
Observation
Dropbox's Dash team adopted Braintrust once its scattered eval artifacts became unmanageable, and now runs about 150 judged queries on every pull request in under 10 minutes.
Takeaway
Seed eval sets from your own employees' real use, and make a red-line miss block the merge.
What happened
Dropbox's engineering blog of Oct 2, 2025 says the Dash team adopted Braintrust to manage datasets, scorers and experiments once scattered artifacts became unmanageable. Eval sets come partly from production logs of Dropbox employees dogfooding Dash. Every pull request runs about 150 judged queries in under 10 minutes, a miss on any red-line check blocks the merge, and one click opens the trace behind a score.
“…we turned to internal datasets by collecting production logs from Dropbox employees dogfooding Dash.”
Ranjitha Gurunath Kulkarni
Public engineering write-up
Team metrics reported
Deployment details
Dropbox Dash, a conversational AI search product, evaluated on datasets built partly from employees' dogfooding logs.
Public engineering write-up
Team metrics reported
Team
Dropbox Dash AI engineering
Workload
Conversational search answers, judged on every pull request
Product version
Braintrust datasets, scorers, experiments and automation
Measurement
About 150 judged queries per pull request in under 10 minutes
MuraOct 21, 2025James Mackey's newsletter on SubstackMura's co-founder built the startup's eval CLI on Braintrust's datasets and experiments after finding other eval tools immature or too opinionated about code structure.
James MackeyCo-founderMura
Mura, seed-stage startup automating billing for commercial HVAC providers. Field-level extraction from emails, PDFs, photos, and work orders.
James Mackey's newsletter on Substack, published Oct 21, 2025Read the write-up (opens James Mackey's newsletter on Substack in a new tab)
Observation
Mura's co-founder built the startup's eval CLI on Braintrust's datasets and experiments after finding other eval tools immature or too opinionated about code structure.
Takeaway
Own the domain logic of your evals, such as what expected means and how the app runs, and buy the storage, comparison, and visualization.
What happened
Mura spent 10 to 20 minutes evaluating each prompt experiment, yet parsing errors still reached customers. It tried Vellum, W&B Weave, and Patronus, then built BOLT. The CLI sends entity test cases to one Braintrust dataset per environment and runs the app concurrently. Each extracted field becomes a separate eval, scored by field type. Expected values come from user-approved results in Mura. During a migration from GPT-4 to GPT-5, BOLT showed GPT-5-mini performed as well as GPT-5 for one use case, saving more than 80% of its cost. Mackey keeps domain logic in BOLT and uses Braintrust for visualization and comparison.
“Braintrust’s Experiments visualization would have been very hard for Mura to replicate internally…”
James Mackey
Public engineering write-up
Team metrics reported
Models named
Deployment details
Mura, seed-stage startup automating billing for commercial HVAC providers. Field-level extraction from emails, PDFs, photos, and work orders.
Public engineering write-up
Team metrics reported
Models named
Team
Mura engineering, with developers as BOLT's primary users
Duration
About a year of building AI features before the write-up
Workload
Field suggestions for HVAC billing from messy inputs, reviewed by users
Product version
Braintrust Datasets and Experiments, driven by Mura's BOLT CLI
Model
GPT-4, GPT-5, and GPT-5-mini during a model migration
Measurement
Field-level accuracy against user-approved values; 10 to 20 minutes per experiment before BOLT; more than 80% cost saving on one use case
NotionJun 27, 2025AI Engineer on YouTubeNotion's AI team spends most of its AI work on evals and usage review in Braintrust, and moves a prompt to a new model only where that prompt's evals pass.
Sarah SachsAI LeadNotion
Notion AI: search, Q&A, meeting notes, and research mode. More than 100 million Notion users overall. Evals run before relevant changes ship. They do not run on every CI run.
AI Engineer on YouTube, published Jun 27, 2025Watch the talk (opens AI Engineer on YouTube in a new tab)
Observation
Notion's AI team spends most of its AI work on evals and usage review in Braintrust, and moves a prompt to a new model only where that prompt's evals pass.
Takeaway
Write expectations as rules a scoped judge can check, inspect the judge's failures, and let product people read the same traces engineers do.
What happened
Sachs estimates the team spends 10% of its time prompting and 90% evaluating, improving evaluations, and inspecting usage in Braintrust. Data specialists curate the datasets, and Sachs advises starting a new capability, such as a Jira connector for search, with about ten examples. They use deterministic scorers where possible and rule-based LLM judges, sometimes per example, and skip frozen answers. Product managers and designers read thumbs-down traces directly. Those traces showed research-mode users often wanted a drafted report. New models usually reach users in under a day, with each prompt switched after its evals pass.
Conference talk (AI Engineer conference 2025, co-presented with Braintrust)
Team practice reported
Models not named
Deployment details
Notion AI: search, Q&A, meeting notes, and research mode. More than 100 million Notion users overall. Evals run before relevant changes ship. They do not run on every CI run.
Conference talk (AI Engineer conference 2025, co-presented with Braintrust)
Team practice reported
Models not named
Team
Notion AI team: engineers, data specialists, product managers, and designers
Workload
Notion AI features, including search with connectors, Q&A, and research mode
Product version
Braintrust SDK, with Notion's own authoring layers on top
Measurement
Time from a new model's release to production users, under a day
ZapierNov 7, 2024AI Engineer on YouTubeZapier grew its AI Zap Builder tests from 7 run by hand to more than 800 run on every merge request. Its evals caught a quality drop when Copilot moved to a faster model.
Olmo MaldonadoSenior AI engineerZapier
Zapier AI Zap Builder and Copilot. Workflows across more than 7,000 apps. Zapier was Braintrust's first user.
AI Engineer on YouTube, published Nov 7, 2024Watch the talk (opens AI Engineer on YouTube in a new tab)
Observation
Zapier grew its AI Zap Builder tests from 7 run by hand to more than 800 run on every merge request. Its evals caught a quality drop when Copilot moved to a faster model.
Takeaway
Let product managers write the eval criteria, run the suite on every merge request, and expect prompts tuned to one model to regress on the next.
What happened
Zapier product managers set criteria for choosing triggers and actions, including support for the top 25 apps. The team seeded Braintrust datasets and graded runs in CI with logic-based and LLM graders. Maldonado reports nearly 300% better accuracy, without a baseline. When Zapier's Copilot moved from GPT-4 Turbo to GPT-4o for speed, scores that had been 80% or better in most areas fell below 80%, and the comparison showed 22 regressions. The team traced the drop to prompts tuned to GPT-3.5 Turbo, parameters it had added, and the deprecated functions API, and changed them to recover.
Conference talk (AI Engineer conference 2024, co-presented with Braintrust)
Team metrics reported
Deployment details
Zapier AI Zap Builder and Copilot. Workflows across more than 7,000 apps. Zapier was Braintrust's first user.
Conference talk (AI Engineer conference 2024, co-presented with Braintrust)
Team metrics reported
Team
Zapier AI team, with product managers defining eval criteria
Workload
AI Zap Builder, which generates workflows from prompts, and Copilot in the Zap editor
Product version
Braintrust datasets, experiments, tracing, and playgrounds
Model
GPT-3.5 Turbo, GPT-4 Turbo, and GPT-4o at different stages
Measurement
Test count, 7 to more than 800, and eval scores before and after the model switch
SupabaseOct 8, 2026GitHub, supabase/evalsSupabase links each CI-triggered Braintrust experiment in its evals repository to the GitHub Actions run, and to its pull request when one triggered it.
Matt RossmanAI tooling engineerSupabase
Supabase's evals repository, which runs coding agents against Supabase tasks and records each run as a Braintrust experiment.
GitHub, supabase/evals, published Oct 8, 2026Read the write-up (opens GitHub, supabase/evals in a new tab)
Observation
Supabase links each CI-triggered Braintrust experiment in its evals repository to the GitHub Actions run, and to its pull request when one triggered it.
Takeaway
Tie every CI experiment to the run and pull request behind it, so a failing score leads straight to the change that caused it.
What happened
A pull request opened Oct 8, 2026 and merged Oct 9, 2026 adds the CI run URL to every Braintrust experiment and lists the related pull requests in each GitHub Actions job summary, so the team can go from a trace to its CI logs and from a run to its source. The CI link is added only when the run has GitHub environment metadata, and pull request details exist only for runs a pull request triggers. Its example experiment ran Codex on GPT-6 Luna.
Public engineering write-up (GitHub pull request)
No team metrics reported
Model named
Deployment details
Supabase's evals repository, which runs coding agents against Supabase tasks and records each run as a Braintrust experiment.
Public engineering write-up (GitHub pull request)
No team metrics reported
Model named
Team
Supabase AI tooling
Workload
Coding-agent evals on Supabase tasks, run in GitHub Actions
Product version
Braintrust experiments with CI metadata
Model
GPT-6 Luna, through Codex, in the pull request's example experiment
Nicolas BustamanteJan 24, 2026nicolasbustamante.comFintool evaluates every model, prompt and skill change against a Braintrust test set, and its auto-filed production bugs link to the Braintrust traces.
Nicolas BustamanteFounderFintool
Fintool, which builds AI agents for financial research and uses Braintrust for experiment tracking and LLM call tracing.
nicolasbustamante.com, published Jan 24, 2026Read the write-up (opens nicolasbustamante.com in a new tab)
Observation
Fintool evaluates every model, prompt and skill change against a Braintrust test set, and its auto-filed production bugs link to the Braintrust traces.
Takeaway
Gate every model and prompt change on a fixed test set, and put the trace link in the bug report so a fix starts from the failing run.
What happened
Bustamante wrote on Jan 24, 2026 that every model, prompt and skill change is evaluated against about 2,000 test cases, and a pull request is blocked when the eval score drops more than 5%. Production errors file GitHub issues automatically, with the conversation ID, traceback and links to the Braintrust traces and Temporal workflows.
“We use Braintrust for experiment tracking. Every model change, every prompt change, every skill change gets evaluated against a test set.”
Nicolas Bustamante
Public practitioner write-up
Team metrics reported
Deployment details
Fintool, which builds AI agents for financial research and uses Braintrust for experiment tracking and LLM call tracing.
Public practitioner write-up
Team metrics reported
Team
Fintool engineering
Workload
Financial research agents and their skills
Product version
Braintrust experiments and tracing
Measurement
About 2,000 test cases; a score drop of more than 5% blocks a pull request

Reported by others1

Jaime BlascoMay 6, 2026TechCrunchNudge Security was among the Braintrust customers emailed in May 2026 to rotate every AI provider key stored on the platform, after an attacker reached a backup holding those keys.
Jaime BlascoCo-founderNudge Security
Braintrust customers that stored organization-level AI provider keys in Braintrust for playgrounds, scorers and the Gateway.
TechCrunch, published May 6, 2026Read the report (opens TechCrunch in a new tab)
Observation
Nudge Security was among the Braintrust customers emailed in May 2026 to rotate every AI provider key stored on the platform, after an attacker reached a backup holding those keys.
Takeaway
Connect providers through workload identity federation or assume-role where Braintrust supports it, and put a provider-side spend limit on every key you still store.
What happened
TechCrunch reported on May 6, 2026 that a Braintrust email sent to customers that Monday confirmed unauthorized access to one of its AWS accounts, which held API keys customers use to reach AI models, and asked every customer to rotate the keys they store with Braintrust. Blasco, who received the email, told TechCrunch the incident could have downstream implications for affected customers. Braintrust disclosed the incident on its website on the Tuesday, and a spokesperson told TechCrunch the company had confirmed a security incident but saw no evidence of a breach at that time.
“Jaime Blasco, the co-founder of cybersecurity startup Nudge Security who received a breach email alert from Braintrust, told TechCrunch that the incident could have “downstream implications for affected customers,”…”
Lorenzo Franceschi-Bicchierai, TechCrunch
Second-hand news report (TechCrunch, from a Braintrust customer email)
No team metrics reported
Deployment details
Braintrust customers that stored organization-level AI provider keys in Braintrust for playgrounds, scorers and the Gateway.
Second-hand news report (TechCrunch, from a Braintrust customer email)
No team metrics reported
Team
Every Braintrust customer storing AI provider keys on the platform
Duration
Customer email on Monday, May 4, 2026; website disclosure on Tuesday, May 5
Workload
Stored provider keys used by playgrounds, scorers and the Gateway
Product version
Braintrust's control plane, which holds organization-level provider secrets for every deployment

Left it or turned it down2

CREATE SOMETHING left Braintrust for Langfuse across its evals and telemetry, then reran its eval suites on Langfuse with none failing.

Ultrafuzz stopped using Braintrust for reporting and judging after cutting paid CI, so its evaluations run locally without a Braintrust account.

Switched to or from it1

PostHog migrated the evals for its Max AI insight queries from DeepEval to Braintrust and added LLM classifiers that check the plan and the query.

Evaluated it1

Parlance Labs ran Braintrust Loop, Arize Alyx and LangSmith on the same 100 hand-labelled production traces. On this dataset Loop found the most failures, and the authors found it easy to use from upload to export.

Adopted it7

Rox has Devin turn about 350 production conversations a week from Braintrust traces into evals. The first sweep covered 67 behaviors, where 142 hand-written evals had covered 34.

Dropbox's Dash team adopted Braintrust once its scattered eval artifacts became unmanageable, and now runs about 150 judged queries on every pull request in under 10 minutes.

Mura's co-founder built the startup's eval CLI on Braintrust's datasets and experiments after finding other eval tools immature or too opinionated about code structure.

Notion's AI team spends most of its AI work on evals and usage review in Braintrust, and moves a prompt to a new model only where that prompt's evals pass.

Zapier grew its AI Zap Builder tests from 7 run by hand to more than 800 run on every merge request. Its evals caught a quality drop when Copilot moved to a faster model.

Supabase links each CI-triggered Braintrust experiment in its evals repository to the GitHub Actions run, and to its pull request when one triggered it.

Fintool evaluates every model, prompt and skill change against a Braintrust test set, and its auto-filed production bugs link to the Braintrust traces.

Reported by others1

Nudge Security was among the Braintrust customers emailed in May 2026 to rotate every AI provider key stored on the platform, after an attacker reached a backup holding those keys.