Skip to main content
BraintrustProduct page

Braintrust Signals

What teams building AI features on Braintrust report, with every finding tied to its evidence.

Read the findings

What the evidence says

Whose sources count

Practitioner sentiment, teams and leaders

+50 Strong positive from 34 sources that take a side

How this figure is made

Strong positive overall signal, net 50 points

From 32 authors in 33 discussions. Run a pilot on one production app to measure which failures it surfaces and how long tracing them takes.

Analyzed
73 total sources
Top praise
Datasets, Experiments & Scorers (20 of 21 sources) See praise evidence
Top concern
SDK & Ingestion Reliability (6 of 6 sources) See concern evidence

Apr 2025 to Oct 2026.

59 sources from practitioners, by month and sentiment.

Team accounts lean positive: of the 34 team and leadership sources that give an opinion, 20 are positive, 11 mixed and 3 negative. Evaluation draws the most praise. Seven of 8 discussing trace-level debugging praise it. All 6 discussing SDK or ingestion reliability report problems. Teams also weigh building their own eval workflow. Platform decisions run 12 to 3 in favor of keeping or choosing Braintrust.

Positive, mixed and negative sources

The score counts them as 19 positive, 10 mixed and 3 negative votes, one per author. See How the numbers work.

See the evidence

Below: 3 praise themes, 2 concerns, 1 point of agreement and 1 disagreement.

Findings

Counted for Teams and technical leaders, from 41 sources. Change group

What practitioners praise

Every capability practitioners describe positively, ranked by how many sources say so. Open one to read who said it.

  1. Datasets, Experiments & Scorers20 of 21 sources on it praise itTeams run a task against a dataset and compare scored experiments. They use code or LLM judges, with human scorers for other criteria.

    Counted for Teams and technical leaders.

    Positive, 20

    1. Positive

      Matt Rossman (mattrossman), AI tooling at Supabase

      feat(evals): link Braintrust experiments to their CI run and the PRGitHub pull request

      Linked each Braintrust experiment to the GitHub Actions run and pull requests that produced it, shown on an experiment running Codex on GPT-6 Luna. The team can go from a trace to its CI logs and from a run to its source.

      Published Oct 8, 2026self-described: Supabase evals repository, experiments run from GitHub Actions

    2. Positive

      Cheap-Ad-2360, Reddit user, r/LangChain

      The same evals score production traffic, not only experimentsReddit comment

      Disputes that Braintrust is only for experiments: the team scores production traffic with the same evals, which catches quality changes after a release, and that link is why it kept Braintrust.

      Published Sep 9, 2026self-described: A team scoring production traffic with its release evals

    3. Positive

      Low_Dust_9566, Reddit user, r/mlops

      Our PM runs prompt changes against the dataset without an engineerReddit comment

      Says Braintrust has worked for the team because, once evals are set up, its product manager can change a prompt, run it against the dataset and review the regressions in the UI without an engineer.

      Published Aug 19, 2026self-described: A team whose product manager changes prompts and reviews regressions

    4. Positive

      Pedro Rodrigues, AI tooling engineer, Supabase

      Combine Skills and MCP to Close the Context GapConference talk with transcript, AI Engineer Europe 2026

      Supabase graded task completeness on Braintrust while testing coding agents with and without its MCP server and skill, and Rodrigues calls Braintrust a very cool product.

      Published May 15, 2026Supabase testing coding agents with its MCP server and Agent Skills

    5. Positive

      David Gomes, Engineer on coding agents, Cursor

      Replacing 12K LoC with a 200 LoC SkillConference talk with transcript, AI Engineer Europe 2026

      Gomes wrote evals for Cursor's worktree skill in Braintrust, running tasks through the headless Cursor CLI with scorers that check where the agent made changes. Gomes reports that writing them was easy for someone early in writing evals.

      Published Apr 30, 2026Cursor team replacing its worktree feature with an agent skill

    6. Positive

      mierz00, Hacker News commenter

      Comment on: Ask HN: How are people doing AI evals these days?Hacker News comment

      Rates Braintrust highly: datasets were easy to set up, and evals showed whether prompts were working, which mattered most for classification that has to stay consistent.

      Published Mar 11, 2026self-described: Team using an LLM for classification

    7. Positive

      yodem, Developer (pull request author)

      chore(sc-41159): Add Braintrust evaluation cronjob infrastructureGitHub pull request

      Proposes, in an open pull request, production jobs that back up Braintrust logs to Google Cloud Storage weekly and use Claude to tag logs into Braintrust datasets daily.

      Published Feb 4, 2026self-described: Sefaria, AI chatbot in production

    8. Positive

      aantn, Maintainer (pull request author)

      feat: implement remote Braintrust evals for all HolmesGPT testsGitHub pull request

      Proposes, in an open pull request, exposing HolmesGPT's roughly 53 pytest LLM evals as remote Braintrust evals so the team can run them from the playground.

      Published Jan 3, 2026self-described: HolmesGPT, an open-source SRE agent, with about 53 LLM evals

    9. Positive

      Nick Sullivan (TechNickAI), Builder at Carmenta Collective

      Migrate to Braintrust-native eval frameworkGitHub pull request

      Replaced about 2,000 lines of custom eval code with Braintrust's Eval() for 28 routing and attachment test cases, for stored and versioned results, side-by-side experiment comparison, and regression tracking.

      Published Dec 11, 2025self-described: Carmenta, evals for model routing and file handling

    10. Positive

      James Mackey, Co-founder of Mura

      BOLT - How Mura wrote an in-house LLM Eval FrameworkEngineering blog post, James's Newsletter (Substack)

      After trialing Vellum, W&B Weave and Patronus, Mura built its BOLT eval CLI on Braintrust Datasets and Experiments, praising experiment visualization while owning its domain logic.

      "We ended up using Braintrust’s Datasets and Experiments functionality as some of the core primitives in BOLT."
      Excerpt, James Mackey

      Published Oct 21, 2025self-described: Mura, seed-stage startup automating billing for commercial HVAC providers

    11. Positive

      Ranjitha Gurunath Kulkarni, Ameya Bhatawdekar, and Gonzalo Garcia, Dropbox Dash engineers

      A practical blueprint for evaluating conversational AI at scaleEngineering blog post, Dropbox Tech blog

      Dash adopted Braintrust once scattered eval artifacts became unmanageable. Every pull request runs about 150 judged queries in under 10 minutes, merges are blocked on any red-line miss, and one click opens the trace behind a score.

      "…we turned to internal datasets by collecting production logs from Dropbox employees dogfooding Dash."
      Excerpt, Ranjitha Gurunath Kulkarni

      Published Oct 2, 2025Dropbox Dash, a conversational AI search product

    12. Positive

      Ido Pesok, engineer on v0, Vercel

      Evals Are Not Unit TestsConference talk with transcript, AI Engineer World's Fair 2025

      Recommends adding evals to CI because Braintrust reports each change's improvements and regressions across the whole dataset, and says the v0 team runs its evals at least daily.

      Published Aug 6, 2025Vercel v0 team running evals in CI and on a daily schedule

    13. Positive

      Michael Matloka (Twixes), Engineer at PostHog

      test(max): Migrate insight deepevals to BraintrustGitHub pull request

      Moved PostHog's Max AI insight evals from DeepEval to Braintrust and added LLM classifiers that score whether a generated plan answers the question and whether the final query follows the plan.

      Published May 6, 2025self-described: PostHog Max AI, evals for generated insight queries

    14. Positive

      Howie Liu, Product Hunt reviewer, Airtable

      Review of BraintrustProduct Hunt review

      Says Braintrust evals transformed AI development at Airtable, raising confidence within weeks and giving the team the feedback loop to ship reliable AI features faster.

      Published Apr 9, 2025self-described: Airtable AI Assistant

    15. Mixed

      Nima Taheri (nima-taheri-mongodb), Contributor to MongoDB's MCP server

      chore(eval-tests): upgrade eval deps, weekly evals, standalone Braintrust CLI CLOUDP-450203GitHub pull request

      Proposes moving the MongoDB MCP server's Braintrust evals from every push to weekly runs plus labeled pull requests. Also proposes calling the standalone Braintrust CLI by its direct path because the SDK bundles a conflicting bt binary.

      Published Sep 25, 2026self-described: MongoDB MCP server, eval tests in CI

    16. Positive

      Disastrous_Injury561, Reddit user, r/mlops

      The best evals we've written came from production failuresReddit post

      The team manages its traces and runs its regression tests in Braintrust, and every new model must pass the cases drawn from production failures before the averages are trusted. Curating which failures join the suite is still hard.

      Published Aug 5, 2026self-described: A team that turns odd production traces into regression tests before each model upgrade

    17. Mixed

      mierz00, Hacker News commenter

      Comment on: Ask HN: How are people doing AI evals these days?Hacker News comment

      Says the value came from running evals against a dataset and different models; Braintrust makes that easy, but the commenter would now vibe-code the same functionality themselves.

      Published Mar 17, 2026self-described: Team using an LLM for classification

    18. Positive

      Nicolas Bustamante, Founder of Fintool

      Lessons from Building AI Agents for Financial ServicesPersonal blog post, nicolasbustamante.com

      Fintool tracks experiments in Braintrust and evaluates every model or prompt change against about 2,000 test cases. A pull request is blocked when the eval score drops more than 5%, and auto-filed production errors link to Braintrust traces.

      "We auto-file GitHub issues for production errors. Error happens, issue gets created with full context: conversation ID, user info, traceback, links to Braintrust traces and Temporal workflows."
      Excerpt, Nicolas Bustamante

      Published Jan 24, 2026Fintool, AI agents for financial research and equity analysis

    19. Mixed

      gabe-kent, Developer at a Braintrust customer (issue author)

      Remote Evals (examples and support)GitHub issue

      As a Braintrust customer, their org wanted remote evals in its Rails app; the Ruby SDK lacked them, so the author built support modeled on the Python SDK and offered it upstream.

      Published Jan 13, 2026self-described: Customer organization with a main Rails app

    20. Positive

      Paul Klein IV, Founder and CEO, Browserbase

      The Web Browser Is All You NeedConference talk with transcript, AI Engineer World's Fair 2025

      Klein distrusts public web-agent benchmarks and says Browserbase runs its own evals with Braintrust's support, which lets the team pick the best model for each web task or website.

      Published Jun 20, 2025Browserbase, browser infrastructure for AI agents

    Mixed or negative, 1

    1. Negative

      fazlerocks, Hacker News commenter

      Ask HN: What tools are you using for AI evals? Everything feels half-bakedHacker News post

      Evaluated Braintrust among several eval tools and found it an interesting approach that feels early-stage, with sparse documentation and limited integrations; the team still relies on custom scripts.

      Published Jun 5, 2025self-described: Team running LLMs in production for content generation, customer support, and code review assistance

    GILD's fact checks, which don't count as sources

    Checked Oct 12, 2026.Braintrust records immutable snapshots of eval runs. Teams create them in code, the UI, or CI and compare improvements and regressions. Experiments

    Open them in the record
  2. Trace-Level Debugging7 of 8 sources on it praise itBraintrust shows each model and tool call within a run. Engineers inspect inputs and outputs, plus token counts and latency, to locate failures.

    Counted for Teams and technical leaders.

    Positive, 7

    1. Mixed

      dealvz, Developer (pull request author)

      Remove the Braintrust integrationGitHub pull request

      Prepared, but did not merge, a change removing Braintrust in case the team drops it, listing the losses: all LLM tracing, prompt loading and the prompt A/B eval harness.

      Published Oct 5, 2026self-described: GoodParty.org omni platform: chat, briefings, PMF runner

    2. Positive

      giga-james, Developer (pull request author)

      feat: trace simulations to Braintrust instead of the GigaFlow SDKGitHub pull request

      Proposes, in an open pull request, replacing the team's own SDK with Braintrust tracing for agent simulations, verified on 9 runs and 210 spans whose rewards matched when queried back.

      Published Sep 17, 2026self-described: GigaFlow, tau2 agent simulations

    3. Positive

      Miguel (miguelg719), Contributor to Browserbase's Stagehand evals

      feat(evals): bump braintrust to 3.x; first-party BraintrustSpanProcessor OTEL sinkGitHub pull request

      Proposes upgrading Stagehand's evals from Braintrust 0.4 to 3.x, because the old pin blocked Braintrust's first-party OpenTelemetry span processor, which captures AI SDK spans better than a generic OTLP exporter.

      Published Jul 13, 2026self-described: Browserbase Stagehand, browser-agent evals

    4. Positive

      Matt Rossman (mattrossman), AI tooling at Supabase

      feat(evals): link Braintrust experiments to their CI run and the PRGitHub pull request

      Linked each Braintrust experiment to the GitHub Actions run and pull requests that produced it, shown on an experiment running Codex on GPT-6 Luna. The team can go from a trace to its CI logs and from a run to its source.

      Published Oct 8, 2026self-described: Supabase evals repository, experiments run from GitHub Actions

    5. Positive

      BlueHotDog, Developer (issue author)

      feat(observability): replace Arize with Braintrust OTel + exact wire logging via Finch telemetryGitHub issue

      Plans to replace Arize with Braintrust over OTLP, mapping spans to the OpenTelemetry GenAI conventions and logging the exact payload sent to the LLM.

      Published Apr 10, 2026self-described: Frontman, agent platform on Elixir

    6. Positive

      Nicolas Bustamante, Founder of Fintool

      Lessons from Building AI Agents for Financial ServicesPersonal blog post, nicolasbustamante.com

      Fintool tracks experiments in Braintrust and evaluates every model or prompt change against about 2,000 test cases. A pull request is blocked when the eval score drops more than 5%, and auto-filed production errors link to Braintrust traces.

      "We auto-file GitHub issues for production errors. Error happens, issue gets created with full context: conversation ID, user info, traceback, links to Braintrust traces and Temporal workflows."
      Excerpt, Nicolas Bustamante

      Published Jan 24, 2026Fintool, AI agents for financial research and equity analysis

    7. Positive

      Ranjitha Gurunath Kulkarni, Ameya Bhatawdekar, and Gonzalo Garcia, Dropbox Dash engineers

      A practical blueprint for evaluating conversational AI at scaleEngineering blog post, Dropbox Tech blog

      Dash adopted Braintrust once scattered eval artifacts became unmanageable. Every pull request runs about 150 judged queries in under 10 minutes, merges are blocked on any red-line miss, and one click opens the trace behind a score.

      "…we turned to internal datasets by collecting production logs from Dropbox employees dogfooding Dash."
      Excerpt, Ranjitha Gurunath Kulkarni

      Published Oct 2, 2025Dropbox Dash, a conversational AI search product

    Mixed or negative, 1

    1. No opinion stated

      Rafal Wilinski, Tech Lead for Zapier Agents, with Vitor Balocco

      Turning Fails into Features: Zapier's Hard-Won Eval LessonsConference talk with transcript, AI Engineer World's Fair 2025

      Reading one Braintrust experiment screen at a time made aggregate regressions hard to explain. The team pointed a reasoning model at two runs through Braintrust MCP to compare Gemini Pro with Claude before deploying.

      Published Jun 30, 2025Zapier Agents team comparing models on Braintrust experiments

    Open them in the record
  3. Instrumentation & SDK Coverage3 of 5 sources on it praise itTeams assess setup time through SDK wrappers and auto-instrumentation, including OpenTelemetry. Language and framework support determine the work required.

    Counted for Teams and technical leaders.

    Positive, 3

    1. Positive

      giga-james, Developer (pull request author)

      feat: trace simulations to Braintrust instead of the GigaFlow SDKGitHub pull request

      Proposes, in an open pull request, replacing the team's own SDK with Braintrust tracing for agent simulations, verified on 9 runs and 210 spans whose rewards matched when queried back.

      Published Sep 17, 2026self-described: GigaFlow, tau2 agent simulations

    2. Positive

      mierz00, Hacker News commenter

      Comment on: Ask HN: How are people doing AI evals these days?Hacker News comment

      Rates Braintrust highly: datasets were easy to set up, and evals showed whether prompts were working, which mattered most for classification that has to stay consistent.

      Published Mar 11, 2026self-described: Team using an LLM for classification

    3. No opinion stated

      Rafal Wilinski, Tech Lead for Zapier Agents, with Vitor Balocco

      Turning Fails into Features: Zapier's Hard-Won Eval LessonsConference talk with transcript, AI Engineer World's Fair 2025

      Reading one Braintrust experiment screen at a time made aggregate regressions hard to explain. The team pointed a reasoning model at two runs through Braintrust MCP to compare Gemini Pro with Claude before deploying.

      Published Jun 30, 2025Zapier Agents team comparing models on Braintrust experiments

    Mixed or negative, 2

    1. Mixed

      gabe-kent, Developer at a Braintrust customer (issue author)

      Remote Evals (examples and support)GitHub issue

      As a Braintrust customer, their org wanted remote evals in its Rails app; the Ruby SDK lacked them, so the author built support modeled on the Python SDK and offered it upstream.

      Published Jan 13, 2026self-described: Customer organization with a main Rails app

    2. Negative

      fazlerocks, Hacker News commenter

      Ask HN: What tools are you using for AI evals? Everything feels half-bakedHacker News post

      Evaluated Braintrust among several eval tools and found it an interesting approach that feels early-stage, with sparse documentation and limited integrations; the team still relies on custom scripts.

      Published Jun 5, 2025self-described: Team running LLMs in production for content generation, customer support, and code review assistance

    GILD's fact checks, which don't count as sources

    Checked Oct 12, 2026.Braintrust has Python, TypeScript, Go, Java, Ruby, and C# SDKs with automatic instrumentation for supported providers and frameworks. An OTLP exporter or span processor sends OpenTelemetry traces. Trace LLM calls

    Open them in the record

What concerns them

Every concern practitioners raise, most raised first.

  1. SDK & Ingestion ReliabilityAll 6 sources on it raise it as a concernSDK releases break or slow logging. Other reports describe ingestion failures and missed calls, plus logger memory and latency costs.

    Counted for Teams and technical leaders.

    1. Mixed

      Nima Taheri (nima-taheri-mongodb), Contributor to MongoDB's MCP server

      chore(eval-tests): upgrade eval deps, weekly evals, standalone Braintrust CLI CLOUDP-450203GitHub pull request

      Proposes moving the MongoDB MCP server's Braintrust evals from every push to weekly runs plus labeled pull requests. Also proposes calling the standalone Braintrust CLI by its direct path because the SDK bundles a conflicting bt binary.

      Published Sep 25, 2026self-described: MongoDB MCP server, eval tests in CI

    2. Mixed

      dknecht, Developer (issue author)

      Flue adapter: logging the full conversation-so-far per turn makes background-logger retention O(turns²), OOMs 128 MB Workers isolatesGitHub issue

      The Flue adapter logged the whole conversation on every turn, so the SDK's background logger grew quadratically and crash-looped their production agent on Cloudflare Workers.

      Published Aug 11, 2026self-described: Production agent on Cloudflare Workers Durable Objects

    3. Mixed

      CodingCanuck, Developer (issue author)

      Regular 500 responses from /logs3GitHub issue

      Their eval pipeline regularly hit 500 errors and failed fetches from the Braintrust log ingestion endpoint while running evals.

      Published Feb 27, 2026self-described: Team eval pipeline on the TypeScript SDK

    4. Mixed

      CodingCanuck, Developer (issue author)

      Significant latency regression between TypeScript SDKs v0.4.10 and v1.0.0GitHub issue

      Upgrading the TypeScript SDK from 0.4.10 to 1.0.0 slowed eval runs of a few hundred tasks from 1.5 to 5 minutes, traced to a sequential log upload loop.

      Published Feb 19, 2026self-described: Team eval pipeline on the TypeScript SDK

    5. Mixed

      ai-simple-ai, Developer (issue author)

      Node.js SDK has broken/bad bundlingGitHub issue

      The Node.js SDK bundle published to npm was broken because OpenTelemetry modules were not marked external, so the team patched it locally.

      Published Nov 20, 2025self-described: Team patching the SDK locally

    6. Mixed

      paulosuzart, Developer (issue author)

      FunctionCall tool-span instrumentation for Agno leaks the full Agent/Team object (incl. model.api_key) into span metadataGitHub issue

      The Agno integration logged the full agent object, including the provider API key, into tool-span metadata; the team found a real OpenAI key in its own project logs.

      Published Sep 3, 2026self-described: Team tracing Agno agents with MCP tools

    Open them in the record
  2. Build It Yourself2 of 6 sources on it raise it as a concernTeams compare Braintrust with building their own dataset runner and score storage.

    Counted for Teams and technical leaders.

    Raised as a concern, 2

    1. Mixed

      mierz00, Hacker News commenter

      Comment on: Ask HN: How are people doing AI evals these days?Hacker News comment

      Says the value came from running evals against a dataset and different models; Braintrust makes that easy, but the commenter would now vibe-code the same functionality themselves.

      Published Mar 17, 2026self-described: Team using an LLM for classification

    2. Negative

      aviggiano, Developer (issue author)

      Remove Braintrust reporting and implicit judge gateway integrationGitHub issue

      Removed Braintrust reporting and its implicit judge gateway from Ultrafuzz after cutting paid CI, so evaluations run locally without a Braintrust account.

      Published Sep 22, 2026self-described: Ultrafuzz, Monad developers' fuzzing project

    Discussed without concern, 4

    1. Positive

      mierz00, Hacker News commenter

      Comment on: Ask HN: How are people doing AI evals these days?Hacker News comment

      Rates Braintrust highly: datasets were easy to set up, and evals showed whether prompts were working, which mattered most for classification that has to stay consistent.

      Published Mar 11, 2026self-described: Team using an LLM for classification

    2. Positive

      Nick Sullivan (TechNickAI), Builder at Carmenta Collective

      Migrate to Braintrust-native eval frameworkGitHub pull request

      Replaced about 2,000 lines of custom eval code with Braintrust's Eval() for 28 routing and attachment test cases, for stored and versioned results, side-by-side experiment comparison, and regression tracking.

      Published Dec 11, 2025self-described: Carmenta, evals for model routing and file handling

    3. Positive

      James Mackey, Co-founder of Mura

      BOLT - How Mura wrote an in-house LLM Eval FrameworkEngineering blog post, James's Newsletter (Substack)

      After trialing Vellum, W&B Weave and Patronus, Mura built its BOLT eval CLI on Braintrust Datasets and Experiments, praising experiment visualization while owning its domain logic.

      "We ended up using Braintrust’s Datasets and Experiments functionality as some of the core primitives in BOLT."
      Excerpt, James Mackey

      Published Oct 21, 2025self-described: Mura, seed-stage startup automating billing for commercial HVAC providers

    4. Negative

      fazlerocks, Hacker News commenter

      Ask HN: What tools are you using for AI evals? Everything feels half-bakedHacker News post

      Evaluated Braintrust among several eval tools and found it an interesting approach that feels early-stage, with sparse documentation and limited integrations; the team still relies on custom scripts.

      Published Jun 5, 2025self-described: Team running LLMs in production for content generation, customer support, and code review assistance

    Open them in the record

Where the evidence agrees

Statements this group's sources agree on, then the ones close to agreement.

  1. Recurring point

    A dataset and scored experiments decide whether a prompt, model, or code change ships.

    10 supporting sources

Close to agreement: each needs 9 supporting sources

  1. Below threshold

    Teams test replacement models against their evals. Prompts tuned to the current model regress on the new one.

    7 supporting sources

Where it disagrees

Questions the sources split on, with the count on each side.

  • Evidence remains split

    When practitioners weigh Braintrust against another option, which do they keep?

    • Chose or kept Braintrust, 12 sources

      Most chose Braintrust over Langfuse, Arize, Vellum, W&B Weave, Patronus, their own SDK, or in-house scoring, for its experiments and comparison views.

    • Chose another tool, 3 sources

      Others moved to Langfuse, a local store and runner, or custom scripts, citing exhausted credits, a wish to run evaluations without an account, or one place for traces and prompts. One early-stage team ruled Braintrust out because self-hosting needs Enterprise.

The evidence

Whose experience this is

The three groups, then every practitioner by the context they work in, and how emphasis changes with scale.

Three groups drawn from the same sources. Choose a group to select whose experience counts. Within that group, each author counts once in the practitioner sentiment, with their latest view. Signals open on Teams and technical leaders.

  • Teams and technical leaders

    41 sources. Product teams and startups describe what Braintrust evals block from release. They also report SDK problems at scale.

  • Individual builders

    18 sources. Developers report SDK setup and reliability in their own projects, alongside costs and data controls.

  • All practitioners

    59 sources. These accounts combine product teams and solo developers. Their bug reports and platform decisions carry different contexts.

59 sources from practitioners, by the context they work in. Filled rows count for this group.

  1. Individual builderDevelopers building on the platform on their own18 sources

    Emphasizes SDK reliability, setup, and tool choice

  2. Small product team2 to 10 engineers, or an unnamed startup team14 sources, counted here

    Emphasizes SDK problems in production eval pipelines

  3. Product or platform team11 to 50 engineers, or a named company's team23 sources, counted here

    Emphasizes Evaluation workflows and model changes

  4. Technical leadershipLeaders setting model and platform choices across teams4 sources, counted here

    Emphasizes Building eval tooling on Braintrust primitives

  5. Organization-wideEnterprise or company-wide AI platformNo sources from this setting yet

What changes with scale. Individual builders mainly report SDK and ingestion problems or decisions to replace Braintrust, and one early-stage team ruled it out in October 2026 because self-hosting needs Enterprise. Named teams describe dataset experiments and regressions caught before prompt or model changes shipped: Dropbox runs about 150 judged queries on every pull request, and Rox's weekly sweep of Braintrust traces covered 67 behaviors where its hand-written suite covered 34.

Where the sources come from

Where this group's sources were published.

Counted for Teams and technical leaders, from 41 sources. Change group

55 sources for this group, 14 of them documentation

  • Developer Communities26, 47% of 55 sources
  • Product Documentation14, 25% of 55 sources
  • Podcasts, Talks & Videos8, 15% of 55 sources
  • Engineering Blogs & Reviews7, 13% of 55 sources

Older accounts remain when applicable. Accounts tied to retired plans or deployment paths, or resolved SDK gaps, carry Needs a recheck.

See sources by origin

The record

Who each source is, how it knows, whether it still applies and what it counts toward.

Newest first, on its own page. Every count above opens its sources there.

Read all 73 sources

About this read

What changed, and how we count

What changed

  • Each source links to a public page and names its author and publication date.
  • A finding needs at least 5 sources from 3 different authors for each audience. A disagreement also needs at least 2 sources on each side.
  • Merges lock-in and hosted-only concerns into Data Security & Control, and drops Playgrounds & Prompt Iteration, Cross-Functional Review, and two statements, none of which reaches 5 sources.
  • Checks the pricing, retention, hosting, and security findings against Braintrust's current documentation and its May 2026 incident report.
  • Counts only sources published on or after January 1, 2025, and caps GitHub issues at about a third of the sources. Set aside eight issue threads that repeated SDK bug reports. Their replacements are pull requests from PostHog, Supabase, MongoDB, Browserbase, and Carmenta, plus a Reddit account.
  • Oct 25, 2026: adds 2 engineering blog posts, 2 conference talks, and 1 independent tool evaluation.
  • Oct 26, 2026: adds AI Engineer talks by Browserbase's founder and by Cursor and Supabase engineers, and the Fintool founder's account of gating changes on Braintrust experiments. A first-party speaker page or the author's own site states each role.
  • Oct 28, 2026: adds Operator Insights from Parlance Labs, Rox, Fintool, Dropbox, PostHog and Supabase, CREATE SOMETHING's and Ultrafuzz's exits, and Nudge Security's account of the May 2026 key incident, with ledger quotes on 7 insights. Adds 9 records: 5 from Reddit, Rox's engineering post, Parlance Labs' evaluation, an X post and a How I AI episode with Braintrust's CEO. Parlance Labs' record is listed and not counted, since Hamel Husain already has two counted records here; the X post, drawn from GILD's search for problem reports, and the vendor episode carry no stance.
  • Oct 30, 2026: a second-model check found Rox's post was published in partnership with Cognition and Braintrust, so it is listed and no longer counted, and Parlance Labs' record no longer reads as a vote for Braintrust, since its authors say ranking vendors was not the goal. It corrected Nudge Security's card to what TechCrunch reported on May 6, Supabase's CI links to the runs that carry CI metadata, Parlance Labs' card from easiest to easy, and a Reddit commenter's context. Team and leadership accounts now run 20 positive, 11 mixed and 3 negative among 34.

Method

A running review of what teams building AI features on Braintrust report, from individual builders to technical leaders. Each finding links to its sources, and each conflict shows both sides.

Counts cover only the sources GILD reviewed.

Who counts

Teams and technical leaders
Small product team, Product or platform team, Technical leadership, Organization-wide
Individual builders
Individual builder
All practitioners
Everyone in the teams and leaders and individual builders groups

Choose a group to select whose experience counts. Within that group, each author counts once in the practitioner sentiment, with their latest view. GILD files each source in a group from how the account describes its author. A role or employer marked self-described is what the source states about themselves. Documentation checks facts for every group and never counts toward sentiment.

How the numbers work

  • Overall signal. Positive votes minus negative ones, per 100 votes from the sources that take a side. Each author counts once, with their latest view, whoever they are and wherever they posted. The older weighted reading, where accounts GILD matched to a named person count three times when GILD lists the author among technical leaders and twice otherwise, is shown beside it as a second view. Where an author wrote more than one source, the line beside the split gives the votes, one per author, so the score follows from it. When one author's sources from the same day disagree, that author counts as mixed, and a half point rounds away from zero. Sources with no opinion are listed and not counted. Strong positive is +50 or higher, positive +20 to +49, mixed −19 to +19, negative −20 to −49, strong negative −50 or lower.
  • Weighted reading, second view. Counted the older way, where an account GILD matched to a named person counts two or three times, teams and leaders read +62 strong positive (method sentiment/S1). GILD printed that number as "GILD score" until October 29, 2026; most of those matches rest on a name or a self-described role.
  • Method change, Oct 30, 2026. GILD changed how practitioner sentiment counts, to one vote per author. Weighted, it read +63 Strong positive; with one vote per author it reads +50 Strong positive. No evidence changed.
  • Unscored. A group gets a score only with 20 sources from 15 authors. Below that, one source moves the score about 5 points, so the read says "Unscored until 20 sources from 15 authors" and shows its counts.
  • Independence. A source whose author works for the vendor, for one of its partners or programs, or for a competitor, and a page the vendor published or ran, is listed with that tie and counts toward nothing. One author counts on at most two records per product. The line under the score says how many authors and how many separate discussions the counted sources come from: comments in one thread are one discussion.
  • Praise and concerns. How many of the sources that discuss a theme praise it or raise it.
  • Consensus. A statement joins when 20% of the group's sources support it and fewer contradict it. Contradicting sources are counted separately. Below that line a statement shows as building. At 40% or more it reads Strong agreement; below, Recurring point.
  • Disagreements. The count on each side. GILD names no winner and gives no share of all users.
  • Minimum evidence. A finding needs 5 sources from 3 authors; a disagreement also needs 2 on each side. Smaller counts stay in the table below, marked under the minimum.

Labels on a source

A source's date and whether it still applies are separate: a recent post can describe an old version, and an old one can still hold.

Documented
Stated in official documentation.
Reported
A practitioner's account of their own use.
Measured
Backed by the source's own measurements, such as a comparative test or tracked team metrics.
GILD analysis
GILD's check of a claim against official documentation.
Current
Describes the product as it is now.
Still applicable
Older, and the behavior or lesson still holds.
Needs a recheck
May depend on a plan, version or setting that changed since it was written. It keeps counting until GILD rechecks it, since the rest of the account usually holds, and its card says so.
Out of date
No longer accurate. Kept in the record; counts toward nothing current.

Flagged for a recheck, of the sources that take a side: teams and leaders 1 of 34, individual builders 0 of 17, all practitioners 1 of 51.

Limits

  • The figures count reviewed public sources. Bug reports lean negative, so GILD caps GitHub issues at about a third, 15 of 59, and counts an issue as mixed unless its author rejects or abandons Braintrust. Another 15 sources are pull requests documenting adoption, upgrades, or removal. Braintrust closed all 11 issues on its own repositories as fixed. The two talks co-presented with Braintrust staff lean positive. On Oct 25, 2026, GILD added Dropbox's and Browserbase's engineering blog posts, an independent tool evaluation, and AI Engineer World's Fair talks by Vercel and Zapier engineers. Hacker News returned 5 qualifying sources from 2025 onward. Arctic Shift returned one qualifying r/LLMDevs post. Searches of r/MachineLearning timed out. On Oct 28, 2026, Arctic Shift returned dozens of short 2026 comments in r/LangChain and r/mlops that recommend Braintrust; GILD kept the 5 that give a first-hand account of using or rejecting it.
  • Records include the SDK versions and plans their sources name, though many omit them. Starter replaced Free on March 16, 2026. Model access moved from the AI proxy to Gateway, and SDK fixes ship often. Accounts tied to earlier plans or deployment paths, or resolved SDK gaps, carry Needs a recheck.

Every count, and the sources behind it

13 findings for 3 groups

Each row opens its sources for the group you are reading.

FindingTeams and technical leadersIndividual buildersAll practitionersSources
Current readCurrent sentiment20 positive, 11 mixed, 3 negative7 positive, 6 mixed, 4 negative27 positive, 17 mixed, 7 negativeSources
PraiseDatasets, Experiments & Scorers20 of 21 positive3 of 4 positive (under the minimum)23 of 25 positiveSources
PraiseTrace-Level Debugging7 of 8 positive2 of 3 positive (under the minimum)9 of 11 positiveSources
PraiseInstrumentation & SDK Coverage3 of 5 positive1 of 1 positive (under the minimum)4 of 6 positiveSources
ConcernUsage-Based Cost0 of 1 raise it (under the minimum)1 of 4 raise it (under the minimum)1 of 5 raise itSources
ConcernSDK & Ingestion Reliability6 of 6 raise it3 of 3 raise it (under the minimum)9 of 9 raise itSources
ConcernData Security & Control4 of 4 raise it (under the minimum)2 of 2 raise it (under the minimum)6 of 6 raise itSources
ConcernBuild It Yourself2 of 6 raise it2 of 5 raise it4 of 11 raise itSources
ConsensusEvals gate prompt and model changes10 supporting0 supporting (under the minimum)10 supporting (below threshold)Sources
ConsensusEvery model swap is re-evaluated7 supporting (below threshold)1 supporting (under the minimum)8 supporting (below threshold)Sources
ConsensusOpenTelemetry keeps the backend swappable2 supporting (under the minimum)3 supporting (under the minimum)5 supporting (below threshold)Sources
DisagreementWhen practitioners weigh Braintrust against another option, which do they keep?Chose or kept Braintrust 12, Chose another tool 3Chose or kept Braintrust 4, Chose another tool 4Chose or kept Braintrust 16, Chose another tool 7Sources
Current readEvidence mix55 sources32 sources73 sourcesSources

Fact checks

GILD's checks of claims against official documentation, kept apart from the source counts.

  • Datasets, Experiments & Scorers. Braintrust records immutable snapshots of eval runs. Teams create them in code, the UI, or CI and compare improvements and regressions. Experiments, checked Oct 12, 2026
  • Instrumentation & SDK Coverage. Braintrust has Python, TypeScript, Go, Java, Ruby, and C# SDKs with automatic instrumentation for supported providers and frameworks. An OTLP exporter or span processor sends OpenTelemetry traces. Trace LLM calls, checked Oct 12, 2026
  • Usage-Based Cost. Starter includes 1 GB and 10,000 scores monthly, then charges US$4 per GB and US$2.50 per 1,000 scores. Pro costs US$249 with 5 GB and 50,000 scores, then US$3 per GB and US$1.50 per 1,000. Braintrust meters data at ingestion. It offers spend alerts without hard limits. Plans and limits, checked Oct 12, 2026
  • Data Security & Control. SaaS uses a US or EU data plane. Enterprise BYOC and self-hosting keep traces and datasets, plus prompts, in your cloud. Braintrust retains the control plane in every deployment. After an attacker reached decrypted provider keys in a May 2026 migration backup, Braintrust added workload identity federation for supported providers. Deployment options, checked Oct 12, 2026