Make every prompt or model change pass a Braintrust experiment built from your own production failures. Instrument the feature and mask sensitive fields before the first trace leaves the application. Have domain experts curate the datasets, and agree on pass thresholds before the first run.
When you need this
- An LLM feature or agent is moving to production, and nobody can say whether the next prompt or model change makes it better.
- Prompt or model changes ship on a hunch, and regressions surface as customer complaints.
- Product managers or specialists judge quality but have to ask engineers for logs to see the outputs.
Evaluation criteria
Instrument the feature
How do traces reach Braintrust?
Look forCreate one Braintrust project per application. Wrap provider clients such as OpenAI or Anthropic, or turn on automatic instrumentation. Braintrust ships SDKs for Python, TypeScript, Go, Java, Ruby, and C#. Teams with an OpenTelemetry pipeline add an OTLP exporter or Braintrust's span processor. One developer tail-samples whole traces at an OpenTelemetry gateway and lets only chosen services write to each project.
Test the SDK under load
What does the logger cost the application?
Look forPin the SDK version and load-test tracing before launch. Three regressions showed up under real workloads after an upgrade or a new integration. A TypeScript upgrade slowed evals from 1.5 to 5 minutes. Per-turn logging exhausted a Cloudflare Workers isolate's memory. A Ruby runner added 10 to 25 seconds per eval case. Braintrust fixed each within weeks.
Keep secrets and personal data out
Which values must never leave the application?
Look forMask values in the SDK or in span customizers before upload. Never log span or framework objects that hold credentials. Two public issues show Braintrust and provider keys reaching logs that way. Ingestion redaction runs after upload, and Braintrust offers it only to selected customers.
Curate datasets with domain experts
Who decides what good looks like?
Look forStart with about ten examples per capability, as Notion does, and check their structure before generating more. Add production failures. Let product managers write the criteria, as Zapier did for its top 25 apps. Pro includes unlimited human review scorers. Starter allows one per project.
Write scorers that match the product
How is each output graded?
Look forUse code scorers for structure and exact fields. Use LLM judges for rule-based criteria and human review for the judgments left over. Notion writes a judge for each example that checks rules such as required language and format, so no output is graded against a frozen answer. Its team reviews judge failures by hand.
Gate the merge
What must an experiment show before a change ships?
Look forRun evals on every pull request with braintrustdata/eval-action or bt eval, and compare each experiment with the previous run. Agree on scorer pass thresholds before the first run. Zapier grew from 7 manual tests to more than 800 that run on merge requests, with regression alerts.
Re-run evals on every model change
What happens when a cheaper or faster model arrives?
Look forRun replacement models against the same datasets. Zapier's scores fell below 80% after it moved its Copilot to GPT-4o, and removing instructions tuned to the old model recovered them. Notion moves a prompt to a new model only when its evals pass there.
Score production
What tells you quality dropped after release?
Look forSet online scoring rules with a sampling rate, because each recorded score adds usage. Topics groups logs daily by task, sentiment, and issue. SQL conditions send alerts to Slack or a webhook.
Questions to ask vendors
- Which projects exist, who owns each, and which data plane region holds them?
- Which fields are masked before upload, and who signs off on the masking code?
- Which scorers gate a merge, with what thresholds, and who reviews judge scores against human grades?
- How does a production failure get into a dataset, and who curates it?
- Which SDK version is pinned, and was the last upgrade load-tested?
Red flags
- Raw prompts and outputs from a regulated workload traced with no masking.
- SDK span objects or agent objects logged as metadata.
- LLM-as-a-judge scores treated as ground truth with no human review.
- Online scoring on every production trace with no sampling rate.
- A model switch shipped without re-running the eval suite.
Products in this piece
Sources
- Official documentation
- 13
- Public source
- 8
- Braintrust docs: Trace LLM calls
- Braintrust docs: Send OpenTelemetry traces and logs
- Braintrust docs: Protect sensitive data
- Braintrust docs: Create datasets
- Braintrust docs: Set up human review
- Braintrust docs: Write scorers
- Braintrust docs: Experiments
- Braintrust docs: Run experiments in CI/CD
- Braintrust docs: Compare experiments
- Braintrust docs: Score production traces
- Braintrust docs: Topics
- Braintrust docs: Set up alerts
- Braintrust docs: Plans and limits
- Sarah Sachs (Notion) with Carlos Esteban and Doug Guthrie (Braintrust): How to build world-class AI products, AI Engineer World's Fair 2025 (Jun 27, 2025)
- Olmo Maldonado (Zapier) with Ankur Goyal (Braintrust): How Zapier Builds AI Products and Features with the Help of Braintrust, AI Engineer World's Fair 2024 (Nov 7, 2024)
- Jerred Shepherd, pull request #2827: tail-sample LLM traces into Braintrust over OpenTelemetry (Sep 10, 2026)
- braintrust-sdk-javascript issue #1394: eval runs slowed from 1.5 to 5 minutes after the 1.0 upgrade (Feb 19, 2026)
- braintrust-sdk-javascript issue #2353: per-turn conversation logging exhausts Cloudflare Workers memory (Aug 11, 2026)
- braintrust-sdk-ruby issue #210: per-case flush adds 10 to 25 seconds to Ruby evals (Aug 19, 2026)
- braintrust-sdk-javascript issue #912: logging a span object writes the API key into the log (Sep 7, 2025)
- braintrust-sdk-python issue #733: Agno tool spans leak the provider API key into span metadata (Sep 3, 2026)