Turn every failure you see in a LangSmith trace into a dataset example, and make each prompt or model change pass an experiment on that dataset before it ships. This playbook covers instrumenting the agent, keeping tracing from taking the app down, masking sensitive data, retention per project, scoring production, and alerts.
When you need this
- An agent is moving from a prototype to production, and nobody can say why a given run went wrong.
- Prompt or model changes ship on a hunch, and regressions surface as user complaints.
- Several teams trace into one organization and nobody owns projects, retention, or evaluators.
Evaluation criteria
Instrument the agent
How do traces reach LangSmith?
Look forSet LANGSMITH_TRACING=true and LANGSMITH_API_KEY for LangChain and LangGraph. Use LANGSMITH_PROJECT to separate services. For other code, use traceable or the OpenAI, Anthropic, and Gemini wrappers. OpenTelemetry uses LANGSMITH_OTEL_ENABLED. Send spans through a Collector when Datadog or Honeycomb also needs them.
Fail open
What happens to the application when LangSmith is slow or down?
Look forTest LangSmith failure before launch and load-test with tracing on. In 2025, a Python client stopped sending traces without logging an error until the application restarted. A long-running LangGraph service also grew in memory only with tracing enabled. Give operations a LANGSMITH_TRACING=false switch.
Keep sensitive data out
Which inputs and outputs must never leave the application?
Look forUse LANGSMITH_HIDE_INPUTS and LANGSMITH_HIDE_OUTPUTS to omit those fields. For selective redaction, use rule-based masking, the secret-redaction anonymizer, or Microsoft Presidio. Skip zero-retention requests with conditional tracing.
Set retention per project
How long does each project keep its traces?
Look forDefault production projects to base retention of 14 days. Choose extended retention of 180 days for traces you need later. Set this before traffic starts, because a change applies only to new traces.
Collect feedback and failures
How does a bad run become a test case?
Look forAttach user feedback through the SDK and send runs to annotation queues. Choose single-run queues for rubric review or pairwise queues for comparisons. Enable reservations to avoid duplicate reviews. Export annotated runs to datasets, whose examples never expire. One team builds golden datasets from outputs corrected in the queue.
Gate changes on experiments
What must an experiment show before a prompt or model change ships?
Look forRun the dataset through code evaluators for structure and exact matches, LLM-as-a-judge evaluators for criteria, and pairwise evaluators when two versions are easier to compare than to score. The pytest and Vitest integrations run them with the test suite. Agree on a pass threshold for each evaluator before the first run, and review judge scores against human annotation, as LangChain's own guidance asks.
Score production
What tells you quality dropped after release?
Look forFilter online evaluators to relevant traces, such as negative feedback or a specific tool call. Add multi-turn evaluation for threads. One developer saw a normal trace for each verification step while the full session failed. A new online evaluator upgrades every trace it scores to extended retention unless you opt out.
Alert on thresholds
Who hears about an error spike first?
Look forSet alerts for error count or rate, average latency, feedback score, or cost over a 5 or 15 minute window. Route them to Slack, PagerDuty, Dynatrace, or a webhook. Check that cost tracking works first. One developer's Claude Code traces showed model names with no token usage or cost.
Questions to ask vendors
- Which projects exist, who owns each, and what is each one's default retention?
- Which fields are masked, and who signs off on the masking rules?
- What does the application do when LangSmith returns errors or times out?
- Which evaluators gate a release, with what thresholds, and who reviews judge scores against human grades?
- How does a production failure get from a trace to a dataset, and who does it?
Red flags
- Production traces at extended retention by default, with no one deciding why.
- Raw user input and model output traced from a regulated workload with no masking.
- LLM-as-a-judge scores treated as ground truth, with no human review of a share of them.
- Online evaluators on every trace with the retention upgrade left on.
- Tracing that can take the application down with it.
Products in this piece
Sources
- Official documentation
- 10
- Public source
- 5
- LangSmith docs: Tracing quickstart
- LangSmith docs: Trace with OpenTelemetry
- LangSmith docs: Prevent logging of sensitive data in traces
- LangSmith docs: Usage and billing (retention tiers and rate limits)
- LangSmith docs: Annotation queues
- LangSmith docs: Evaluation concepts
- LangSmith docs: Run evaluations with pytest
- LangSmith docs: LLM-as-a-judge online evaluators
- LangSmith docs: Alerts
- LangSmith docs: Cost tracking
- langsmith-sdk issue #1630: client silently stops posting traces (Apr 3, 2025)
- langsmith-sdk issue #2009: memory growth with tracing on (Sep 10, 2025)
- langsmith-sdk issue #2593: custom input rendering for annotation queues (Mar 18, 2026)
- Hacker News comment on turn-level traces missing a broken flow (Mar 4, 2026)
- LangChain docs issue #1999: Claude Code traces without token usage or cost (Dec 21, 2025)