GILD verdict
Choose LangSmith for LLM agents in production, above all on LangGraph, when you want tracing, evaluation, and agent hosting from one vendor.
Before you trace production traffic, set retention tiers, masking, and usage limits. Run Engine on traces your experts have already labeled before you rely on its issue list.
LangSmith
AI Observability by LangChain
+17 MixedFit and limits
Who it fits
- Agents are in production or close to it, and the team needs to see every model call, tool call, and intermediate step when a run goes wrong.
- Prompt and model changes ship often, and the team wants to score each change against a dataset before release and with online evaluators after it.
- The agents are built on LangGraph or LangChain, where tracing turns on with two environment variables and Studio and LangSmith Deployment run the same graphs.
- Other code paths need tracing too: the traceable wrapper, provider wrappers for OpenAI, Anthropic, and Gemini, and OpenTelemetry cover applications built without LangChain. Harvey runs its LangSmith evals on OpenTelemetry traces from the OpenAI Agents SDK.
- Domain experts grade the output: SonderMind's clinicians and Lyft's product managers and QA label production traces in LangSmith annotation queues, and each label can become an eval case.
Where it falls short
- Base traces are kept 14 days; keeping a trace 180 days costs more, and online evaluators and automation rules upgrade the traces they touch unless each one opts out.
- Self-hosting is an Enterprise add-on, so a team that must keep traces in its own network has no self-serve path.
- Developer and Plus offer Google and GitHub sign-in; custom SSO, workspace RBAC with custom roles, and ABAC need Enterprise.
- Plus caps ingestion at 500,000 trace events and 5.0 GB of trace data per hour, and a single trace at 25,000 runs.
- LangSmith Engine can miss much of what a full review finds: in Parlance Labs' test of 100 hand-labeled traces, published Jul 11, 2026, it surfaced too few issues to score.
- Teams that must keep trace data inside their own boundary leave for self-hosted tools: Cue Labs and one developer's personal platform retired LangSmith for Langfuse in October 2026, and GAIA ran Langfuse beside it after hitting the monthly trace limit.
Implementation reality
Instrument one agent first
For LangChain or LangGraph, set LANGSMITH_TRACING=true and LANGSMITH_API_KEY. Use LANGSMITH_PROJECT for a named project. For other code, wrap functions with traceable and model clients with provider wrappers. Alternatively, enable LANGSMITH_OTEL_ENABLED and send OpenTelemetry spans through a Collector to LangSmith and another backend.
Set retention before volume
Set a default retention tier per project, base at 14 days or extended at 180 days. Changes affect new traces only. Keep production on base and reserve extended retention for traces with feedback. Decide whether each online evaluator or automation rule may upgrade traces.
Keep sensitive data out
Set LANGSMITH_HIDE_INPUTS and LANGSMITH_HIDE_OUTPUTS to omit inputs and outputs. For selective redaction, use rule-based masking, the secret-redaction anonymizer, or a third-party anonymizer such as Microsoft Presidio. Use conditional tracing to skip requests under a client's zero-retention terms.
Gate changes on experiments
Save failed traces in a dataset, which never expires. On each prompt or model change, run offline experiments with code, LLM-as-a-judge, or pairwise evaluators. pytest and Vitest integrations include them in the test suite. Online evaluators score production traces. Alerts send latency, error, feedback, and cost thresholds to Slack, PagerDuty, or a webhook. LangSmith Engine, which proposes fixes from traces, uses about 7 to 45 LSUs a run, with a default cap of 750 LSUs a month per organization.
What to measure in your pilot
- GILD found no figure in LangChain's docs or practitioner posts for how many traces evaluators upgrade; log the weekly share at extended retention and lower evaluator sampling past 20%.
- Compare LLM-judge scores with annotation-queue grades on 100 runs per use case; Zheng et al. measured over 80% agreement for GPT-4 judges, so rework any judge below that.
- Run Engine on one busy project for a month at the Standard analysis level, and compare its LSU spend with the engineer hours its accepted fixes saved.
Sources
- Official documentation
- 18
- Public source
- 1
- LangSmith plans and pricing
- LangSmith docs: Usage and billing (retention tiers and rate limits)
- LangSmith docs: Manage billing and spend limits
- LangSmith docs: Data purging for compliance
- LangSmith docs: Cloud (SaaS) architecture and regions
- LangSmith docs: Regions FAQ
- LangSmith docs: Self-hosted LangSmith
- LangSmith docs: LangSmith for Enterprise
- LangSmith docs: Prevent logging of sensitive data in traces
- LangSmith docs: Tracing quickstart
- LangSmith docs: Trace with OpenTelemetry
- LangSmith docs: Evaluation concepts
- LangSmith docs: LLM-as-a-judge online evaluators
- LangSmith docs: Run evaluations with pytest
- LangSmith docs: Alerts
- LangSmith docs: LangSmith Deployment
- LangSmith docs: Annotation queues
- LangSmith docs: Find and fix your agent's issues with LangSmith Engine (costs and spend limits)
- arXiv: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Dec 24, 2023)