LangSmith+17 Mixed

Operator Insights

What are teams learning from building on LangSmith and running it in production?

9 accounts: 2 left it or turned it down, 1 evaluated it and 6 adopted it

Left it or turned it down2

AmitAuthor of the NonBioS engineering postNonBioS
NonBioS's AI engineer product, whose runs use over a million tokens and about a hundred inference calls in five minutes, with sensitive customer data.
Its own announcement, published Mar 5, 2025Read the write-up (opens the announcement in a new tab)
Observation
NonBioS ruled out LangSmith because it seemed tightly coupled to LangChain, which the team had removed from its code, and chose Langfuse.
Takeaway
If your code has left LangChain, test LangSmith through its SDK wrappers or OpenTelemetry before you rule it out on coupling alone.
What happened
NonBioS had found LangChain too complex and rewritten its systems without it, so it declined LangSmith as strongly coupled to LangChain (Mar 5, 2025). It looked at Arize Phoenix briefly and found the naming and website confusing, narrowed the field to Langfuse and Helicone, and chose Langfuse for session tracking that groups a run's calls into one session, self-hosting and open-source extensibility.
“Langsmith appeared popular, but we had encountered challenges with Langchain from the same company, finding it overly complex for previous NonBioS tooling. We rewrote our systems to remove dependencies on Langchain and chose not to proceed with Langsmith as it seemed strongly coupled with Langchain.”
Amit
- Public engineering write-up
- No team metrics reported
Deployment details
NonBioS's AI engineer product, whose runs use over a million tokens and about a hundred inference calls in five minutes, with sensitive customer data.
- Public engineering write-up
- No team metrics reported
Team
NonBioS engineering
Workload
Long agent runs with about a hundred inference calls each
Product version
Evaluated and passed over

GAIAMay 28, 2026GitHub (theexperiencecompany/gaia)GAIA started leaving LangSmith for a self-hosted Langfuse, running both in parallel during the migration, because its development workspace kept hitting LangSmith's monthly trace limit.
aryanranderiyaGitHub contributorGAIA
GAIA, an open-source personal AI assistant with sub-agents and MCP tools. The self-hosted Langfuse runs on the team's own server and is reachable only over its private network.
GitHub (theexperiencecompany/gaia), published May 28, 2026Read the write-up (opens GitHub (theexperiencecompany/gaia) in a new tab)
Observation
GAIA started leaving LangSmith for a self-hosted Langfuse, running both in parallel during the migration, because its development workspace kept hitting LangSmith's monthly trace limit.
Takeaway
Check LangSmith's monthly trace limit against your development traffic, and run a self-hosted tracer in parallel before you cut over.
What happened
A pull request merged on May 28, 2026 hooked Langfuse into every agent run while LangSmith kept receiving the same traces in parallel during the migration. The stated reasons: LangSmith is paid, its development workspace kept failing with a monthly unique-trace limit error, and Langfuse is open source with control over residency and retention.
Public engineering write-up (merged pull request in GAIA's repository)
No team metrics reported
Deployment details
GAIA, an open-source personal AI assistant with sub-agents and MCP tools. The self-hosted Langfuse runs on the team's own server and is reachable only over its private network.
Public engineering write-up (merged pull request in GAIA's repository)
No team metrics reported
Team
GAIA engineering at Experience Inc.
Duration
Both tools in parallel from May 28, 2026
Workload
Personal assistant agents, sub-agents and tool calls
Product version
LangSmith beside self-hosted Langfuse 3

Evaluated it1

Antaripa Saha and Hamel HusainAI evals consultants and co-authorsParlance Labs
Independent test of the trace-reading agents in Braintrust, Arize and LangSmith, and of three coding agents, on 100 traces from Nurture Boss's apartment-leasing assistant with 39 failures labeled by a domain expert.
Its own announcement, published Jul 11, 2026Read the write-up (opens the announcement in a new tab)
Observation
Parlance Labs tested LangSmith Engine on 100 hand-labeled production traces, found it surfaced too few issues to score, and scored LangSmith's chat agent instead.
Takeaway
Run LangSmith Engine on traces your experts have already labeled before you trust its issue list, and recheck every finding it ranks high severity.
What happened
Parlance Labs masked the human labels and gave every system the same light prompt (published Jul 11, 2026). LangSmith Engine linked the recurring issues it found to their traces but surfaced far fewer than a full review, so the authors scored LangSmith's chat agent: 79.5% recall and 77.5% precision, against 87.2% recall for Braintrust Loop and 91.0% precision for Arize Alyx. LangSmith had the best interface for reading single traces, and its chat agent cited the exact message or tool call behind each real failure, yet several of its wrong findings were marked high severity. No system caught failures that needed business context, such as Markdown sent over SMS. The post states that the authors have no affiliation with, and no sponsorship from, any vendor it names.
“But it surfaced only a small set of issues, far fewer than a full review turns up, so we could not use it for the 100-trace comparison. We ran the scored experiment with LangSmith’s chat agent instead.”
Antaripa Saha and Hamel Husain
- Public practitioner write-up
- Team metrics reported
Deployment details
Independent test of the trace-reading agents in Braintrust, Arize and LangSmith, and of three coding agents, on 100 traces from Nurture Boss's apartment-leasing assistant with 39 failures labeled by a domain expert.
- Public practitioner write-up
- Team metrics reported
Team
Antaripa Saha and Hamel Husain of Parlance Labs, with a domain expert labeling the traces
Workload
Error analysis of 100 production traces from an apartment-leasing assistant
Product version
LangSmith Engine and LangSmith's chat agent, July 2026
Measurement
Recall 79.5% and precision 77.5% for LangSmith's chat agent, against 39 hand-labeled failures

Adopted it6

Lysia Li and five co-authorsEngineeringHarvey
Harvey Agentic Search for legal research, built on the OpenAI Agents SDK, scored on queries written by legal experts.
Its own announcement, published Dec 10, 2025Read the write-up (opens the announcement in a new tab)
Observation
Harvey runs offline evaluations of its legal research agent in LangSmith on OpenTelemetry traces from the OpenAI Agents SDK, with no LangChain in the path.
Takeaway
A team on another agent framework can keep its code and still score it in LangSmith by sending OpenTelemetry traces.
What happened
Harvey built central evaluation infrastructure on the OpenAI Agents SDK's OpenTelemetry traces and runs offline evaluations in LangSmith, tracking hallucination, tool recall, retrieval recall, formatting and answer quality (Harvey blog, Dec 10, 2025).
“Centralized evaluation infrastructure that we built using the OpenAI Agent SDK's OpenTelemetry traces enables us to run offline evaluations in LangSmith. We track metrics like hallucination, tool recall, retrieval recall, formatting, and answer quality.”
Lysia Li
- Public engineering write-up
- No team metrics reported
Deployment details
Harvey Agentic Search for legal research, built on the OpenAI Agents SDK, scored on queries written by legal experts.
- Public engineering write-up
- No team metrics reported
Team
Harvey search engineering
Workload
Agentic legal research search
Product version
LangSmith offline evaluation on OpenTelemetry traces from the OpenAI Agents SDK
Measurement
Hallucination, tool recall, retrieval recall, formatting and answer quality tracked; no figures published

LyftJun 15, 2026Interrupt 26 (LangChain's YouTube channel)Lyft's customer-care AI team gates each agent release on offline evals and routes production traces that fail its LLM judge to human review.
Nick UngML lead for safety and customer careLyft
AI Assist customer-care agents. About 270,000 AI interactions a month. Seven or more agents in production.
Interrupt 26 (LangChain's YouTube channel), published Jun 15, 2026Watch the talk (opens Interrupt 26 (LangChain's YouTube channel) in a new tab)
Observation
Lyft's customer-care AI team gates each agent release on offline evals and routes production traces that fail its LLM judge to human review.
Takeaway
Write judges per task, avoiding generic helpfulness scores, and send every judge failure to a human queue so the evals keep tracking production.
What happened
Lyft tests each agent before release with a simulator whose LLM plays the user. Task-specific judges apply success and failure criteria. LangSmith records simulated and production trajectories and hosts the judges. An automation sends failed traces to annotation queues, where operations staff label them and engineers examine recurring failures. Ung reports a 65% deflection rate and an AI resolution rate that rose from 10% to 35%.
Vendor customer story (Interrupt 26, LangChain's conference, on LangChain's YouTube channel)
Team metrics reported
Deployment details
AI Assist customer-care agents. About 270,000 AI interactions a month. Seven or more agents in production.
Vendor customer story (Interrupt 26, LangChain's conference, on LangChain's YouTube channel)
Team metrics reported
Team
Lyft's safety and customer care AI team, with an operations team annotating traces
Workload
AI Assist customer-care chat agents
Product version
LangChain, LangGraph, and LangSmith
Measurement
Deflection rate and AI resolution rate
CiscoJun 10, 2026Interrupt 26 (LangChain's YouTube channel)Cisco's Customer Experience team has an agent pull every thumbs-down and error trace from LangSmith, cluster them, and file Jira issues only for real problems.
Carlos PereiraCustomer Experience teamCisco
Cisco technical support, which runs its support agents' feedback and error traces through LangSmith. About 16 million customer interactions a year.
Interrupt 26 (LangChain's YouTube channel), published Jun 10, 2026Watch the talk (opens Interrupt 26 (LangChain's YouTube channel) in a new tab)
Observation
Cisco's Customer Experience team has an agent pull every thumbs-down and error trace from LangSmith, cluster them, and file Jira issues only for real problems.
Takeaway
Once feedback outgrows the team, let an agent cluster failures from traces before anyone files a ticket, and monitor the tracing platform like any other production dependency.
What happened
Cisco records every thumbs-down and error in LangSmith, plus low-confidence routing decisions. A triage agent retrieves traces through LangSmith's MCP server and reads the relevant code. It groups similar failures and files Jira issues through a Jira MCP server. People approve write actions only. Pereira reports support cases falling from 1.6 million last year to 1.4 million this year, partly because of these systems. Cisco monitors LangSmith with Splunk as business-critical infrastructure.
Vendor customer story (Interrupt 26, LangChain's conference, on LangChain's YouTube channel)
Team metric reported
Deployment details
Cisco technical support, which runs its support agents' feedback and error traces through LangSmith. About 16 million customer interactions a year.
Vendor customer story (Interrupt 26, LangChain's conference, on LangChain's YouTube channel)
Team metric reported
Team
Cisco Customer Experience team
Workload
Technical support agents that triage customer cases
Product version
LangSmith tracing and the LangSmith MCP server
Measurement
Support cases per year, 1.6 million to 1.4 million
monday.comSep 9, 2026monday.com Engineeringmonday.com traces its Feed agent in LangSmith and runs an LLM judge in CI before each production deploy, and its Production Engineering team keeps one LangSmith trace per infrastructure-agent run.
Kamie Shami-SchnitzerFeed agent buildermonday.com
feedAgent, a deep agent that curates a personal activity feed for each workspace member, and separately the Infra agents platform, whose agents work on production infrastructure code.
monday.com Engineering, published Sep 9, 2026Read the write-up (opens monday.com Engineering in a new tab)
Observation
monday.com traces its Feed agent in LangSmith and runs an LLM judge in CI before each production deploy, and its Production Engineering team keeps one LangSmith trace per infrastructure-agent run.
Takeaway
Make the judge part of CI, so the trace that explains a failure and the check that blocks it live in the same place.
What happened
monday.com wraps each Feed agent invocation in LangSmith's traceable(), which records tool calls, model responses and token counts, and runs an LLM judge with metrics for each part in CI to check for regressions before deploying (engineering blog, Sep 9, 2026). On Oct 26, 2026 its Production Engineering team described giving each Infra agent run one parent LangSmith trace, so any run can be reconstructed after an incident. LangChain's own customer story of Feb 18, 2026 adds that monday.com logs each CI run as a LangSmith experiment for its LangGraph-based service agents.
“We use an LLM as a judge with specific metrics for each part, and run it on the CI to make sure we have no regressions before deploying to production.”
Kamie Shami-Schnitzer
Public engineering write-up
No team metrics reported
Deployment details
feedAgent, a deep agent that curates a personal activity feed for each workspace member, and separately the Infra agents platform, whose agents work on production infrastructure code.
Public engineering write-up
No team metrics reported
Team
monday.com Feed agent team and Production Engineering
Workload
Personal activity feed agent and infrastructure agents
Product version
LangSmith traceable() tracing, with LLM-judge evals in CI
SonderMindJul 25, 2026AI EngineerSonderMind's clinical experts review production conversations of its mental health coach in LangSmith, and each missed disclosure they mark becomes an eval case.
Akele Reed and Dave ReverePrincipal and Staff AI EngineersSonderMind
Sonder, an AI mental health coach that supports people before or between therapy sessions. SonderMind uses LangSmith annotation queues on production conversations.
AI Engineer, published Jul 25, 2026Watch the talk (opens AI Engineer in a new tab)
Observation
SonderMind's clinical experts review production conversations of its mental health coach in LangSmith, and each missed disclosure they mark becomes an eval case.
Takeaway
Let the clinicians or other experts who spot the miss write the expected answer, and turn each one into a test the next change must pass.
What happened
At AI Engineer World's Fair 2026, SonderMind's engineers described a clinical subject-matter expert reviewing a production conversation in LangSmith and marking a missed disclosure with the expected observation; a script then writes it to the eval set, which every prompt or model change must pass in CI.
“A clinical SME reviews a production conversation in LangSmith, marks the missed disclosure with an expected observation, and the script writes it to the eval set.”
Akele Reed and Dave Revere
Conference talk (AI Engineer World's Fair 2026)
No team metrics reported
Deployment details
Sonder, an AI mental health coach that supports people before or between therapy sessions. SonderMind uses LangSmith annotation queues on production conversations.
Conference talk (AI Engineer World's Fair 2026)
No team metrics reported
Team
SonderMind AI engineering with clinical subject-matter experts
Workload
Mental health coaching conversations
Product version
LangSmith annotation queues
Spring HealthSep 29, 2026Interrupt NYC 2026 (LangChain's YouTube channel)Spring Health's clinicians write the rubrics and review the guardrail flags for its mental health agent in LangSmith, and non-engineers start evaluation runs themselves.
Marissa SaundersSenior Director of EngineeringSpring Health
Member-facing AI agent on a mental health care platform. Crisis guardrails with escalation to a person. Enterprise customers.
Interrupt NYC 2026 (LangChain's YouTube channel), published Sep 29, 2026Watch the talk (opens Interrupt NYC 2026 (LangChain's YouTube channel) in a new tab)
Observation
Spring Health's clinicians write the rubrics and review the guardrail flags for its mental health agent in LangSmith, and non-engineers start evaluation runs themselves.
Takeaway
When experts disagree on the right answer, have them write the rubric and grade traces in the tool the engineers use, and turn what they catch into new monitors.
What happened
Spring Health's agent matches members to providers and supports them between therapy sessions, so it has to catch a member in crisis and hand off to a person. Clinicians could tell a good response from a bad one but could not say why. The team built test sets and rubrics with them in LangSmith, which replaced spreadsheets, documents, and Slack threads. Guardrail evals run as LangSmith experiments that non-technical staff can start, and whole conversations are scored by VERA-MH, an open-source evaluator Spring Health built from a simulated patient and a judge agent. In production, clinicians mark guardrail flags true or false positive in annotation queues. A clinician's review of one member who kept testing the agent's boundaries led the team to monitor turns per member, conversations per period, and guardrail trips.
Vendor customer story (Interrupt NYC 2026, LangChain's conference, on LangChain's YouTube channel)
No team metrics reported
Deployment details
Member-facing AI agent on a mental health care platform. Crisis guardrails with escalation to a person. Enterprise customers.
Vendor customer story (Interrupt NYC 2026, LangChain's conference, on LangChain's YouTube channel)
No team metrics reported
Team
Spring Health engineering teams, with clinicians reviewing
Duration
About a year and a half of work on chat agents
Workload
Member-facing chat: provider matching, support between sessions, and crisis escalation
Product version
LangSmith datasets, experiments, and annotation queues

Left it or turned it down2

NonBioS ruled out LangSmith because it seemed tightly coupled to LangChain, which the team had removed from its code, and chose Langfuse.

GAIA started leaving LangSmith for a self-hosted Langfuse, running both in parallel during the migration, because its development workspace kept hitting LangSmith's monthly trace limit.

Evaluated it1

Parlance Labs tested LangSmith Engine on 100 hand-labeled production traces, found it surfaced too few issues to score, and scored LangSmith's chat agent instead.

Adopted it6

Harvey runs offline evaluations of its legal research agent in LangSmith on OpenTelemetry traces from the OpenAI Agents SDK, with no LangChain in the path.

Lyft's customer-care AI team gates each agent release on offline evals and routes production traces that fail its LLM judge to human review.

Cisco's Customer Experience team has an agent pull every thumbs-down and error trace from LangSmith, cluster them, and file Jira issues only for real problems.

monday.com traces its Feed agent in LangSmith and runs an LLM judge in CI before each production deploy, and its Production Engineering team keeps one LangSmith trace per infrastructure-agent run.

SonderMind's clinical experts review production conversations of its mental health coach in LangSmith, and each missed disclosure they mark becomes an eval case.

Spring Health's clinicians write the rubrics and review the guardrail flags for its mental health agent in LangSmith, and non-engineers start evaluation runs themselves.