Operator Insights

What are teams learning from building on Datadog Agent Observability and running it in production?

10 accounts: 2 failures, incidents or rollbacks, 1 left it or turned it down, 1 run it beside another tool, 5 adopted it and 1 reported by others

Failures, incidents and rollbacks2

Tomas AlabesSoftware developerSource.ag
A Node.js agent loop on LangGraph with LangChain and the Anthropic SDK, traced with Datadog LLM Observability.
GitHub (DataDog/dd-trace-js), published Jun 16, 2026Read the issue (opens GitHub (DataDog/dd-trace-js) in a new tab)
Observation
At Source.ag, LangChain and Anthropic spans for the same model call made Datadog's cost rollup overcount spend about 2.2 times.
Takeaway
Check that each model call produces one LLM span before trusting cost views, and turn off duplicate provider integrations under a framework.
What happened
With LLM Observability on, both the LangChain and the Anthropic integrations emitted an LLM span for each model call, nested one inside the other (GitHub issue, Jun 16, 2026). Datadog's cost rollup summed both layers, so the cost view did not deliver the team's real spend: it overcounted about 2.2 times, and the two spans disagreed on cache reads. The workaround was to turn the Anthropic integration off. Datadog closed the issue as completed in June 2026. A month later the same engineer found every MCP tool call captured twice, with the full payload stored on each span.
“…cost rollup sums both layers and overcounts spend (~2.2× in our case)…”
Tomas Alabes
- Public engineering write-up (the developer's GitHub issue)
- Team metric reported
Deployment details
A Node.js agent loop on LangGraph with LangChain and the Anthropic SDK, traced with Datadog LLM Observability.
- Public engineering write-up (the developer's GitHub issue)
- Team metric reported
Team
Source.ag, the team's own agent application
Workload
An agent loop on LangGraph calling Anthropic models through LangChain
Product version
dd-trace 5.106.x on Node.js 22 with LLM Observability enabled
Measurement
Cost rollup overcounted spend about 2.2 times
In SignalsCounted once, as the developer's own issue. In the record

SeatGeekAug 18, 2026GitHub (DataDog/dd-trace-py)At SeatGeek, streamed OpenAI Responses calls lost their LLM spans and token metrics in Datadog LLM Observability.
Zach HammerSoftware developerSeatGeek
Instrumented OpenAI streaming calls on the Responses API, traced by ddtrace with LLM Observability on.
GitHub (DataDog/dd-trace-py), published Aug 18, 2026Read the issue (opens GitHub (DataDog/dd-trace-py) in a new tab)
Observation
At SeatGeek, streamed OpenAI Responses calls lost their LLM spans and token metrics in Datadog LLM Observability.
Takeaway
Test streamed calls separately when you roll out the tracer, since a span that fails to build disappears without failing the request.
What happened
The team saw a run of error logs from LLM Observability for instrumented OpenAI streaming calls (GitHub issue, Aug 18, 2026). On the Responses API with streaming on, an error in token counting dropped the openai.request span entirely, orphaned its children, and lost its token metrics. The issue traces the cause to two defects in ddtrace's stream handling and is still open.
Public engineering write-up (the developer's GitHub issue)
No team metrics reported
Deployment details
Instrumented OpenAI streaming calls on the Responses API, traced by ddtrace with LLM Observability on.
Public engineering write-up (the developer's GitHub issue)
No team metrics reported
Team
SeatGeek engineering
Workload
OpenAI Responses API calls with streaming
Product version
ddtrace 4.9.7 on Python 3.13
In SignalsCounted once, as the developer's own issue. In the record

Left it or turned it down1

AmitAuthor, NonBioS engineering blogNonBioS
NonBioS, an AI software engineering agent company whose five-minute runs can consume over a million tokens and a hundred model calls.
NonBioS blog, published Mar 5, 2025Read the write-up (opens NonBioS blog in a new tab)
Observation
NonBioS weighed Datadog among about a dozen LLM observability tools and chose self-hosted Langfuse instead, for stable open-source software it could run on premises.
Takeaway
If prompts must stay on your own servers, settle that first: Datadog runs only as SaaS on Datadog's sites, so on-premises teams rule it out before features matter.
What happened
NonBioS listed LangSmith, Lunary, Phoenix, Portkey, Datadog, Langfuse and Helicone among about a dozen offerings (NonBioS blog, Mar 5, 2025). It narrowed the choice to Langfuse and Helicone and chose Langfuse, because it needed mature, stable software, deployment on its own premises for sensitive customer data, and open source it could extend.
“We narrowed down our choices to Langfuse and Helicone. Both appeared solid, but we ultimately chose Langfuse.”
Amit
- Public engineering write-up
- No team metrics reported
Deployment details
NonBioS, an AI software engineering agent company whose five-minute runs can consume over a million tokens and a hundred model calls.
- Public engineering write-up
- No team metrics reported
Team
NonBioS engineering
Workload
Tracing for NonBioS's AI software engineer
In SignalsListed once, as the company's own post; it describes no use of Datadog, so it takes a side only on where traces live. In the record

Run it beside another tool1

Kyle GallatinTechnical Lead Manager of ML InfrastructureHandshake
LLM Orca, Handshake's internal service that feature teams build LLM features on.
Arize AI customer story, published Aug 17, 2026Read the write-up (opens Arize AI customer story in a new tab)
Observation
Handshake built Datadog and Arize AX into its internal LLM Orca service, so every model call its feature teams make is traced for monitoring and evaluation in both.
Takeaway
Run both when two groups read the same calls: Datadog for the operators who watch production, a dedicated tool for the AI engineers who evaluate it.
What happened
Arize's customer story of Aug 17, 2026 says LLM Orca has Arize AX and Datadog integrations built in, so every call is traced, and that Handshake shipped more than 15 LLM use cases on it in under six months. Feature teams evaluate prompts, break down cost and export production traces in Arize AX.
“The user experience of LLM Orca is designed to be easy for a set of LLM use cases. Arize AX and Datadog integrations are built in, so every call is traced for evaluation and monitoring.”
Arize customer story, on Handshake
- Vendor customer story (Arize customer page, Arize being a Datadog competitor)
- Team metrics reported
Deployment details
LLM Orca, Handshake's internal service that feature teams build LLM features on.
- Vendor customer story (Arize customer page, Arize being a Datadog competitor)
- Team metrics reported
Team
Handshake ML infrastructure and its feature teams
Duration
Under six months for more than 15 use cases
Workload
LLM features built on the LLM Orca service
Product version
Datadog and Arize AX, both built into the service
Measurement
More than 15 LLM use cases in under six months
In SignalsNot counted. A competitor of Datadog published this story, and Handshake has published no account of its own.

Adopted it5

Twine SecurityApr 16, 2026Datadog case studyTwine Security cut token usage per task 40% over three months on its IAM agent after tracing it in Datadog Agent Observability, Datadog's case study says.
Twine Security engineering teamBuilders of the Alex IAM agentTwine Security
Alex, Twine Security's AI digital employee that runs identity and access management tasks end to end.
Datadog case study, published Apr 16, 2026Read the write-up (opens Datadog case study in a new tab)
Observation
Twine Security cut token usage per task 40% over three months on its IAM agent after tracing it in Datadog Agent Observability, Datadog's case study says.
Takeaway
Use the traces to find redundant tool loops and oversized prompts, and measure tokens per task before and after each fix.
What happened
Datadog's case study, dated Apr 16, 2026 in its page metadata, says Twine traced Alex in Agent Observability, then optimized tool calls, shortened long prompts and removed redundant tool loops. Token usage per task fell 40% over three months, and mean time to resolution fell about 80%.
“Over a three-month period, token usage per task decreased by 40% after the team optimized tool calls, streamlined long and heavy prompts, and eliminated redundant tool loops altogether.”
Datadog case study, on Twine Security
Vendor customer story (Datadog case study, dated Apr 16, 2026 in its page metadata)
Team metrics reported
Deployment details
Alex, Twine Security's AI digital employee that runs identity and access management tasks end to end.
Vendor customer story (Datadog case study, dated Apr 16, 2026 in its page metadata)
Team metrics reported
Team
Twine Security agent engineering
Duration
Three months
Workload
The Alex IAM agent
Product version
Datadog Agent Observability
Measurement
Tokens per task down 40%; mean time to resolution down about 80%
In SignalsNot counted. Datadog published this story, and Twine Security has published no account of its own.
FintoolFeb 19, 2025Datadog case studyFintool monitors its multi-provider financial copilot with Datadog Agent Observability, and Datadog's case study credits it with 60% faster complex queries and 84% lower churn.
Fintool engineering teamBuilders of a financial research copilotFintool
Fintool's financial copilot, running a multi-provider LLM pipeline on AWS and Azure.
Datadog case study, published Feb 19, 2025Read the write-up (opens Datadog case study in a new tab)
Observation
Fintool monitors its multi-provider financial copilot with Datadog Agent Observability, and Datadog's case study credits it with 60% faster complex queries and 84% lower churn.
Takeaway
Ask a vendor which baseline and period a churn or latency figure covers before you use it in a business case.
What happened
Datadog's case study, dated Feb 19, 2025 in its page metadata, says Fintool instrumented its pipeline with Datadog's LLM Observability, improved response times for complex, multi-step queries by 60 percent, and reduced customer churn by 84 percent. The case study gives no baseline period, and the churn figure is Datadog's attribution.
“Fintool reduces customer churn by 84% and monitors complex operations with Datadog Agent Observability…”
Datadog case study, on Fintool
Vendor customer story (Datadog case study, dated Feb 19, 2025 in its page metadata)
Team metrics reported
Deployment details
Fintool's financial copilot, running a multi-provider LLM pipeline on AWS and Azure.
Vendor customer story (Datadog case study, dated Feb 19, 2025 in its page metadata)
Team metrics reported
Team
Fintool engineering
Workload
A multi-provider financial copilot on AWS and Azure
Product version
Datadog LLM Observability, now Agent Observability
Measurement
Complex queries 60% faster; churn down 84%, by Datadog's attribution
In SignalsNot counted. Datadog published this story, and Fintool has published no account of its own.
AppFolioNov 7, 2024Datadog case studyAppFolio chose Datadog LLM Observability to monitor Realm-X Messages, its generative AI messaging feature for property managers, Datadog's case study says.
AppFolio Realm-X teamBuilders of Realm-X generative AI featuresAppFolio
Realm-X Messages, the generative AI feature AppFolio offers property managers among its 20,000-plus customers.
Datadog case study, published Nov 7, 2024Read the write-up (opens Datadog case study in a new tab)
Observation
AppFolio chose Datadog LLM Observability to monitor Realm-X Messages, its generative AI messaging feature for property managers, Datadog's case study says.
Takeaway
Monitoring a customer-facing AI feature beside the rest of the product is the case Datadog makes; ask for the latency and quality numbers it reports in your own trial.
What happened
Datadog's case study, dated Nov 7, 2024 in its page metadata, says AppFolio chose Datadog to monitor the performance and quality of Realm-X Messages, which saves property managers an average of five hours a week on communication tasks. The case study attributes that time saving to Realm-X Messages and leaves observability uncredited.
“AppFolio chose Datadog Agent Observability to monitor the performance and quality of its new application, which they named Realm-X Messages…”
Datadog case study, on AppFolio
Vendor customer story (Datadog case study, dated Nov 7, 2024 in its page metadata)
Product metric reported
Deployment details
Realm-X Messages, the generative AI feature AppFolio offers property managers among its 20,000-plus customers.
Vendor customer story (Datadog case study, dated Nov 7, 2024 in its page metadata)
Product metric reported
Team
AppFolio Realm-X
Workload
Realm-X Messages, generative AI messaging for property managers
Product version
Datadog LLM Observability, now Agent Observability
Measurement
Five hours a week saved per property manager, a product outcome
In SignalsNot counted. Datadog published this story, and AppFolio has published no account of its own.
AirbyteSep 30, 2026GitHub (airbytehq/PyAirbyte)Airbyte made native Datadog tracing the default for its hosted MCP server whenever a deployment turns LLM Observability on.
Caleb EvansSoftware EngineerAirbyte
PyAirbyte's hosted MCP server, which deployments trace through OpenTelemetry by default or through Datadog's native backend.
GitHub (airbytehq/PyAirbyte), published Sep 30, 2026Read the pull request (opens GitHub (airbytehq/PyAirbyte) in a new tab)
Observation
Airbyte made native Datadog tracing the default for its hosted MCP server whenever a deployment turns LLM Observability on.
Takeaway
Let the tracer's own switch pick the backend, and keep an explicit override for teams that route traces elsewhere.
What happened
Hosted MCP deployments that already enabled Datadog LLM Observability still traced through OpenTelemetry unless they also set a separate backend variable. Airbyte changed the default so DD_LLMOBS_ENABLED alone selects native Datadog tracing, while explicit settings keep precedence (pull request, Sep 30, 2026). The change notes that the native backend needs the Datadog extra and automatic MCP instrumentation turned off. It passed 155 tests and merged the same day.
Public engineering write-up (the engineer's merged pull request)
Team metric reported
Deployment details
PyAirbyte's hosted MCP server, which deployments trace through OpenTelemetry by default or through Datadog's native backend.
Public engineering write-up (the engineer's merged pull request)
Team metric reported
Team
Airbyte, PyAirbyte maintainers
Workload
Tracing for PyAirbyte's hosted MCP server deployments
Product version
Datadog LLM Observability through ddtrace, selected by DD_LLMOBS_ENABLED
Measurement
155 tests passed across the OpenTelemetry and Datadog tracing suites
In SignalsCounted once, as the engineer's own pull request. In the record
cBioPortalJun 9, 2026GitHub (cBioPortal/LibreChat)cBioPortal added dd-trace to its LibreChat image so every Anthropic and OpenAI call is traced in Datadog LLM Observability without code changes.
Gaofei ZhaocBioPortal contributorcBioPortal
cBioPortal's LibreChat deployment, traced in Datadog.
GitHub (cBioPortal/LibreChat), published Jun 9, 2026Read the pull request (opens GitHub (cBioPortal/LibreChat) in a new tab)
Observation
cBioPortal added dd-trace to its LibreChat image so every Anthropic and OpenAI call is traced in Datadog LLM Observability without code changes.
Takeaway
Auto-instrumentation through NODE_OPTIONS gets a Node chat app's model calls traced in one change, with no code edits.
What happened
A pull request of Jun 9, 2026 loads dd-trace through NODE_OPTIONS in cBioPortal's LibreChat image, so Anthropic and OpenAI SDK calls are auto-instrumented into LLM Observability with no code changes.
Public engineering write-up (GitHub pull request)
No team metrics reported
Deployment details
cBioPortal's LibreChat deployment, traced in Datadog.
Public engineering write-up (GitHub pull request)
No team metrics reported
Team
cBioPortal contributors
Duration
Pull request of Jun 9, 2026
Workload
LibreChat for cBioPortal users
Product version
dd-trace loaded through NODE_OPTIONS
In SignalsCounted once, as the contributor's own pull request. In the record

Reported by others1

Bert StewartJun 12, 2025TechTargetThomson Reuters' head of its global command center told TechTarget that LLM Observability Experiments saves building model comparison infrastructure, and that unclear pricing decides whether new AI features reach production.
Bert StewartHead of the global command centerThomson Reuters
Thomson Reuters' operations team, assessing Datadog's AI features announced at DASH 2025.
TechTarget, published Jun 12, 2025Read the report (opens TechTarget in a new tab)
Observation
Thomson Reuters' head of its global command center told TechTarget that LLM Observability Experiments saves building model comparison infrastructure, and that unclear pricing decides whether new AI features reach production.
Takeaway
Price experiments and judge evaluations before a pilot ends, since the team here already valued the new AI features and their cost decided adoption.
What happened
TechTarget reported on Jun 12, 2025 that Bert Stewart valued LLM Observability Experiments because comparing models yourself takes a lot of infrastructure. He also said that unclear pricing of Datadog's new AI features would decide whether they reach production.
Second-hand news report (TechTarget)
No team metrics reported
Deployment details
Thomson Reuters' operations team, assessing Datadog's AI features announced at DASH 2025.
Second-hand news report (TechTarget)
No team metrics reported
Team
Thomson Reuters global command center
Workload
Evaluating LLM Observability Experiments for model comparison
Product version
LLM Observability Experiments, announced at DASH 2025
In SignalsCounted once, as the leader's words reported by TechTarget. In the record

Failures, incidents and rollbacks2

At Source.ag, LangChain and Anthropic spans for the same model call made Datadog's cost rollup overcount spend about 2.2 times.

At SeatGeek, streamed OpenAI Responses calls lost their LLM spans and token metrics in Datadog LLM Observability.

Left it or turned it down1

NonBioS weighed Datadog among about a dozen LLM observability tools and chose self-hosted Langfuse instead, for stable open-source software it could run on premises.

Run it beside another tool1

Handshake built Datadog and Arize AX into its internal LLM Orca service, so every model call its feature teams make is traced for monitoring and evaluation in both.

Adopted it5

Twine Security cut token usage per task 40% over three months on its IAM agent after tracing it in Datadog Agent Observability, Datadog's case study says.

Fintool monitors its multi-provider financial copilot with Datadog Agent Observability, and Datadog's case study credits it with 60% faster complex queries and 84% lower churn.

AppFolio chose Datadog LLM Observability to monitor Realm-X Messages, its generative AI messaging feature for property managers, Datadog's case study says.

Airbyte made native Datadog tracing the default for its hosted MCP server whenever a deployment turns LLM Observability on.

cBioPortal added dd-trace to its LibreChat image so every Anthropic and OpenAI call is traced in Datadog LLM Observability without code changes.

Reported by others1

Thomson Reuters' head of its global command center told TechTarget that LLM Observability Experiments saves building model comparison infrastructure, and that unclear pricing decides whether new AI features reach production.