DatadogAI Observability

Datadog Agent Observability

Datadog Agent Observability, sold as LLM Observability until 2026, traces LLM apps and agents and evaluates their output. It shares one trace ID with Datadog APM, infrastructure, and user monitoring. It bills only LLM spans, one per model call, and can be bought without other Datadog products.

The 30-Second Take

Pick Datadog Agent Observability if your operators work in Datadog and want agent traces under the same trace ID as APM.

It bills only LLM spans, with 40,000 free a month. Handshake runs it beside Arize AX in one internal LLM service.

Best fit: Teams already running Datadog APM that want agent traces, model cost, and evaluations beside their service and user data.
Main constraint: The bill and the cost views depend on the tracer counting each model call once, and public reports show releases that miss or double them.
First move: Trace one agent on the Free tier, then compare a week of LLM span counts and token totals with your model invoices.

Why

Watch the span count: Source.ag found the cost rollup overcounting its spend about 2.2 times, because two integrations each recorded the same model call. Datadog renamed LLM Observability to Agent Observability in 2026 without announcing it, so search under both names.

Practitioner sentiment, teams and leaders+32 Positive

25 practitioners, 63 sources reviewed, Updated Oct 30, 2026How this is measured

Read the GILD Review

Visit Website

Operating stack: AI Application Platform

Datadog Agent Observability vs Arize
- AI Observability
Handshake built both into one internal LLM service. Pick Datadog Agent Observability when your operators already live in Datadog and want agent traces beside APM, billed per LLM span. Pick Arize when AI engineers need their own evaluation workspace or Phoenix to self-host.
Datadog Agent Observability vs Braintrust
- AI Observability
- AI Costs
Pick Datadog Agent Observability to watch agents beside APM, infrastructure, and user sessions; it prices LLM spans and keeps traces 15 days by default. Pick Braintrust when evals gate releases; it prices gigabytes and scores and builds around datasets, experiments, and scorers.

Kyle GallatinTechnical Lead Manager of ML InfrastructureHandshake
LLM Orca, Handshake's internal service that feature teams build LLM features on.
Arize AI customer story, published Aug 17, 2026Read the write-up (opens Arize AI customer story in a new tab)
Observation
Handshake built Datadog and Arize AX into its internal LLM Orca service, so every model call its feature teams make is traced for monitoring and evaluation in both.
Takeaway
Run both when two groups read the same calls: Datadog for the operators who watch production, a dedicated tool for the AI engineers who evaluate it.
What happened
Arize's customer story of Aug 17, 2026 says LLM Orca has Arize AX and Datadog integrations built in, so every call is traced, and that Handshake shipped more than 15 LLM use cases on it in under six months. Feature teams evaluate prompts, break down cost and export production traces in Arize AX.
“The user experience of LLM Orca is designed to be easy for a set of LLM use cases. Arize AX and Datadog integrations are built in, so every call is traced for evaluation and monitoring.”
Arize customer story, on Handshake
- Vendor customer story (Arize customer page, Arize being a Datadog competitor)
- Team metrics reported
Deployment details
LLM Orca, Handshake's internal service that feature teams build LLM features on.
- Vendor customer story (Arize customer page, Arize being a Datadog competitor)
- Team metrics reported
Team
Handshake ML infrastructure and its feature teams
Duration
Under six months for more than 15 use cases
Workload
LLM features built on the LLM Orca service
Product version
Datadog and Arize AX, both built into the service
Measurement
More than 15 LLM use cases in under six months
In SignalsNot counted. A competitor of Datadog published this story, and Handshake has published no account of its own.
Tomas AlabesSoftware developerSource.ag
A Node.js agent loop on LangGraph with LangChain and the Anthropic SDK, traced with Datadog LLM Observability.
GitHub (DataDog/dd-trace-js), published Jun 16, 2026Read the issue (opens GitHub (DataDog/dd-trace-js) in a new tab)
Observation
At Source.ag, LangChain and Anthropic spans for the same model call made Datadog's cost rollup overcount spend about 2.2 times.
Takeaway
Check that each model call produces one LLM span before trusting cost views, and turn off duplicate provider integrations under a framework.
What happened
With LLM Observability on, both the LangChain and the Anthropic integrations emitted an LLM span for each model call, nested one inside the other (GitHub issue, Jun 16, 2026). Datadog's cost rollup summed both layers, so the cost view did not deliver the team's real spend: it overcounted about 2.2 times, and the two spans disagreed on cache reads. The workaround was to turn the Anthropic integration off. Datadog closed the issue as completed in June 2026. A month later the same engineer found every MCP tool call captured twice, with the full payload stored on each span.
“…cost rollup sums both layers and overcounts spend (~2.2× in our case)…”
Tomas Alabes
- Public engineering write-up (the developer's GitHub issue)
- Team metric reported
Deployment details
A Node.js agent loop on LangGraph with LangChain and the Anthropic SDK, traced with Datadog LLM Observability.
- Public engineering write-up (the developer's GitHub issue)
- Team metric reported
Team
Source.ag, the team's own agent application
Workload
An agent loop on LangGraph calling Anthropic models through LangChain
Product version
dd-trace 5.106.x on Node.js 22 with LLM Observability enabled
Measurement
Cost rollup overcounted spend about 2.2 times
In SignalsCounted once, as the developer's own issue. In the record

How you buy it

US$160 a month

Pro billed annually, for the first 100,000 LLM spans a month. Free covers up to 40,000 LLM spans.

Source 1, Datadog, Datadog pricing: Agent Observability: Datadog pricing: Agent Observability

Details

Free covers up to 40,000 LLM spans a month with 15-day trace retention. Pro costs US$160 a month billed annually, US$200 billed month to month, or US$240 billed on demand, for the first 100,000 LLM spans a month.

Cost drivers

US$3.50 per 10,000 spans

LLM spans beyond the included 100,000, billed annually. On demand the rate is US$5.

Source 1, Datadog, Datadog pricing: Agent Observability: Datadog pricing: Agent Observability

Details

The metering unit is the LLM span, a single call to an LLM provider; tool, retrieval, and workflow spans are not billed. LLM spans beyond the included 100,000 cost US$3.50 per 10,000 billed annually, US$4.20 month to month, or US$5 on demand. Trace retention of 30, 60, or 90 days costs US$6.50, US$8, or US$9 per 10,000 LLM spans a month, and Sensitive Data Scanner is included at 1 GB per 10,000 LLM spans.

Where it runs

Datadog SaaS by site

Your organization's Datadog site. Each site is independent, and data is not shared across sites.

Source 2, Datadog, Datadog docs: Getting started with Datadog sites: Datadog docs: Getting started with Datadog sites Source 3, Datadog, Datadog docs: Agent Observability data governance: Datadog docs: Agent Observability data governance

Details

Datadog runs it as SaaS on the site your organization uses: US1, US3, and US5 in the US, EU1 in Germany, AP1 in Japan, AP2 in Australia, or UK1. Each site is independent, and data is not shared across sites.

What to verify

15-day default retention

Judge evaluations bill tokens to the model account you connect. Longer retention is an add-on.

Source 4, Datadog, Datadog docs: Connect an LLM provider for bring-your-own-key evaluations: Datadog docs: Connect an LLM provider for bring-your-own-key evaluations Source 3, Datadog, Datadog docs: Agent Observability data governance: Datadog docs: Agent Observability data governance Source 5, docs.datadoghq.com, Datadog docs: Single Sign On With SAML: Datadog docs: Single Sign On With SAML Source 6, docs.datadoghq.com, Datadog docs: User Provisioning with SCIM: Datadog docs: User Provisioning with SCIM Source 7, datadoghq.com, Datadog Legal: HIPAA-Eligible Services: Datadog Legal: HIPAA-Eligible Services Source 8, docs.datadoghq.com, Datadog docs: Datadog Audit Trail: Datadog docs: Datadog Audit Trail Source 1, Datadog, Datadog pricing: Agent Observability: Datadog pricing: Agent Observability

Details

For judge evaluations you bring your own key, and tokens bill to the account you connect. Traces keep 15 days by default, or 30, 60, or 90 days with a retention add-on. SCIM is available on the Infrastructure Pro, Infrastructure Enterprise, and Startup plans. Datadog lists Agent Observability as a HIPAA Extended Eligible Service under the applicable Business Associate Agreement. Audit Trail is a separate product, from 2% of monthly spend billed annually or 3% on demand.

Practitioner sentiment, teams and leaders

+32 Positive net

of 26 sources that take a side, each author counted once. Small sample.

How this figure is made

Current sentiment

Positive, mixed and negative sources

9161

The score counts them as 9 positive, 15 mixed and 1 negative votes, one per author.

The 26 counted team and leader accounts are 9 positive, 16 mixed, and 1 negative; two integration builders are listed and not counted. Twelve keep their agent traces in Datadog, most because the rest of their stack already runs there, and NonBioS chose self-hosted Langfuse instead. Missing or dropped traces are raised in 7 sources, and double-counted spans or wrong token totals in 5.

Run a pilot on one production app to measure which failures it surfaces and how long tracing them takes.

Evidence field. All 63 sources, one mark each.

Arrange by

Each mark is one source: positive, mixed, negative, no opinion stated, or documentation (not counted).

By sentiment, Teams and technical leaders. Every source GILD read, grouped as the field draws it.
Group	Sources	Positive	Mixed	Negative	No opinion stated and not counted in the score	Documentation, checked and not counted
Positive	9	9	0	0	0	0
Mixed	16	0	16	0	0	0
Negative	1	0	0	1	0	0
Not counted	3	0	0	0	3	0
Documentation	9	0	0	0	0	9
Outside this group	25	5	17	3	0	0

Every source GILD read, by the quarter it was published, as the field's time axis counts it. Documentation is dated by the day it was checked.
Quarter	Sources
Q1 2025	3
Q2 2025	4
Q3 2025	1
Q4 2025	3
Q1 2026	10
Q2 2026	10
Q3 2026	14
Q4 2026	9
Documentation, checked	9

By sentiment

9 documentation pages check facts and are not counted; 25 sit outside this group.

Hover a mark for a quick look, click it or press Enter to open its source.Tap a mark to open its source, then step to its neighbors with Previous and Next.

What practitioners praise

Top praise of 2. Each mark is a source on it; filled marks praise it.

One Trace With APM and InfrastructureAll 6 sources on it praise it

Most common concerns

Top concern of 2. Each mark is a source on it; filled marks raise it.

Missing or Dropped TracesAll 7 sources on it raise it as a concern

All 63 sources behind these findings

Datadog pricing: Agent ObservabilityDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: Agent ObservabilityDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: Automatic instrumentation for Agent ObservabilityDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: OpenTelemetry instrumentation for Agent ObservabilityDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: Agent Observability SDK referenceDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: Connect an LLM provider for bring-your-own-key evaluationsDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: Agent Observability data governanceDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog docs: Getting started with Datadog sitesDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Datadog Trust Hub: platform securityDatadog (Official Datadog pricing, documentation, and trust pages), Product documentation, checked Oct 28, 2026
Reply on agent tracing at workaiexplorations, X user who builds the Praval agent framework (self-described: Agent tracing at the author's workplace), Post on X, published Oct 22, 2026
Reply to @datadogdevs on Agent Observability traces over MCPOfek Shaked, AI engineer (X profile) (self-described: Debugging agent loops with Claude Code and Cursor), Post on X, published Oct 21, 2026
feat(datadog): add otel_llmobs_conversion and llmobs_gateway_moderoroghost17, Bifrost contributor (self-described: Bifrost AI gateway's Datadog plugin), GitHub pull request (Bifrost), published Oct 17, 2026
LLM Observability: streamed Amazon Nova invoke_model calls report 0 tokenspcbeingused333, GitHub user filing an issue (self-described: Amazon Bedrock Nova streaming through the botocore integration), GitHub issue, published Oct 17, 2026
[BUG]: llmobs.flush() is documented as blocking but returns before anything is sentAnurag Ojha, engineer at Mastra (GitHub profile) (self-described: Building the @mastra/datadog integration for short-lived processes), GitHub issue, published Oct 14, 2026
Promote #168 to main: fix Datadog LLM Observability tool spansIno de Bruijn, cBioPortal and Memorial Sloan Kettering (GitHub profile) (self-described: cBioPortal MCP server in production), GitHub pull request (cBioPortal), published Oct 12, 2026
[BUG]: Generated dd tracestate exceeds 256-character limit and is rejected by OpenTelemetryKyle Purdon, engineer at Semgrep (GitHub profile) (self-described: LLM Observability propagation alongside the OpenTelemetry API), GitHub issue, published Oct 5, 2026
[BUG]: Anthropic streamed spans report 1h cache writes as ephemeral_5m_input_tokensChristopher Krah, engineer at UiPath (GitHub profile) (self-described: Streamed Anthropic calls with 1-hour prompt caching), GitHub issue, published Oct 4, 2026
[BUG]: vllm integration raises AttributeError 'EngineCoreOutput' object has no attribute 'num_cached_tokens' with vLLM 0.20.0 and laterSierraJY, GitHub user filing an issue (self-described: vLLM served through Single Step Instrumentation), GitHub issue, published Oct 3, 2026
fix(datadog): include DD_TAGS on LLM Observability spanscflocke, engineer at codecentric AG (GitHub profile) (self-described: LiteLLM proxy operators attributing LLM usage by cost center), GitHub pull request (LiteLLM), published Sep 30, 2026
feat(mcp): default to Datadog when LLM Observability is enabledCaleb Evans, engineer at Airbyte (GitHub profile) (self-described: PyAirbyte's hosted MCP server), GitHub pull request (PyAirbyte), published Sep 30, 2026
[BUG]: litellm integration: fallbacks parameter creates duplicate LLM spans (one call to two spans with identical usage)Phillipe Silva (phillipehsantos), GitHub user filing an issue (self-described: LiteLLM acompletion calls with fallbacks), GitHub issue, published Sep 29, 2026
Instrument AI SDK with Datadog LLM Observability and add content-redaction telemetryMinh Cung (cungminh2710), GitHub user (self-described: Server-side Vercel AI SDK calls with customer content), GitHub pull request, published Sep 24, 2026
[Bug]: Datadog LLM Obs embedding spans lack top-level model_name, causing Partial costRafael G. Fernandes (rafagfe), GitHub user filing an issue (self-described: LiteLLM proxy sending to Datadog LLM Observability), GitHub issue (LiteLLM), published Sep 17, 2026
MonitorFormulaAndFunctionEventsDataSource missing llm_observability: multi-query llm-observability alert monitors cannot be createdJesse Mills (jrmils89), GitHub user filing an issue (self-described: Managing Datadog monitors with Terraform), GitHub issue, published Sep 8, 2026
[BUG]: LLMObs: errors creating openai spans - missing span kind in span contextZach Hammer, engineer at SeatGeek (GitHub profile) (self-described: Streamed OpenAI Responses calls that end early), GitHub issue, published Aug 18, 2026
[BUG]: google_adk LLMObs agent span records every streaming partial as a separate messageJonathan van Westendorp, GitHub user filing an issue (self-described: Google ADK agents in streaming mode), GitHub issue, published Aug 6, 2026
MCP integration broken with mcp 2.0.0 and later: ModuleNotFoundError: No module named 'mcp.shared.session'Felipe Paiva (felipepaivawebmotors), GitHub user filing an issue (self-described: MCP servers on mcp 2.0.0 and fastmcp 4), GitHub issue, published Aug 6, 2026
Anthropic AsyncMessageStream.get_final_message()/get_final_text() bypass TracedAsyncStream, so the span never finishesDan Aronson (AronsonDan), GitHub user filing an issue (self-described: Async Anthropic streams consumed with get_final_message), GitHub issue, published Jul 21, 2026
LLMObs: langchain + modelcontextprotocol-sdk integrations double-capture MCP tool callsTomas Alabes, engineer at Source.ag (GitHub profile) (self-described: Production Node.js agent calling MCP tools through LangChain), GitHub issue, published Jul 17, 2026
Comment on: My CTO forced a microservices migration for our tiny user baseQuickNick123, Reddit user, r/devops, self-described engineer at an S&P 500 company (self-described: Large company running most Datadog products), Reddit comment, published Jul 16, 2026
[BUG]: ai plugin leaks heap on every AI SDK call: LLMObs sub-plugin retains every tool() object forevervasilii-kurochkin, GitHub user filing an issue (self-described: Production chat service on the Vercel AI SDK), GitHub issue, published Jul 9, 2026
Add Datadog LLM Observability GenAI spansKentaro Wakayama, engineer at Coder Society (GitHub profile) (self-described: Veryfront agent runtime exporting over OTLP), GitHub pull request, published Jul 9, 2026
LiteLLM is becoming much more than just an LLM gateway. Has anyone actually deployed its Datadog integration in production?jeann1977, Reddit user, r/devops (self-described: Testing LiteLLM's Datadog integration before production), Reddit post, published Jun 29, 2026
[BUG]: invoke_model_with_response_stream token metrics are all 0 in LLMObs tracesJulien Perrin (jpr175), GitHub user filing an issue (self-described: Claude Haiku 4.5 on Amazon Bedrock, called with invoke_model_with_response_stream), GitHub issue, published Jun 26, 2026
[BUG]: google_adk LLMObs drops agent span (missing span kind) when a Gemini response Part is unhandledDaniel Polistchuck (dpolistwm), GitHub user filing an issue (self-described: Google ADK agents on Gemini), GitHub issue, published Jun 22, 2026
LLMObs: LangChain + provider (anthropic) integrations emit two llm spans per call, double-counting costTomas Alabes, engineer at Source.ag (GitHub profile) (self-described: Node.js agent on LangChain over the Anthropic SDK), GitHub issue, published Jun 16, 2026
Comment on: GenAI is the first cost line my allocation playbook completely falls apart onDifficultyIcy454, Reddit user, r/FinOps (self-described: FinOps team allocating GenAI spend), Reddit comment, published Jun 11, 2026
feat: add dd-trace for Datadog LLM ObservabilityGaofei Zhao (dippindots), cBioPortal contributor (self-described: cBioPortal's LibreChat deployment), GitHub pull request (cBioPortal), published Jun 9, 2026
Disable Datadog OTel LLMObs duplicate conversionRita Agafonova (rita-aga), GitHub user (self-described: Production agent exporting to Datadog through both OTLP and the LLMObs API), GitHub pull request, published Jun 9, 2026
[BUG]: LLM Obs anthropic plugin mutates caller's options.messages arrayTroy Rhinehart, engineer at Attentive (GitHub profile) (self-described: Anthropic tool runners and multi-turn loops in Node.js), GitHub issue, published Apr 25, 2026
[BUG]: Circular import in llmobs._experiment when anthropic is installed (ddtrace 4.6.0 and later)Daniel Bernardes, GitHub user filing an issue (self-described: Python app with anthropic installed and auto-patching on), GitHub issue, published Apr 15, 2026
Comment on: AI governance isn't failing because we lack regulationnijave, Reddit user, r/AI_Agents (self-described: Team already on Datadog for other monitoring), Reddit comment, published Apr 15, 2026
[FEATURE]: Graceful SIGTERM handling and Configurable shutdown timeout for LLMObs writersSirish Dwa (sdwa0), GitHub user filing a feature request (self-described: Python services on pods that restart during deployments), GitHub feature request, published Mar 25, 2026
Comment on: What do people use for tracing and observabilityWowSoWholesome, Reddit user, r/LangChain (self-described: LangSmith user weighing Datadog's evals), Reddit comment, published Mar 19, 2026
Datadog AI Observability: What the Getting Started Guide Doesn't CoverDaniel, DevOps engineer and author of the Signals & Codes newsletter (self-described: Moving an agentic LLM app from plain APM tracing to Agent Observability), Personal newsletter post, Signals & Codes, published Mar 19, 2026
Datadog's hallucination evaluatorDaniel, DevOps engineer and author of the Signals & Codes newsletter (self-described: Testing hallucination detection without a full retrieval pipeline), Personal newsletter post, Signals & Codes, published Mar 16, 2026
Anthropic streaming calls don't capture input/output content in LLM ObservabilityIsaac Sherrill (imsherrill), GitHub user filing an issue (self-described: Anthropic SDK streaming in Node.js), GitHub issue, published Feb 28, 2026
Add Datadog LLM Observability tutorial for AgentCore Runtime with Strands Agentskolaak, GitHub user filing a content request (self-described: Amazon Bedrock AgentCore Runtime with Strands Agents), GitHub issue (AWS AgentCore), published Feb 13, 2026
70% of our AI agent failures had nothing to do with AIRavenPack's Bigdata.com engineering team (self-described: Production financial research agent on LangChain), Engineering blog post, RavenPack on Medium, published Jan 29, 2026
[BUG]: LLM Observability should not throw an unhandled exception if the agent is unavailable.Ian Kaplan, engineer at Unify (GitHub profile) (self-described: Node.js service with LLM Observability enabled), GitHub issue, published Jan 28, 2026
Comment on: Ask HN: Where are you keeping your LLM logs?platypii, Hacker News commenter (self-described: Choosing where to store LLM logs), Hacker News comment, published Jan 9, 2026
Comment on: Anyone else finding observability for LLM workloads is a completely different beast?RasheedaDeals, Reddit user, r/devops (self-described: Team that already ran Datadog for its main stack), Reddit comment, published Jan 9, 2026
dd-trace-js issue 7146: ToolLoopAgent in AI SDK 6 and LLM Observability tracesMustafa Akin, engineer at JumpCloud (GitHub profile) (self-described: Agent on AI SDK 6 ToolLoopAgent), GitHub issue, published Dec 19, 2025
[BUG]: ddtrace breaks applications with OpenAI usageDmitry (vilmar-hillow), GitHub user filing an issue (self-described: Python app creating an OpenAI client with Azure identity), GitHub issue, published Dec 10, 2025
[BUG]: LLMObs for ai-sdk not capturing traceScott Prue (prescottprue), GitHub user filing an issue (self-described: Next.js app on the Vercel AI SDK), GitHub issue, published Oct 22, 2025
Comment on: LiteLLM started breaking down for us past 300 RPS, what are folks using in prod?SigmoidGrindset, Reddit user, r/LocalLLaMA (self-described: Team running an LLM gateway in production), Reddit comment, published Aug 6, 2025
Datadog AI agent observability, security seek to boost trustBert Stewart, head of the global command center at Thomson Reuters, quoted by TechTarget (self-described: Enterprise operations team assessing Datadog's new AI features), Trade press article, TechTarget, published Jun 12, 2025
[FEATURE]: LLMObs - Instrumentation for mastraKosuke Oya, engineer at PLAID (GitHub profile) (self-described: Mastra agents running in production), GitHub feature request, published Apr 27, 2025
[BUG]: LLM Observability Spans Not Appearing When Payload > 1MBlucasiscovici, GitHub user filing an issue (self-described: The team's LLM app sending spans to Datadog's EU intake), GitHub issue, published Apr 23, 2025
Datadog LLM observability alternativesPictti, Reddit user, r/learnmachinelearning (self-described: Looking for an alternative after using Datadog for LLM observability), Reddit post, published Apr 3, 2025
The NonBioS LLM observability pickAmit, author on the NonBioS engineering blog (self-described: NonBioS choosing an observability tool for its AI software engineering agent), Company engineering blog, NonBioS, published Mar 5, 2025
[BUG]: LLMObs agentLessEnabled does not mean no agent?TJ Koblentz (tjk), GitHub user filing an issue (self-described: The team's Node.js services on Amazon Linux, reaching the Datadog agent over a Unix socket), GitHub issue, published Feb 5, 2025
[BUG]: Traces Dropped After 5 Retries When Sending to Local Agent ProxyRyuichi Watanabe (ryuichi1208), GitHub user filing an issue (self-described: The team's WebSocket service on gunicorn with uvicorn workers), GitHub issue, published Jan 22, 2025

Jan 2025 to Oct 2026.

54 sources from practitioners, by month and sentiment.

The full readExplore all Datadog Agent Observability Signals4 praise themes, 4 concerns, 1 point of agreement and 1 disagreement across all three groups, with all 63 sources open to inspect.

Datadog Agent Observability is Core across Evaluate and Operate and touches Model, Build, and Data.

Business area: Product & Engineering
Function: Software Development
Operating stack: AI Application Platform
Primary role: Agent Observability and Evaluation

Datadog Agent Observability beside the rest of its stackThe Stack Field, every product on every layer

FootprintDatadog Agent Observability across the AI Application Platform stack. Bar height is its role on each layer.

Model
1. Model APIs: Touches
  Model APIs, touches. Datadog traces application calls to OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Vertex AI, Google GenAI, and MistralAI. LLM judges run on the model account you connect. It supplies no models.
2. Model gateway: Touches
  Model gateway, touches. The Python SDK auto-instruments LiteLLM, and judges can call any OpenAI-compatible endpoint you register as a custom provider. The gateway controls routing and spend limits.
Build
1. Orchestration: Touches
  Orchestration, touches. The SDKs auto-instrument LangChain, LangGraph, CrewAI, the OpenAI Agents SDK, the Claude Agent SDK, Google ADK, Pydantic AI, and the Vercel AI SDK. The framework builds and runs the agent.
2. Tool use: Touches
  Tool use, touches. Tool spans record arguments and outputs, and the Python and Node.js SDKs trace MCP calls. Tool spans are not billed. The application executes the tools.
Data
1. Retrieval: Touches
  Retrieval, touches. Retrieval spans show the documents a step returned, and are not billed. The vector store handles search and indexing.
Evaluate
1. Evaluation: Core
  Evaluation, core. Versioned datasets, offline experiments, a playground, and annotation queues test prompt and model changes before release. Custom LLM-as-a-judge evaluators run on your own model account, beside managed evaluators and the NeMo integration.
Operate
1. Observability: Core
  Observability, core. Every request becomes a trace of agent, workflow, LLM, tool, and retrieval spans with cost, latency, and token counts. Traces share a trace ID with Datadog APM, infrastructure, and real user monitoring, and Insights flags recurring cost and reliability problems.
2. Cloud deployment: Touches
  Cloud deployment, touches. Datadog runs as SaaS on regional sites in the US, Germany, Japan, Australia, and the UK. Agent Observability is not offered on the two US government sites, and it hosts neither your application nor your models.
3. Guardrails: Supports
  Guardrails, supports. Datadog Agent Observability includes the Sensitive Data Scanner managed evaluator. It scans LLM prompt response pairs, identifies sensitive information, and redacts it.

Market mapThe layers Datadog Agent Observability covers, and who else owns each one.

Evaluation & testing
CoreEvaluate
Observability & tracing
CoreOperate
Guardrails & AI security
SupportsOperate
- Prisma AIRS
- NeMo Guardrails
Model APIs
TouchesModel
Model gateways & routing
TouchesModel
- LiteLLM
- OpenRouter
- Portkey
- Vercel AI Gateway

4 more layers on the full map. Core, Supports and Touches describe coverage. They say nothing about quality.

Datadog Agent Observability

The 30-Second Take

Comparisons