The call
Start on the Free tier with one agent and a week of real traffic, and commit to a Pro term only after Datadog's span and token totals match your provider invoices.
Datadog Agent Observability pays off when every model call lands once in a trace your SREs already read. Choose the SDK or OpenTelemetry path, mask sensitive fields before spans leave the application, and count LLM spans per run against your model bills before you pick a billing term. Then set retention, access, judge accounts, and an experiment gate for prompt and model changes.
When you need this
- An agent is moving to production on services Datadog APM already traces, and nobody can say which step of a bad run failed.
- Finance asks what tracing will cost once traffic grows, and the team has not counted model calls per run.
- Security must approve sending prompts and outputs to Datadog before rollout.
Rollout steps
Instrumentation path
How do spans reach Datadog?
What to doThe SDK for Python, Node.js, or Java (Go is experimental) auto-instruments OpenAI, Azure OpenAI, Anthropic, Bedrock, Vertex AI, Google GenAI, LangChain, LangGraph, CrewAI, LiteLLM, the OpenAI Agents SDK, the Claude Agent SDK, Pydantic AI, MCP, and vLLM in Python, and a shorter list including the Vercel AI SDK in Node.js. Each integration has a minimum library and tracer version. Without the SDK, send OpenTelemetry spans that follow the GenAI 1.37+ conventions or OpenInference, OpenLLMetry 0.47+, or Langfuse attributes.
Sensitive data
Which inputs and outputs must never leave the application?
What to doWrite SDK span processors that redact inputs and outputs or drop a span before it is sent. Sensitive Data Scanner then scans and redacts what arrives, with 1 GB of scanning included per 10,000 LLM spans. Customers under Datadog's BAA may use third-party generative AI services only if they are listed as HIPAA-eligible, so check that list first.
Span count
How many LLM spans does one run produce?
What to doDatadog bills one LLM span per call to a model provider, and tool, retrieval, and workflow spans are free. Public bug reports show how the count goes wrong: a Source.ag engineer saw nested LangChain and Anthropic spans overcount spend about 2.2 times, LiteLLM fallbacks emitted two LLM spans per successful call, and MCP tool calls were captured twice. Count spans per run on a week of traffic and compare token totals with your invoices.
Billing term and sampling
Which term and volume control fit?
What to doPro costs US$160 a month billed annually, US$200 month to month, or US$240 on demand for 100,000 LLM spans, then US$3.50, US$4.20, or US$5 per 10,000 more. Free covers 40,000 a month. Billing follows the spans you send, so a trace sampling rate set in the SDK lowers cost by keeping a fraction of traces; sampling needs ddtrace 4.12.0, dd-trace 5.110.0, or dd-trace-java 1.66.0.
Retention
How long must traces and experiments stay queryable?
What to doTraces keep 15 days on every plan and ml_obs metrics 15 months. Offline experiments keep 15 days on Free and on demand, and 90 days on a committed plan. The current version of a dataset keeps 3 years. Trace retention of 30, 60, or 90 days costs US$5, US$6.50, or US$7.50 per 10,000 LLM spans a month billed annually, and up to US$9 on demand. Datadog's account team arranges it, and Free cannot add it.
Judge evaluations
Whose model runs the judges, and who pays?
What to doCustom LLM-as-a-judge evaluators run on an account you connect: OpenAI, Azure OpenAI, Anthropic, Amazon Bedrock, Vertex AI, or an OpenAI-compatible custom provider. Datadog calls these bring-your-own-key evaluations, so your provider bills the judge tokens. Datadog does not support data residency for an OpenAI judge account. Tracing your evaluations adds billed spans.
Release gate
What must an experiment show before a prompt or model change ships?
What to doBuild versioned datasets from production traces, then run offline experiments that compare prompts, models, and agent logic on the same inputs. Agree on each evaluator's threshold before the first run. Managed evaluators flag language mismatch and sensitive data, and the NeMo integration and the external evaluations API bring other scores in.
Access and audit
Who can see which traces, and who changed what?
What to doRestrict each AI application's traces, experiments, datasets, and annotation queues to chosen teams with the ml_app tag. SAML single sign-on and custom roles cover the organization. SCIM is documented for the Infrastructure Pro and Infrastructure Enterprise plans. Audit Trail costs 2% of monthly spend billed annually, or 3% otherwise, and keeps events 90 days by default.
Questions to settle internally
- How many LLM spans does a typical run produce, and do Datadog's token totals match our provider invoices?
- Which tracer versions did we validate, and who retests span counts before an upgrade?
- Which fields are redacted in span processors, and who signs off on the rules?
- Which applications need more than 15 days of trace retention, and on which billing term?
- Which model account runs our judges, and does it meet our residency and HIPAA terms?
Red flags
- Committing to a Pro term before anyone has counted LLM spans per run on real traffic.
- Taking Datadog's cost views as the invoice without reconciling them with the model providers' bills.
- Upgrading the tracer in production without rechecking span counts on staging.
- Raw prompts from a regulated workload traced with no span processor redacting them.
Products in this piece
Sources
- Official documentation
- 15
- Public source
- 3
- Datadog pricing: Agent Observability
- Datadog pricing list
- Datadog docs: Agent Observability
- Datadog docs: Agent Observability SDK reference
- Datadog docs: Automatic instrumentation for Agent Observability
- Datadog docs: OpenTelemetry instrumentation for Agent Observability
- Datadog docs: Agent Observability evaluations
- Datadog docs: LLM-as-a-judge evaluations
- Datadog docs: Connect an LLM provider for bring-your-own-key evaluations
- Datadog docs: Managed evaluators
- Datadog docs: Agent Observability offline experiments
- Datadog docs: Agent Observability data governance
- Datadog docs: User provisioning with SCIM
- Datadog docs: Datadog Audit Trail
- Datadog docs: HIPAA compliance
- dd-trace-js issue #8936: two LLM spans per call overcount spend about 2.2x (Jun 16, 2026)
- dd-trace-js issue #9416: every MCP tool call captured twice (Jul 17, 2026)
- dd-trace-py issue #20670: LiteLLM fallbacks emit two LLM spans per successful call (Sep 29, 2026)