GILD verdict
Choose Datadog when operators already use it, Arize when AI engineers need their own evaluation workspace, and both when both groups read the calls.
- The workflow
- a platform or AI team deciding where agent traces live and who reads them
- The decisive evidence
- ReportedArize's customer story of Aug 17, 2026 says Handshake's LLM Orca service has Arize AX and Datadog built in, so every call reaches both.1
- ReportedA Source.ag engineer reported on Jun 16, 2026 that duplicate LLM spans made Datadog's cost rollup overcount spend about 2.2 times with LangChain over the Anthropic SDK.2
- GILD analysisThe AX bill at volume is unresolved. GILD's calculation from published rates puts 200,000 traces with 2 model calls each at US$465 a month on Datadog's annual commitment, while AX Pro's US$50 base fee leaves 1,950,000 spans unpriced until Arize quotes a rate.
- The tradeoff
- Handshake runs both; Datadog puts agent traces beside APM on one bill you can compute, and Arize offers self-hosting. Datadog bills only model calls and keeps traces 15 days unless you buy retention; Arize meters every span and keeps 30 days on Pro, or as long as you like on self-hosted Phoenix.15613
- What would change the call
- The call moves when Arize publishes an AX Pro overage rate, or when Datadog's span counts match your provider invoice for a month of real traffic.
The named teams split by who reads the traces. Handshake built Arize AX and Datadog into one internal LLM service, so AI engineers evaluate in Arize while every call also reaches Datadog. Datadog's bill grows only with model calls, because tool, retrieval, and workflow spans are free, while Arize AX meters every span plus gigabytes. Arize is the side you can self-host, through free Phoenix or AX Enterprise.
What teams report
Failures and incidents 1
Source.agOn Datadog Agent Observability
With LangChain over the Anthropic SDK, dd-trace-js emitted two LLM spans for each model call, and Datadog's cost rollup summed both, overcounting Source.ag's spend about 2.2 times (GitHub issue, Jun 16, 2026).
“…cost rollup sums both layers and overcounts spend (~2.2× in our case)…”
Tomas Alabes, Engineer, Source.agGitHub, DataDog/dd-trace-js, , post
Left or turned down 1
NonBioSOn Arize
NonBioS weighed about a dozen tools, Phoenix and Datadog among them, and chose self-hosted Langfuse. It passed on Phoenix after a brief look at its naming and website (Mar 5, 2025).
“We looked at Phoenix Arize very briefly, but the naming was a bit confusing. Is it Phoenix, or is it Arize? The website also seemed challenging from a usability perspective, and we were left wondering if that would translate to the tool as well.”
Amit, Author, NonBioS engineering blog, NonBioSNonBioS, , write-up
Run both 1
Handshake
Handshake's internal LLM Orca service has Arize AX and Datadog built in, so every call is traced for evaluation in Arize and monitoring beside the rest of its stack. Handshake shipped more than 15 LLM use cases on it in under six months.
“The user experience of LLM Orca is designed to be easy for a set of LLM use cases. Arize AX and Datadog integrations are built in, so every call is traced for evaluation and monitoring.”
Kyle Gallatin, Technical Lead Manager of ML Infrastructure, HandshakeArize AI, , write-upVendor customer story
Choosing between Arize and Datadog Agent Observability
Choose Arize if
- AI engineers want a dedicated workspace for datasets, experiments, prompt iteration, and labeling, as Handshake's feature teams use Arize AX for evals and cost breakdowns.
- Traces must stay in your own network: Phoenix self-hosts free and Arize AX self-hosts on Enterprise.
- The same team also monitors classic ML models for drift and performance, which Arize logs beside its LLM traces.
Choose Datadog Agent Observability if
- Your SRE and platform teams already run Datadog and want agent traces under the same trace ID as APM, infrastructure, and real user monitoring.
- You want a bill you can compute: GILD's calculation puts 200,000 traces with 2 model calls each at US$465 a month on an annual commitment.
- Agent runs have many tool and retrieval steps, which Datadog does not bill. Check the cost rollup against your provider invoice, since Source.ag found duplicate LLM spans overcounting spend about 2.2 times.
Side by side
| Dimension | Arize | Datadog Agent Observability |
|---|---|---|
| What teams report | Handshake built Arize AX into its LLM Orca service for evals, cost breakdowns and trace export, with more than 15 use cases shipped in under six months (Arize customer story, Aug 17, 2026).1 | Handshake's LLM Orca has Datadog built in beside Arize AX. Source.ag found dd-trace-js emitting two LLM spans per model call with LangChain over the Anthropic SDK, so the cost rollup overcounted spend about 2.2 times (GitHub, Jun 16, 2026).12 |
| PlansList prices in US dollars. Agent Observability can be bought without other Datadog products. | Phoenix free to self-host; AX Free US$0; AX Pro US$50 a month; AX Enterprise custom4 | Free up to 40,000 LLM spans a month; Pro from US$160 a month on an annual commitment, US$200 month to month, or US$240 on demand, for the first 100,000 LLM spans13 |
| Billing unitExample: an agent run with 10 spans, 3 of them model calls, is 10 Arize spans and 3 Datadog LLM spans. | Spans of every type, plus gigabytes ingested | LLM spans only, each a call to a model; tool, retrieval, and workflow spans are not billed |
| Usage price | Pro includes 50,000 spans and 10 GB a month, and Arize's pricing page lists no overage rate (Oct 28, 2026); Enterprise volume is custom418 | US$3.50 per 10,000 LLM spans on an annual commitment, US$4.20 month to month, US$5 on demand13 |
| 200,000 traces a month, 10 usersGILD's calculation from published rates, Oct 28, 2026. Inputs: 200,000 traces a month, 10 spans and 2 model calls per trace, 20 KB per trace, 10 users, 30 days retained, no judge calls. | US$50 AX Pro base fee, plus 1,950,000 spans past the 50,000 included at a rate Arize no longer publishes, so the total needs a quote. Self-hosted Phoenix on one 4-vCPU, 8 GiB VM: about US$48 a month in infrastructure, before operations time.45 | US$465 a month on an annual commitment: US$160 for the first 100,000 LLM spans, US$105 for the next 300,000 at US$3.50 per 10,000, and US$200 to keep 400,000 LLM spans for 30 days. Month to month, US$554.1314 |
| Retention | 15 days on AX Free; 30 days on AX Pro, with a self-serve 13-month add-on; custom on Enterprise; Phoenix keeps data until a retention policy you set removes it | Traces 15 days, trace metrics 15 months, datasets 3 years; 30, 60, or 90-day trace retention add-ons at US$6.50, US$8, or US$9 per 10,000 LLM spans a month |
| Instrumentation | OpenTelemetry with OpenInference conventions and auto-instrumentation for major frameworks and providers | Datadog SDK for Python, Node.js, and Java, or OpenTelemetry spans following the GenAI 1.37+ conventions, OpenInference, OpenLLMetry, or Langfuse attributes |
| Where it runs | AX as SaaS in US, EU, or Canada regions, or self-hosted on Kubernetes under Enterprise; Phoenix on your own Docker or Kubernetes, air-gapped if needed | Hosted by Datadog on its regional sites, each fully independent; confirm Agent Observability is offered on your site |
| Sensitive data | Mask inputs and outputs and redact PII in a span processor before export; delete ingested spans through the API | Sensitive Data Scanner included, 1 GB of scanning per 10,000 LLM spans, to detect and redact sensitive data |
| Ownership | Arize AI, part of Dynatrace since October 2026; AX and Phoenix remain available | Datadog, which also sells APM, infrastructure, log, and user monitoring |
What practitioners report, side by side
Each product's figures come from its own practitioners, for the group you pick.
Overall signal
Arize reads +26 and Datadog Agent Observability +32, each from its own practitioners. Arize: 11 positive, 4 negative of 27. Datadog Agent Observability: 9 positive, 1 negative of 26.
The score counts the authors from teams and leaders that take a side, one vote each with their latest view, and is how far positive votes lead negative ones, per 100 votes, so Datadog Agent Observability's +32 rests on votes split 9 positive, 15 mixed and 1 negative. Each of Arize's authors wrote one source here, so each source is one vote.
A lead of a fifth of the votes reads positive and a lead of half strong positive, with the same lines below zero. The dots show each source, positive, mixed, negative. Run a pilot on one production app to measure which failures it surfaces and how long tracing them takes.
Footprint on the AI Application Platform stack
Both cover the same 9 layers at the same depth. Coverage is each product's placement on this stack, as its product page shows it. Core is a job the product owns; Supports and Touches are jobs it helps with. Coverage says nothing about how well a product does them.
| Stage | Arize | Datadog Agent Observability |
|---|---|---|
| Model2 layers | Touches 2 | Touches 2 |
| Build2 layers | Touches 2 | Touches 2 |
| Data1 layer | Touches 1 | Touches 1 |
| Evaluate1 layer | Core 1 | Core 1 |
| Operate3 layers | Core 1, Supports 1, Touches 1 | Core 1, Supports 1, Touches 1 |
Show all 9 layers
| Layer | Arize | Datadog Agent Observability |
|---|---|---|
| Model | ||
| Model APIs | Touches | Touches |
| Model gateway | Touches | Touches |
| Build | ||
| Orchestration | Touches | Touches |
| Tool use | Touches | Touches |
| Data | ||
| Retrieval | Touches | Touches |
| Evaluate | ||
| Evaluation | Core | Core |
| Operate | ||
| Observability | Core | Core |
| Cloud deployment | Touches | Touches |
| Guardrails | Supports | Supports |
What practitioners praise
Among teams and leaders, Arize alone has three praise themes that reach 5 sources from 3 authors. Trace-Level Debugging and Free, Simple Self-Hosting lead, at 10 of 11 and 6 of 6. One Trace With APM and Infrastructure and Auto-Instrumentation of Libraries reach 5 sources from 3 authors only for Datadog Agent Observability, at 6 of 6 and 4 of 5. Arize needs a pilot on the same production traffic before you choose on either.
| Praised for | Arize | Datadog Agent Observability |
|---|---|---|
| Trace-Level Debugging | 10 of 11 | Not a theme in its sources |
| Free, Simple Self-Hosting | 6 of 6 | Not a theme in its sources |
| OpenTelemetry & OpenInference | 5 of 6 | Not a theme in its sources |
| One Trace With APM and Infrastructure | Not a theme in its sources | 6 of 6 |
| Auto-Instrumentation of Libraries | Not a theme in its sources | 4 of 5 |
Each count is the sources that praise a theme, or raise it as a concern, out of those that discuss it. A row pairs the two products when both carry a theme under the same or an equivalent name.
A count ending in "so far" has under 5 sources from 3 authors among teams and leaders. "Not a theme in its sources" means GILD found no theme of that name in that product's sources.
What practitioners raise as concerns
Among teams and leaders, Missing or Dropped Traces and Double-Counted Spans & Wrong Token Totals reach 5 sources from 3 authors only for Datadog Agent Observability, at 7 of 7 and 5 of 5. Arize needs a pilot on the same production traffic before you choose on either. Scale & Self-Host Operations reaches 5 sources from 3 authors only for Arize, at 6 of 6. Datadog Agent Observability needs a pilot on the same production traffic before you choose on it.
| Raised as a concern | Arize | Datadog Agent Observability |
|---|---|---|
| Scale & Self-Host Operations | 6 of 6 | Not a theme in its sources |
| Missing or Dropped Traces | Not a theme in its sources | 7 of 7 |
| Double-Counted Spans & Wrong Token Totals | Not a theme in its sources | 5 of 5 |
18 sources
- Arize customer story: How Handshake deployed and scaled 15+ LLM use cases in under 6 months (Aug 17, 2026)
- GitHub: DataDog/dd-trace-js#8936, duplicate LLM spans overcount spend (Jun 16, 2026)
- NonBioS: The NonBioS LLM observability pick (Mar 5, 2025)
- Arize AX pricing
- Arize AX docs: Pricing and usage
- Phoenix docs: Self-hosting
- Arize AX docs: Self-hosted
- Arize AX docs: Set up tracing
- Arize AX docs: Mask and redact sensitive data
- Arize AX docs: Delete traces with sensitive data
- Arize docs: ML models (predictions, drift, and performance)
- Dynatrace: Dynatrace completes acquisition of Arize (Oct 1, 2026)
- Datadog Agent Observability pricing
- Datadog pricing list
- Datadog docs: Agent Observability
- Datadog docs: OpenTelemetry instrumentation for Agent Observability
- Datadog docs: Getting started with Datadog sites
- Arize: Compare Arize