Arize earns its cost when every failure your team finds in a trace becomes a dataset example and an evaluator that checks each later change. Instrument once with OpenInference, mask sensitive fields before spans leave the application, and agree on pass thresholds before a prompt or model change ships. The same steps work on self-hosted Phoenix and on managed Arize AX.
When you need this
- An agent is moving from prototype to production, and nobody can say why a given run went wrong.
- Prompt or model changes ship on a hunch, and regressions reach the team as user complaints.
- The team is choosing between self-hosted Phoenix and managed Arize AX and wants instrumentation that works with both.
Evaluation criteria
Instrument once
How do spans reach Arize?
Look forRegister framework and provider instrumentors with arize-phoenix-otel for Phoenix or arize-otel for AX. EU and Canada AX spaces override the US default and send traces to their regional OTLP host, otlp.eu-west-1a.arize.com or otlp.ca-central-1a.arize.com. OpenInference emits standard OpenTelemetry spans, so other backends can ingest the same data. During a migration, Arize recommends sending spans to both Phoenix and AX for one traffic cycle.
Name projects and sessions
Will each service's traces land where its owners look?
Look forSet a project name for each service and environment. One engineer found unnamed spans mixed with other runs in Phoenix's default project. Add session and user IDs so the team can rebuild a multi-turn conversation from its traces.
Keep sensitive data out
Which inputs and outputs must never leave the application?
Look forSet OPENINFERENCE_HIDE_INPUTS and OPENINFERENCE_HIDE_OUTPUTS to drop inputs and outputs from spans. Finer switches cover messages, images, embeddings, and prompts. Redact emails, card numbers, and other PII in a span processor with regex or Microsoft Presidio. Do the masking in the application, before spans reach Phoenix or AX.
Turn failures into datasets
How does a bad run become a test case?
Look forAdd failed traces to a dataset and label them. Phoenix labels them with annotations. AX labeling queues send a set share of matching spans to reviewers. PagerDuty keeps its golden questions as AX datasets and refreshes the ground truth for answers that change daily, such as incident counts.
Choose evaluators by cost
Which checks need a judge model?
Look forUse code evaluators for objective checks such as format, exact values, and tool arguments. Use an LLM judge when no rule can express the criterion. For long agent trajectories, AX has offered agent-as-a-judge on every plan since September 1, 2026. Calibrate each judge against human labels. Judge calls run on your configured provider keys, so your model account pays for them.
Gate changes on experiments
What must an experiment show before a prompt or model change ships?
Look forRun the dataset against the new version and compare it with the last version on identical inputs. Agree on each evaluator's pass threshold before the first run. AX documents CI templates for GitHub Actions, GitLab, Jenkins, Azure DevOps, and Harness. PagerDuty runs these experiments as regression tests for every agent.
Score production
What tells you quality dropped after release?
Look forSet a sampling rate for continuous AX online evals. Arize recommends code evals on every span and LLM judges on a fraction of traffic. PagerDuty scores relevance, groundedness, and tool selection without ground truth. Phoenix runs evals on demand against stored traces, so continuous scoring needs AX.
Alert on thresholds
Who hears about a quality drop first?
Look forSchedule AX monitors on span attributes, eval labels, or custom metrics. Route notifications to Slack, PagerDuty, Opsgenie, or Microsoft Teams. PagerDuty pages the agent team's on-call engineer at low urgency and opens a Jira ticket for each quality breach. Wayfair alerts on unusual changes in its flagging metrics.
Questions to ask vendors
- Which projects exist, who owns each, and which environment sends to which?
- Which fields are masked before export, and who signs off on the rules?
- Which evaluators gate a release, at what thresholds, and who checks judge scores against human labels?
- Who moves a production failure from a trace into a dataset, and how?
- Which monitors page a person, and which only open a ticket?
Red flags
- Raw prompts and outputs from a regulated workload traced with no masking.
- LLM judges on every production span with no sampling, billed to the team's model account.
- Judge scores treated as ground truth, with no human labels to check them.
- Instrumentation written against one backend's SDK when OpenInference would keep it portable.
Products in this piece
Sources
- Official documentation
- 12
- Public source
- 3
- Arize AX docs: Set up tracing
- Arize AX docs: Mask and redact sensitive data
- Arize AX docs: Human review
- Arize AX docs: Run online evals on traces
- Arize AX docs: View eval results and costs
- Arize AX docs: CI/CD for automated experiments
- Arize AX docs: Continuously monitor
- Arize AX docs: Migrate from Phoenix to Arize AX
- Phoenix docs: What is Arize Phoenix?
- Phoenix docs: Braintrust open source alternative? (Arize's feature table)
- Arize AX docs: Release notes (13-month retention on Pro, Jul 10, 2026; Signal general availability, Jul 29, 2026)
- Arize AX docs: Data regions and whitelisting
- PagerDuty engineering blog: PagerDuty + Arize, building end-to-end observability for AI agents in production (Aug 13, 2025)
- Wayfair tech blog: Flag Early, Fix Faster, real-time intervention for supplier transfers with Wilma (Jan 15, 2026)
- Folarin Akinloye: Evaluate agents locally with Arize Phoenix (Jul 12, 2026)