Arize+26 Positive

Operator Insights

What are teams learning from building on Arize and running it in production?

11 accounts: 4 left it or turned it down, 1 switched, 1 evaluated it and 5 adopted it

Left it or turned it down4

mfundnContributor, bbvch-ai AI Hubbbv
bbvch-ai AI Hub, a product bbv wants to offer as SaaS, traced with OpenInference instrumentation.
GitHub, bbvch-ai/aihub-core, published Feb 10, 2026Read the write-up (opens GitHub, bbvch-ai/aihub-core in a new tab)
Observation
bbv's AI Hub team dropped Arize Phoenix for Langfuse in February 2026, citing Phoenix's experiment complexity and limited cost tracking.
Takeaway
If your team runs experiments from a UI and needs cost per call, test Phoenix and Langfuse on those jobs first: bbv gave up Phoenix's programmatic evaluation API to get both in Langfuse.
What happened
mfundn opened pull request #907 on Feb 10, 2026, and it merged on Feb 12. It replaced the Phoenix dataset API with Langfuse and moved tracing to OpenTelemetry and Langfuse spans. The decision record written for the review gives Phoenix's experiment complexity, limited cost tracking and development focus as the context, and Langfuse's UI-driven experiments, native cost tracking and Swiss data sovereignty as the drivers, at the cost of a programmatic evaluation API.
“Replace Phoenix dataset API with Langfuse…”
mfundn
- Public engineering write-up (GitHub pull request)
- No team metrics reported
Deployment details
bbvch-ai AI Hub, a product bbv wants to offer as SaaS, traced with OpenInference instrumentation.
- Public engineering write-up (GitHub pull request)
- No team metrics reported
Team
bbv AI Hub developers
Duration
Opened Feb 10, 2026; merged Feb 12, 2026
Workload
Agent tracing and eval datasets for the AI Hub
Product version
Self-hosted Phoenix, replaced by self-hosted Langfuse

NonBioSMar 5, 2025NonBioS blogNonBioS ruled out Arize Phoenix after a brief look at its naming and website, and chose self-hosted Langfuse instead.
AmitAuthor, NonBioS engineering blogNonBioS
NonBioS, an AI software engineering agent company, choosing an LLM observability tool.
NonBioS blog, published Mar 5, 2025Read the write-up (opens NonBioS blog in a new tab)
Observation
NonBioS ruled out Arize Phoenix after a brief look at its naming and website, and chose self-hosted Langfuse instead.
Takeaway
NonBioS dropped Phoenix from its shortlist after a brief look, on its naming and its website's usability; if Phoenix is on yours, judge it in a trial run.
What happened
In March 2025 NonBioS compared a dozen offerings, including LangSmith, Lunary, Phoenix, Portkey, Datadog, Langfuse and Helicone. It looked at Phoenix only briefly: the Phoenix and Arize naming confused the team, and the website seemed hard to use, which made it wonder whether the tool would be too. It chose self-hosted Langfuse.
“We looked at Phoenix Arize very briefly, but the naming was a bit confusing. Is it Phoenix, or is it Arize? The website also seemed challenging from a usability perspective, and we were left wondering if that would translate to the tool as well.”
Amit
Public engineering write-up
No team metrics reported
Deployment details
NonBioS, an AI software engineering agent company, choosing an LLM observability tool.
Public engineering write-up
No team metrics reported
Team
NonBioS engineering
Workload
Tracing for NonBioS's coding agent
Product version
Phoenix, looked at briefly
ElasticJul 29, 2026GitHub, elastic/kibanaElastic dropped Arize Phoenix from Kibana's evaluation framework once an executor inside Kibana stored datasets in Elasticsearch.
arturoliduenaContributor to Kibana's evaluation frameworkElastic
Kibana's LLM evaluation framework (kbn-evals), run in CI and locally by Kibana engineers.
GitHub, elastic/kibana, published Jul 29, 2026Read the write-up (opens GitHub, elastic/kibana in a new tab)
Observation
Elastic dropped Arize Phoenix from Kibana's evaluation framework once an executor inside Kibana stored datasets in Elasticsearch.
Takeaway
Phoenix works as a first backend for evals. A team that already runs a datastore can fold datasets into it and retire the extra service.
What happened
Kibana's evaluation framework launched with Phoenix as its primary backend for dataset storage and experiment orchestration. An executor inside Kibana then stored datasets in Elasticsearch, which made the Phoenix path redundant for CI and local runs. A pull request opened Jul 29, 2026 and merged Aug 7, 2026 removed the Phoenix executor and the @arizeai/phoenix-client dependency.
Public engineering write-up (GitHub pull request)
No team metrics reported
Deployment details
Kibana's LLM evaluation framework (kbn-evals), run in CI and locally by Kibana engineers.
Public engineering write-up (GitHub pull request)
No team metrics reported
Team
Kibana engineers working on its evaluation framework
Duration
From the framework's launch to Aug 7, 2026
Workload
Eval datasets and experiment runs in CI and on laptops
Product version
Phoenix through @arizeai/phoenix-client
ApifySep 14, 2026GitHub, apify/apify-mcp-serverApify dropped its Phoenix eval runner after Langfuse-based gates replaced it, because the runner's verdicts did not transfer to how Claude Code prompts.
Jiří SpilkaEngineerApify
Evals for Apify's MCP server, which coding agents call as a tool.
GitHub, apify/apify-mcp-server, published Sep 14, 2026Read the write-up (opens GitHub, apify/apify-mcp-server in a new tab)
Observation
Apify dropped its Phoenix eval runner after Langfuse-based gates replaced it, because the runner's verdicts did not transfer to how Claude Code prompts.
Takeaway
An eval runner has to reproduce how the real client calls your tool. Check client dependencies against the SDKs your agents use before you build on them.
What happened
In a pull request of Sep 14, 2026, Apify deleted the Phoenix eval runner from apify-mcp-server. Langfuse-based two-tier gates had already replaced it, since its custom loop could not represent built-in tool competition or Claude Code's prompting. The runner had not run in CI since then, and the Phoenix client's peer dependency on an old Anthropic SDK conflicted with the Claude Agent SDK.
Public engineering write-up (GitHub pull request)
No team metrics reported
Deployment details
Evals for Apify's MCP server, which coding agents call as a tool.
Public engineering write-up (GitHub pull request)
No team metrics reported
Team
Apify MCP server maintainers
Workload
Tool-selection evals for an MCP server
Product version
@arizeai/phoenix-client eval runner

Switched to or from it1

Jérémie Doucy (jdoucy)Contributor to the Bayes Impact platformBayes Impact
Bayes Impact's nonprofit platform, deployed with a Helm chart that bundles Phoenix.
GitHub, bayesimpact/bayes-platform, published Sep 29, 2026Read the write-up (opens GitHub, bayesimpact/bayes-platform in a new tab)
Observation
Bayes Impact merged a switch of its platform's LLM tracing from Langfuse to Arize Phoenix, exporting plain OpenTelemetry through a collector.
Takeaway
Emit plain OpenTelemetry and let a collector map it, so the trace backend becomes a setting. Harden a bundled Phoenix with sign-in and PostgreSQL before production.
What happened
A pull request merged Sep 29, 2026 deleted a 673-line Langfuse exporter and the Langfuse packages. The platform now emits standard OTLP, and an OpenTelemetry Collector maps the spans to OpenInference for Phoenix. The plan keeps Langfuse up, read-only, for 30 days, since Phoenix deletes traces after 30 days. The pull request notes that the Phoenix bundled in its Helm chart is not production ready, with no authentication and SQLite by default, and that the Cloud Run production environment has no gateway and exports nothing with this version.
“…the platform emits standard OTLP and knows no trace backend. The OpenTelemetry Collector does the mapping to OpenInference and sends the LLM traces to Phoenix.”
jdoucy
- Public engineering write-up (GitHub pull request)
- Team metrics reported
Deployment details
Bayes Impact's nonprofit platform, deployed with a Helm chart that bundles Phoenix.
- Public engineering write-up (GitHub pull request)
- Team metrics reported
Team
Bayes Impact platform engineers
Duration
Planned: Langfuse kept read-only for 30 days after the Sep 29, 2026 merge
Workload
LLM traces from the Bayes Impact platform
Product version
Phoenix bundled in the platform's Helm chart, fed by an OpenTelemetry Collector
Measurement
673 lines of Langfuse exporter removed

Evaluated it1

Antaripa Saha and Hamel HusainEvaluation practitioners and co-authorsParlance Labs
An evaluation of three platforms' built-in agents on 100 traces from a production apartment-leasing assistant, labelled by hand with a domain expert and then masked.
Parlance Labs blog, published Jul 11, 2026Read the write-up (opens Parlance Labs blog in a new tab)
Observation
Parlance Labs evaluated Arize's Alyx against Braintrust Loop and LangSmith on 100 hand-labelled production traces. Alyx missed more failures than Loop, and 91% of its flags were valid.
Takeaway
Pick Alyx when a flagged failure has to be right, and keep a person reviewing traces for failures that need business context.
What happened
Parlance Labs masked its hand labels and asked Braintrust Loop, Arize Alyx and LangSmith to find the failures on their own (write-up of Jul 11, 2026). Braintrust had the highest recall. Alyx missed more of the labelled failures than Loop, but 91% of its flags were valid and only three were wrong. Alyx first read only the first 200 characters of each trace, and later runs hit compaction and timeout problems, which Arize fixed quickly once reported. Every system missed failures that needed business context, and general coding agents given the same prompt were competitive with the platforms.
“Arize traded recall for precision. Alyx missed more of the human-labeled failures than Loop, but 91 percent of its flags were valid and only three were wrong.”
Antaripa Saha and Hamel Husain
- Public practitioner write-up
- Team metrics reported
Deployment details
An evaluation of three platforms' built-in agents on 100 traces from a production apartment-leasing assistant, labelled by hand with a domain expert and then masked.
- Public practitioner write-up
- Team metrics reported
Team
Two Parlance Labs authors and a domain expert
Workload
Finding the failures in 100 masked production traces
Product version
Arize Alyx, Braintrust Loop, and LangSmith Engine and chat agent
Measurement
Recall against the hand labels; 91% of Alyx's flags valid, three wrong

Adopted it5

PagerDutyAug 13, 2025PagerDuty engineering blogPagerDuty's applied AI team runs each agent's golden questions as Arize experiments and pages the agent team when an output-quality monitor breaches.
Everaldo AguiarSenior Engineering Manager, Applied AIPagerDuty
PagerDuty Advance agents: Insights, Shift, and SRE. These agents are in early access. Several teams changing agents at once.
PagerDuty engineering blog, published Aug 13, 2025Read the write-up (opens PagerDuty engineering blog in a new tab)
Observation
PagerDuty's applied AI team runs each agent's golden questions as Arize experiments and pages the agent team when an output-quality monitor breaches.
Takeaway
Run golden datasets as regression tests in Arize. Score production traces with checks that need no ground truth. Route quality breaches through your outage process.
What happened
PagerDuty first tested its Insights Agent by hand against known answers. It automated those checks with an LLM judge and shared Arize tracing and experiments, storing golden questions as Arize datasets. For production without ground truth, evaluator templates score relevance and groundedness, plus tool selection. Product managers read those dashboards too. A monitor breach opens a PagerDuty incident that pages the agent team at low urgency. It also files a Jira ticket.
“In this article, we will discuss how we partnered with Arize to roll out the observability tooling which gives us visibility into our AI Agents, throughout their development lifecycle.”
Everaldo Aguiar
Public engineering write-up (co-written with Arize)
No team metrics reported
Deployment details
PagerDuty Advance agents: Insights, Shift, and SRE. These agents are in early access. Several teams changing agents at once.
Public engineering write-up (co-written with Arize)
No team metrics reported
Team
Data Science Core and Machine Learning Engineering teams, with each agent team
Duration
Agentic work begun a few months before August 2025
Workload
Insights, Shift, and SRE agents in PagerDuty Advance
Product version
Arize AX tracing, datasets, experiments, custom metrics, dashboards, and monitors
Measurement
Golden dataset pass rates; relevance, groundedness, and tool selection scores
WayfairJan 15, 2026Wayfair tech blogWayfair's Wilma agent clears about 65% of supplier-transfer tickets without a person, watched through associate feedback, offline evals, and Arize traces and alerts.
Farshid BahramiMachine Learning Scientist IIIWayfair
Supplier Transfer Program. LangGraph agent that decides in about a second whether an associate must step in. Five-person author team.
Wayfair tech blog, published Jan 15, 2026Read the write-up (opens Wayfair tech blog in a new tab)
Observation
Wayfair's Wilma agent clears about 65% of supplier-transfer tickets without a person, watched through associate feedback, offline evals, and Arize traces and alerts.
Takeaway
Limit each prompt to applicable cases. Have a person validate every flag and use Arize traces and alerts to monitor results.
What happened
Wilma reads supplier, customer, or associate comments and flags tickets needing a Wayfair associate, with a reason code. One prompt reached 70% accuracy but invented reasons that did not apply to the author. Three author-specific prompts replaced it. The write-up reports 95% precision and 97% recall, and a 29% reduction in time to resolution. An associate validates every flag. Arize supports trace-level investigation and alerts on metric shifts.
“Beyond human feedback, Wilma’s performance is continuously monitored through offline evaluation, quality assurance team reviews and observability tooling in Arize. Arize enables trace-level visibility into flagging behavior, supports rapid debugging and provides automated alerts for anomalous shifts in key metrics…”
Farshid Bahrami
Public engineering write-up
Team metrics reported
Deployment details
Supplier Transfer Program. LangGraph agent that decides in about a second whether an associate must step in. Five-person author team.
Public engineering write-up
Team metrics reported
Team
Five Wayfair authors, including machine learning scientists
Workload
Real-time intervention flags on supplier-transfer tickets
Product version
LangGraph workflow with Arize tracing, debugging, and alerts
Measurement
95% precision, 97% recall, about 65% of tickets cleared, 29% lower time to resolution
UberAug 14, 2026Arize AI blogUber's agent platform provisions Arize tracing when an agent deploys, because teams had added observability only after agents reached production.
Aayush AgrawalSenior AI product managerUber
Uber's internal agent platform. Teams from operations to consumer products. Arize:Observe 2026 talk.
Arize AI blog, published Aug 14, 2026Read the write-up (opens Arize AI blog in a new tab)
Observation
Uber's agent platform provisions Arize tracing when an agent deploys, because teams had added observability only after agents reached production.
Takeaway
Provision tracing when an agent deploys, and track turns per session and other behavior metrics that offline tests miss.
What happened
A ride-booking voice agent rerouted a trip toward a restaurant after hearing a child ask for pizza. Offline evals missed that failure, and a conversational designer found it in production as sessions grew from four or five turns to as many as 20. Teams often added fuller observability only after an agent was in production. Agents deployed through Uber's managed agent system now get Arize tracing, with the Arize environment and access set up at deployment, and code-first teams turn it on with a small configuration change. Uber generates evaluators from agent configurations and traces and sends alerts to Slack. One team found its agent contradicted tool outputs in about 30% of evaluated cases.
Vendor customer story (Arize write-up of an Arize:Observe 2026 talk)
Team metrics reported
Deployment details
Uber's internal agent platform. Teams from operations to consumer products. Arize:Observe 2026 talk.
Vendor customer story (Arize write-up of an Arize:Observe 2026 talk)
Team metrics reported
Team
Uber agent platform team serving operations, infrastructure, and consumer product teams
Duration
More than a year of platform work
Workload
Production agents, including a multi-turn ride-booking voice agent
Product version
Arize tracing provisioned at deployment for agents on Uber's managed system, with generated evaluators and Slack alerts
Measurement
Turns per session, from four or five to as many as 20; tool-output contradictions in about 30% of evaluated cases
HandshakeAug 17, 2026Arize AI customer storyHandshake built Arize AX and Datadog into its internal LLM Orca service, so every model call its feature teams make is traced for evaluation and monitoring.
Kyle GallatinTechnical Lead Manager of ML InfrastructureHandshake
LLM Orca, Handshake's internal service that feature teams build LLM features on.
Arize AI customer story, published Aug 17, 2026Read the write-up (opens Arize AI customer story in a new tab)
Observation
Handshake built Arize AX and Datadog into its internal LLM Orca service, so every model call its feature teams make is traced for evaluation and monitoring.
Takeaway
Build tracing into the shared service teams call, so every new use case arrives instrumented on day one.
What happened
Handshake shipped more than 15 LLM use cases in under six months on LLM Orca, according to Arize's customer story of Aug 17, 2026. The service has Arize AX and Datadog integrations built in, so every call is traced. Feature teams evaluate prompts, break down cost, automate LLM-as-a-judge workflows and export production traces from Arize AX.
Vendor customer story (Arize customer page)
Team metrics reported
Deployment details
LLM Orca, Handshake's internal service that feature teams build LLM features on.
Vendor customer story (Arize customer page)
Team metrics reported
Team
Handshake ML infrastructure and its feature teams
Duration
Under six months for more than 15 use cases
Workload
LLM features built on the LLM Orca service
Product version
Arize AX, with Datadog integrated in the same service
Measurement
More than 15 LLM use cases in under six months
AXAFeb 16, 2025AI EngineerAXA Germany's data team adopted open-source Phoenix when it could not see how the LLM calls inside its DSPy pipeline fit together.
Jeronim MorinaAI and MLOps EngineerAXA
AXA Germany's data team, building a DSPy pipeline on AXA's internal Secure GPT platform.
AI Engineer, published Feb 16, 2025Watch the talk (opens AI Engineer in a new tab)
Observation
AXA Germany's data team adopted open-source Phoenix when it could not see how the LLM calls inside its DSPy pipeline fit together.
Takeaway
Trace a framework's calls before you tune it, because its abstractions hide which prompts actually run.
What happened
The team logged no traces at first and could not see how its LLM calls related. Adding open-source Phoenix showed which calls were bundled together and what ran behind DSPy's abstractions (AI Engineer World's Fair 2024 talk, video published Feb 16, 2025).
Conference talk
No team metrics reported
Deployment details
AXA Germany's data team, building a DSPy pipeline on AXA's internal Secure GPT platform.
Conference talk
No team metrics reported
Team
AXA Germany data team
Workload
DSPy pipeline prototype for insurance tasks
Product version
Open-source Phoenix

Left it or turned it down4

bbv's AI Hub team dropped Arize Phoenix for Langfuse in February 2026, citing Phoenix's experiment complexity and limited cost tracking.

NonBioS ruled out Arize Phoenix after a brief look at its naming and website, and chose self-hosted Langfuse instead.

Elastic dropped Arize Phoenix from Kibana's evaluation framework once an executor inside Kibana stored datasets in Elasticsearch.

Apify dropped its Phoenix eval runner after Langfuse-based gates replaced it, because the runner's verdicts did not transfer to how Claude Code prompts.

Switched to or from it1

Bayes Impact merged a switch of its platform's LLM tracing from Langfuse to Arize Phoenix, exporting plain OpenTelemetry through a collector.

Evaluated it1

Parlance Labs evaluated Arize's Alyx against Braintrust Loop and LangSmith on 100 hand-labelled production traces. Alyx missed more failures than Loop, and 91% of its flags were valid.

Adopted it5

PagerDuty's applied AI team runs each agent's golden questions as Arize experiments and pages the agent team when an output-quality monitor breaches.

Wayfair's Wilma agent clears about 65% of supplier-transfer tickets without a person, watched through associate feedback, offline evals, and Arize traces and alerts.

Uber's agent platform provisions Arize tracing when an agent deploys, because teams had added observability only after agents reached production.

Handshake built Arize AX and Datadog into its internal LLM Orca service, so every model call its feature teams make is traced for evaluation and monitoring.

AXA Germany's data team adopted open-source Phoenix when it could not see how the LLM calls inside its DSPy pipeline fit together.