Skip to main content

Decision brief

Langfuse Review: Open-Source Tracing, Evals, and Prompt Management

Choose Langfuse as the open-source default for LLM tracing if someone will own ClickHouse or the unit bill, and plan for the v4 cutover.

Reading time
3 min read
Sources
21 cited

GILD verdict

Choose Langfuse when you want tracing, evaluation, and prompt management in one open-source platform, above all when trace data must stay in your own infrastructure.

Before traffic grows, finish the v4 migration, mask sensitive fields, and name the person who runs ClickHouse.

Langfuse

AI Observability by ClickHouse

+46 Positive
Product page

Fit and limits

Who it fits

  • LLM apps or agents are in production or close to it. When a run goes wrong, the team needs every model call, tool call, and agent step with its cost and latency.
  • Trace data must stay in your own network, or the tracer ships inside a product you sell: the MIT-licensed core self-hosts with all core features and no usage limit, including enterprise SSO. bbv's AI Hub left Arize Phoenix for that license in February 2026, and Langfuse says ClickHouse's purchase brings no planned licensing changes.
  • Prompts change often and people outside engineering edit them: labeled prompt versions are fetched and cached at runtime, so a change ships without a deploy.
  • The stack mixes frameworks and languages: native integrations cover LangChain, LangGraph, the OpenAI Agents SDK, Pydantic AI, the Vercel AI SDK, and gateways such as LiteLLM, and OpenTelemetry covers the rest.

Where it falls short

  • Self-hosting requires ClickHouse, PostgreSQL, Redis, and S3. Docker Compose has no high availability or backups. Practitioners report ClickHouse's memory and CPU use, plus disk growth, as the main burden: one operator measured tuned v4 at 1,848 MiB across four containers and made Phoenix the default on October 13, 2026.
  • Cloud bills every trace and observation, plus each score. Deep agent runs and eval scores increase charges. Unrelated OpenTelemetry spans count too.
  • Version changes require migrations. v3 moved storage to ClickHouse. On November 16, 2026, Cloud's v4 cutover removes legacy SDKs and APIs, plus trace-level evaluators. Self-hosters pick a backfill that needs about 3x ClickHouse disk headroom, or dual writes until retention rolls over.
  • Governance features sit on higher tiers: retention policies need Pro, project RBAC the US$300 Teams add-on, and audit logs Enterprise. Self-hosted, each needs an Enterprise key, now bundled with ClickHouse Cloud, BYOC, or Private and priced on top of that plan.

Implementation reality

  1. Instrument on v4 from the start

    Use Python SDK v4 or JS/TS SDK v5, or send OpenTelemetry spans over HTTP to /api/public/otel. Langfuse does not support gRPC. Python 4.7.0 and JS/TS 5.4.0 onward propagate trace attributes, including user and session, to every observation as v4 expects.

  2. Mask before data leaves

    Configure SDK masking before exporting sensitive traces. Self-hosted Enterprise adds ingestion masking. Choose the Cloud region at sign-up. Moving between separate regions requires a new account and data migration.

  3. Control volume and retention

    Count a week of units and filter non-LLM spans. Set LANGFUSE_SAMPLE_RATE for high-volume paths and spend alerts on Core or above. Configure project retention on Pro or self-hosted Enterprise. Without a policy, self-hosted data stays indefinitely.

  4. Gate changes on experiments

    Add failures to datasets, whose items outlive source traces. Run experiments with LLM judges or code evaluators, including decision-model evaluators. langfuse/experiment-action fails a pull request when a score crosses its threshold. Use annotation queues to check judges against human labels.

What to measure in your pilot

  • GILD found no public price for self-hosted Enterprise with its ClickHouse plan; get both quotes, and if the bundle costs more than Cloud Pro at your unit volume, use Cloud.
  • Rehearse the v3 to v4 upgrade on a staging copy, timing the backfill and measuring ClickHouse disk, and finish production before v3 security patches end in January 2027.
  • Compare LLM-judge scores with annotation-queue labels on 100 traces per use case; Zheng et al. measured over 80% agreement for GPT-4 judges, so retune any judge below that.

Sources

Official documentation
20
Public source
1
  1. Langfuse pricingOfficial documentationlangfuse.com
  2. Langfuse self-hosted pricingOfficial documentationlangfuse.com
  3. Langfuse docs: Billable unitsOfficial documentationlangfuse.com
  4. Langfuse docs: Self-host LangfuseOfficial documentationlangfuse.com
  5. Langfuse docs: Docker Compose deploymentOfficial documentationlangfuse.com
  6. Langfuse docs: Enterprise license key (self-hosted)Official documentationlangfuse.com
  7. Langfuse changelog: Langfuse v4 is live (Aug 17, 2026)Official documentationlangfuse.com
  8. Langfuse docs: Migrate Langfuse v3 to v4 (self-hosted)Official documentationlangfuse.com
  9. Langfuse docs: Python and JS/TS SDKsOfficial documentationlangfuse.com
  10. Langfuse docs: OpenTelemetryOfficial documentationlangfuse.com
  11. Langfuse docs: MaskingOfficial documentationlangfuse.com
  12. Langfuse docs: Data regions and availabilityOfficial documentationlangfuse.com
  13. Langfuse docs: Data retentionOfficial documentationlangfuse.com
  14. Langfuse docs: SamplingOfficial documentationlangfuse.com
  15. Langfuse docs: Spend alertsOfficial documentationlangfuse.com
  16. Langfuse docs: Evaluation overviewOfficial documentationlangfuse.com
  17. Langfuse docs: Experiments in CI/CDOfficial documentationlangfuse.com
  18. Langfuse docs: Prompt managementOfficial documentationlangfuse.com
  19. Langfuse blog: Langfuse joins ClickHouse (Jan 16, 2026)Official documentationlangfuse.com
  20. Langfuse docs: Annotation queuesOfficial documentationlangfuse.com
  21. arXiv: Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (Dec 24, 2023)Public sourcearxiv.org