Langfuse pays off when every failure seen in a trace becomes a dataset item, and every later prompt or model change has to pass that dataset before it ships. This playbook covers instrumenting on the v4 SDKs or OpenTelemetry, masking sensitive data, scoring production, managing prompts, and the experiment gate your team agrees on.
When you need this
- An LLM app or agent is moving from a prototype to production, and nobody can say why a given run went wrong.
- Prompt or model changes ship on a hunch, and regressions surface as user complaints.
- An existing integration still uses Python SDK v2 or JS/TS SDK v3, which Langfuse Cloud rejects from November 16, 2026.
Evaluation criteria
Instrument on v4
How do traces reach Langfuse, and will they still arrive after November 16, 2026?
Look forUse Python SDK v4 or JS/TS SDK v5 with observe and framework integrations, or send OpenTelemetry spans over HTTP to /api/public/otel. Cloud removes the legacy ingestion and read APIs on November 16, 2026. Medic, Traigent and CAMEL spent September and October 2026 migrating their integrations. One team found that an old import had turned tracing off without any error.
Name traces so evaluators can use them
Will a dashboard, evaluator, or dataset find what it needs on each observation?
Look forSet user, session, tags, environment, and release on every observation. Python 4.7.0 and JS/TS 5.4.0 onward propagate those attributes. Check streaming endpoints. Under Python SDK v3, one developer's FastAPI streaming responses split into separate traces.
Keep sensitive data out
Which inputs and outputs must never leave the application?
Look forMask fields in the SDK before export, which works on every plan. Self-hosted Enterprise adds masking on the server at ingestion. If Langfuse shares an OpenTelemetry pipeline with Sentry or another tool, filter the spans sent to Langfuse. One developer's single bot question produced 278 spans until a collector filter cut 79% of them.
Turn failures into datasets
How does a bad run become a test case?
Look forAdd failed traces or observations to a dataset through the trace view or bulk actions. Dataset inputs and expected outputs survive after the trace expires. DKB turns flagged conversations into regression tests for each release.
Score production
What tells you quality dropped after release?
Look forRun observation-level LLM judges on filtered production traffic. Add code evaluators for deterministic checks, collect user feedback, and send hard cases to annotation queues for experts. Trace-level evaluators stop working at Cloud's v4 cutover. Judges call models through your LLM connection, and every score adds a billable unit.
Manage prompts outside the deploy
Who can change a production prompt, and how is it rolled back?
Look forFetch and cache labeled prompt versions through the SDK. Use the production label for live traffic and latest for development. A recruiting-agent team kept model, provider and temperature settings in the prompt config, so it could switch models without a deploy. Keep a fallback prompt in code and warm the cache so the app runs when Langfuse is down. Protected labels require Teams or Enterprise.
Gate changes on experiments
What must an experiment show before a prompt or model change ships?
Look forRun datasets through the SDK's experiment runner. Add langfuse/experiment-action to pull requests with a threshold for each score. Check LLM judges against human-annotated examples before you trust those thresholds.
Alert on thresholds
Who hears about a cost, latency, or quality spike first?
Look forSet thresholds for cost, latency and quality scores, with notifications to Slack, webhooks, or GitHub Actions. Hobby allows 2 alerts, Core 20, Pro 50, and Enterprise 100.
Questions to ask vendors
- Which services still send through Python SDK v2, JS/TS SDK v3, or the legacy ingestion API?
- Which fields are masked, and who signs off on the masking rules?
- Which evaluators gate a release, at what thresholds, and who checks the judges against human labels?
- How does a production failure get from a trace into a dataset, and who does it?
- What does the application do when Langfuse is slow or unreachable?
Red flags
- Integrations pinned to SDKs that Langfuse Cloud rejects after November 16, 2026.
- Trace-level LLM-as-a-judge evaluators with no observation-level replacement.
- One OpenTelemetry pipeline sending every server span to Langfuse.
- Prompts fetched at runtime with no fallback and no cache warm-up.
- LLM-as-a-judge scores treated as ground truth with no human review.
Products in this piece
Sources
- Official documentation
- 16
- Public source
- 6
- Langfuse docs: Python and JS/TS SDKs
- Langfuse docs: Get started with tracing
- Langfuse docs: OpenTelemetry
- Langfuse changelog: Langfuse v4 is live (Aug 17, 2026)
- Langfuse docs: Masking
- Langfuse docs: Data masking (self-hosted)
- Langfuse docs: Datasets
- Langfuse docs: LLM-as-a-judge
- Langfuse docs: Code evaluators
- Langfuse docs: Annotation queues
- Langfuse docs: LLM connections
- Langfuse docs: Experiments in CI/CD
- Langfuse docs: Get started with prompt management
- Langfuse docs: Alerts
- Langfuse pricing
- Medic cht-agent pull request #162: migrate Langfuse to JS SDK v5 (Sep 23, 2026)
- Traigent pull request #2520: read traces via Observations API v2 before the Cloud v4 cutover (Oct 7, 2026)
- CAMEL pull request #4349: migrate the Langfuse integration to the v4 SDK (Sep 21, 2026)
- langfuse issue #8216: observe decorator splits FastAPI streaming traces (Jul 30, 2025)
- Hugo Lesta on dev.to: Tracing A2A agents without burning your free tier (Aug 14, 2026)
- r/LLMDevs: What I learned using Langfuse in a real AI recruiting agent (Jun 3, 2026)
- Langfuse customer story: How DKB uses Langfuse to keep its AI banking assistant accurate, compliant, and trustworthy at scale (Aug 31, 2026)