01

Short answer

LLM observability connects one request across retrieval, generation, validation, tools, cost, and user outcome. Langfuse can hold those traces and evaluations, but the team must define fields, data access, sampling, retention, and review routines.

An LLM application fails differently from other software. The process does not crash; it answers, confidently, from stale context or a misfiring tool, and the user does not file a ticket because nothing visibly broke. A week later someone asks why the assistant gave the wrong policy answer on Tuesday, and the team starts digging through logs that were never designed to answer the question.

Observability for these systems means traces that reconstruct the whole request, evaluations that run against live traffic, and cost accounting precise enough to price a decision. In our deployments the tool that ties these together is Langfuse, open source and self-hostable, which matters for customers with data-residency commitments. This guide is about what we instrument and why; the tool is the vehicle, not the point.

02

The trace is the unit of understanding

A trace records one request end to end: the input, every retrieval call with its latency and results, every generation with its model, prompt version, token counts, tool invocations in between, and the final output. Structured as a tree, so the rewrite call that happened after the validator flagged a schema violation is visible as the sibling it actually was.

The discipline that makes traces useful is recording the things you will be blamed for not knowing: which prompt version ran, which index snapshot was retrieved against, what the guardrails decided and why. When Tuesday's wrong answer needs explaining, the trace either answers it in two minutes or the investigation starts from nothing.

In Langfuse the primitive is the trace with nested spans and generations; the SDKs make the plumbing modest, and the self-hosted option keeps the data inside the customer's boundary.

03

Evaluations, both directions

Observability without evaluation is a museum of failures. The pairing that works runs in two directions.

Offline, the evaluation set, the versioned questions and judged answers from our AI evaluation guide, scores every candidate prompt and model change before release. Langfuse holds the dataset and the run history, so a prompt tweak shows its effect against the same set as last time, and regressions are diffs rather than debates.

Online, samples of live traffic get judged on a schedule, by a stronger model where the judgment is mechanical, by a person where it is not. This is the only evaluation that cannot drift away from reality, because it is reality. The failure modes it catches, questions users actually ask that no one imagined in the offline set, feed back into the offline dataset. The loop is the system.

04

Cost as a first-class trace dimension

Token counts recorded per generation turn into cost per trace, cost per user, cost per decision type. This sounds like accounting and behaves like architecture: the moment you can see that 3 percent of conversations consume 40 percent of the tokens, usually long-context retries or a verbose tool loop, the optimization writes itself.

The cost dimension also disciplines model routing. When the frontier model answers the hard 10 percent and a small model handles the rest, the trace shows the split actually delivered, and the escalation rate becomes a managed number rather than a hope. Our inference cost guide covers the routing economics; observability is what makes it real rather than aspirational.

05

Prompts as versioned artifacts

The prompt is the most-edited file in an LLM system and, in too many teams, the least-managed: changed in the codebase, deployed on merge, debugged from memory. Langfuse prompt management stores prompts as versioned artifacts the application fetches at runtime, so every trace records the exact version that ran, and a rollback is a version switch rather than a revert and redeploy.

Versioning also enables the comparison that matters: run two prompt versions against the same evaluation set, read the score difference, and promote or discard based on the scores. That is the release gate for the layer of the system that changes most often.

06

The weekly review that keeps it honest

The instrumentation earns its keep in a ritual, not a dashboard. Once a week, the team reviews a sample of traces: the highest-cost conversations, the ones where guardrails fired, the ones users marked wrong. Findings become evaluation cases, backlog items, or guardrail tuning. The review takes an hour and it is the difference between a system that improves in production and one that merely runs there.

07

What should the first trace contain?

Start with fields an operator will use during a failed request. Give the trace a request ID, time, environment, application version, user or tenant reference, workflow type, and final status. Record identifiers rather than raw personal data where possible, and keep the lookup behind the same access controls as the application.

Each span needs a name that describes work, such as retrieve-policy-documents or validate-refund-request, plus its start time, duration, status, input size, and output size. Generation spans add provider, model, prompt version, token counts, finish reason, and retry number. Retrieval spans add index version, filters, candidate count, returned document IDs, and scores. Tool spans add the acting identity, requested operation, authorization decision, and result.

Do not capture every payload by default. Prompts, retrieved passages, and tool parameters may contain customer or employee data. Store redacted samples only where diagnosis requires them, separate sensitive payload access from ordinary trace access, and set retention by risk. Hashing is not anonymization when the original value comes from a small known set.

Propagation must survive asynchronous work. Carry the trace ID through queues, scheduled jobs, callbacks, and approval steps so one user action remains connected even when execution takes hours. If a downstream system cannot accept the context, log the parent request ID at both boundaries.

08

Define the questions before adding dashboards

An operating view should answer a small set of questions:

Build charts from those questions. A panel full of total tokens and average latency can look busy while hiding the one workflow that doubled its failure rate. Slice by workflow, model version, customer group, language, and final status before adding more aggregate metrics.

Use service-level measures tied to the user path. Time to first useful response may matter more than total trace duration for streaming answers. Completed approved action matters more than tool-call latency for an agent. A declined answer with a useful route can be a success, while a fast unsupported answer is a failure.

  • Which workflow is failing, for which users, and since which release?
  • Where is time spent across retrieval, generation, validation, tools, and queues?
  • Which model and prompt version produced the response?
  • What did the request cost, including retries and fallbacks?
  • Which guardrail or authorization rule changed the path?
  • Can an operator find similar failures and replay an approved case?

09

Connect traces to evaluation

Create an evaluation case from the trace rather than copying text into a separate spreadsheet. Preserve the sanitized input, expected behavior, relevant source IDs, policy version, and reviewer decision. Link the case back to the production request so engineers can see the original failure without exposing it to every evaluator.

Run deterministic checks first: schema, citations present, tool parameters allowed, expected refusal, and latency or cost bounds. Use human review for correctness and usefulness where rules cannot decide. Model-based scoring can prioritize review, but calibrate it against human judgments and keep disagreement visible.

Every release runs the fixed regression set. Production sampling supplies new cases from user reports, guardrail events, expensive traces, low retrieval scores, and random traffic. Keep the fixed and changing sets separate so teams can see both regression and current-world performance.

10

Operate Langfuse without creating a second incident

Choose hosted or self-managed deployment from data residency, access, maintenance, and recovery requirements. In either case, separate production and test data, restrict project access by role, rotate ingestion credentials, and prevent client-side code from holding privileged keys.

Set retention and sampling deliberately. Keep all errors and guardrail events, sample ordinary success traffic at a known rate, and preserve enough successful traces to estimate quality. Document the rate in dashboards so a count is never mistaken for total traffic.

Monitor the observability path itself: dropped spans, export delay, queue depth, storage growth, failed evaluations, and prompt-fetch errors. Application requests should follow an explicit policy when tracing is unavailable. Low-risk advisory responses may continue with an alert; acting workflows may need to stop if their audit record cannot be written.

Test export and restore before treating the service as operational. Prompt versions, scores, datasets, and trace metadata may become release dependencies. The team must know how the application behaves during an outage and how investigation continues if the primary interface is unavailable.

11

Related

Where to read next on the same subject.