01
Short answer
Monitor the three layers separately: the serving layer (latency, error rates, token throughput, cost per request), the behavior layer (output quality scores, refusal rates, escalation rates), and the input layer (prompt distribution, retrieval hit rates, out-of-scope share). Traditional uptime monitoring only covers the first. Behavioral failures can occur even when serving metrics are healthy, which means the metrics that catch them look nothing like the metrics your existing dashboards were built for, and the signals live in the content of requests and responses rather than in status codes.
02
Why uptime monitoring misses LLM failures
HTTP 500 errors and rising 99th-percentile latency are common service failure signals. An LLM application can return 200 OK in 800 milliseconds on every request and be badly broken: silently hallucinating, drifting out of policy, or answering questions it should refuse. Every request succeeds at the transport layer and fails at the task layer. This is why teams get blindsided. Their alerts are green, their traces are clean, and customers are tweeting screenshots. Monitoring has to extend one level deeper, into what the model actually said.
03
The serving layer: instrument what you already know
Latency, error rate, tokens per request, cost per request, retry rate. This layer is standard observability practice, and everything you use elsewhere applies. Two additions matter for LLMs. Track tokens per request as a first-class metric, not just a billing artifact, because it can help reveal growth in the request context, a slow upward trend means conversation history or retrieval settings are growing. Track retry rate separately from error rate, because retries can increase both latency and cost even when the final request succeeds. Measure the retry share and the cost of repeated calls.
04
The behavior layer: sample, score, trend
Choose evaluation coverage based on workload volume, risk, latency, and budget. Combine a representative traffic sample with targeted checks for rare, high-risk cases. Set sample sizes from the error rate you need to detect and the uncertainty you can accept. Use the evaluation rubric from your pre-launch tests, and compare model-judge scores with human labels. Then trend the scores, and watch the trend line more than any single day's number. Track gradual drift and sudden regressions. Alert when a meaningful trend crosses its agreed threshold, or when a severe failure requires immediate action. Compare like-for-like traffic segments before attributing a change to the model.
Watch three behavioral ratios alongside the scores. Refusal rate: a spike means either your users turned adversarial or your prompt or model changed. Escalation rate, if you have a human fallback: rising escalations mean the automation is losing ground. Self-reported failure rate from user feedback, thumbs down, "this was wrong," reopen tickets. These signals complement sampled evaluation. Measure how they relate to confirmed failures in your workload; neither strong correlation nor an earlier warning is guaranteed.
05
The input layer: your traffic is a metric
The distribution of incoming prompts is itself a signal, and it deserves its own dashboard. Track the share of prompts that are out of scope, off-language, adversarial-looking, or hitting the edge of your retrieval corpus. A rising out-of-scope share usually means one of two things: users are discovering a use case you did not design for, or something is routing wrong. Either way it is actionable. Track retrieval quality too, zero-hit queries, low similarity scores, because weak retrieval can leave the model without the information it needs. Similarity scores depend on the retrieval method; test their relationship to answer quality before setting alert thresholds.
06
Tie everything to traces
Use a request identifier to connect model and prompt versions, retrieval results, output checks, latency, and cost. Record content only where it is necessary and permitted. Mask sensitive fields before export, restrict access, and define retention and deletion rules. Document sampling so investigators know which requests have detailed traces. When a metric moves, the trace is how you find out why. A drop concentrated in long-context requests is a reason to inspect retrieval, truncation, and context handling. A broad drop is a reason to inspect shared model, prompt, infrastructure, and traffic changes. These patterns guide diagnosis; they do not establish the cause. Without traces, you are correlating aggregate numbers and guessing. Traces help the team reproduce the failure and test the suspected cause.
07
What to alert on, and what to only record
Alert on things a human should act on: serving-layer error spikes, cost per request jumping beyond a threshold, refusal or escalation rate spiking, quality trend crossing a floor. Record but do not alert on the rest. Alert fatigue kills monitoring programs faster than any technical gap, and LLM systems generate a lot of interesting-but-not-urgent numbers. Keep the on-call page short. Everything else goes into the weekly review.
08
Turn a complaint into an investigation
Consider a support assistant that starts giving the wrong return window. This is an illustrative failure scenario. Its availability and latency remain within their normal ranges. The useful question is whether the failure follows a particular policy version, product, language, or conversation length. A dashboard showing only average answer quality will not distinguish these possibilities.
Start with the affected request identifiers. Compare the prompt version, model version, retrieved document identifiers, document dates, and output checks with a successful request for the same question. If the wrong answer cites an obsolete policy, test the document filter. If the current policy reached the model but the answer changed its meaning, investigate generation and the evaluation rubric. If a permission check failed, contain the access problem before examining answer quality.
Keep the investigation reproducible. Store the configuration identifiers needed to replay an allowed, sanitized case. Record which information was removed and whether that removal affects the result. A trace that contains a score without the rubric version is difficult to interpret after the evaluation method changes.
09
Define denominators and review coverage
A refusal rate needs a denominator: all requests, requests within scope, or requests where an answer was expected. These describe different behaviors. An increase in appropriate refusals after an adversarial traffic spike can indicate that a control is working. An increase among ordinary support questions may indicate a regression. Record both the population and the classification rule beside each metric.
The same care applies to cost. Separate provider usage from the cost of retrieval, evaluation, retries, and human handling where those costs affect the decision. Measure cost per completed user task as well as cost per model call. A cheaper call that needs several retries or leaves the user unable to finish can increase the total task cost.
For sampled evaluation, record the selection method and the number of cases in each important segment. Compare the same segments over time. A change in the traffic mix can move the overall score while every segment remains stable. Maintain targeted tests for important failures that ordinary random sampling may rarely encounter.
10
Give each alert an action
An alert should identify the affected workflow, the observed change, the relevant release, and the person responsible for the response. Define an immediate containment action, such as limiting a route, returning a safe fallback, or reverting a prompt release. The action must fit the failure; switching models is not a remedy for a retrieval permission defect.
Write down how the team decides that service has recovered. Confirm that serving metrics have returned to acceptable levels, then check the failed task types against the restored configuration. Continue sampling after the immediate incident closes. A provider recovery can restore availability while a local workaround continues to produce poor answers.
The weekly review should turn recurring failures into owned changes. Keep the request category, suspected cause, investigation result, and regression test together. Close the item when the fix passes the agreed test and the production signal supports recovery. This gives the operator a record of what changed and why the team considers the issue resolved.
11
Related
Where to read next on the same subject.