01
Related
Where to read next on the same subject.
02
Short answer
Test the answers your application is likely to get wrong, define what a safe answer must contain, and compare every proposed change with the version users have now. After launch, review a sample of real requests and turn new failures into test cases. Keep a human in the loop for judgments that rules or a model grader cannot make reliably.
03
Start with an answer someone could act on
Imagine a support assistant asked, “I bought this 45 days ago and it arrived damaged. Can I return it?” The current policy says ordinary returns are accepted for 30 days, while damaged goods go to a specialist regardless of age. A fluent answer that says “You have 60 days to return any item” sounds helpful and is wrong. An answer that simply refuses because 45 is greater than 30 also misses the exception.
That one question gives you three checks: did the assistant use the current policy, did it distinguish an ordinary return from damage, and did it give the customer the right next step? The checks describe behavior a reviewer can observe. “Sounds helpful” does not.
An evaluation set is a collection of such cases, each with an input, the source or policy version needed to answer it, an expected behavior, and a reason the failure matters. Include routine questions as well as edge cases. A set made only of disasters can make a useful system look worse than the traffic it actually receives; a set made only of happy paths hides the failures that hurt customers.
For each case, keep the question as the user actually phrased it. Do not tidy away a missing order number or a contradictory date just to make the example easier to grade. Record which information the assistant was allowed to see and whether the right outcome is an answer, a clarifying question, or a handoff. Those distinctions matter: a system can score well by refusing everything, yet still fail the work it was bought to do.
Give each case a source and an owner. A support lead can confirm the damaged-goods rule; an engineer can confirm which policy version the retriever served. If the two disagree, the case needs correction before it becomes a release gate. The evaluation set should expose disputes about the work, not conceal them inside a pass rate.
04
Write the rubric before scoring the model
For the damaged-item example, a compact rubric could ask:
Grounding: Does the answer state only what the supplied policy supports? Exception handling: Does it recognize that the damage rule applies after 30 days? Action: Does it send the customer to the specialist without promising a refund? Clarity: Can a customer understand the next step without reading the policy?
Mark an unsupported refund promise as a release blocker. Treat a minor wording issue as a lower-severity defect. The team should agree on those distinctions before it sees a candidate model’s score. Otherwise the acceptance bar tends to move toward the result already in hand.
Keep the rubric short enough that two people can apply it to the same answer. Have both review a sample. Where they disagree, fix the wording or the underlying policy interpretation before asking a model grader to reproduce their decisions.
Decide how a missing answer is scored. If the policy does not cover an unusual product, asking for the product type may be correct; inventing a return period is not. If the assistant lacks permission to see a customer's order, a safe handoff may be the only passing response. These are application decisions, so write them into the rubric rather than expecting a general-purpose judge to infer them.
05
Use the right check for each failure
Software can check whether a response is valid JSON, whether a required field exists, or whether a tool call names an allowed operation. It cannot establish that a plausible sentence is true merely because a citation is present. A citation-presence check tells you that the answer contains a reference; grounding review asks whether the referenced passage supports the claim.
A model grader can help triage larger samples against a written rubric. Compare its judgments with human-reviewed answers across the failure types that matter. Look at disagreements, especially confident passes on unsupported claims. Keep the grader prompt and model version with the evaluation record. If the grader changes, rerun a common sample before interpreting a score change as an improvement or regression in the application.
Use human review for high-impact cases, ambiguous policy, and grader disagreements. Detection of personal data is also a control to validate, not a free and infallible test. A missed identifier matters even when the rest of the answer is correct.
06
Compare a candidate with what is live
Run the same fixed cases against the current release and the proposed prompt, model, retrieval, or tool change. Compare results by failure type and severity, not just one average. A candidate that improves routine answers but starts promising refunds it cannot authorize should not pass because its overall score rose.
Record the case-set version, policy sources, application configuration, grader version, human decisions, latency, and cost. Keep a few cases out of day-to-day tuning and use them as a check for overfitting. The release decision is then inspectable: what improved, what regressed, and who accepted the remaining risk.
Use a side-by-side review when the score hides a trade-off. Show reviewers the same question and source beside the current and candidate answers, with the model names concealed where possible. Ask which answer they would let a customer act on, and why. A tie on the total score can still reveal that one version gives better explanations while the other makes fewer unsupported promises. Log that distinction instead of turning the whole comparison into a single winner.
Release thresholds should be tied to failure severity. A team may accept a few awkward phrases while refusing any new unsupported refund promise. Record the number of cases in each slice beside its pass rate; a perfect score on two rare examples is weaker evidence than a stable result on a well-sampled category. When the sample is small, review the individual answers rather than reading too much into a percentage.
07
Let production add the cases you missed
Before launch, test cases represent your best understanding of the work. After launch, sample real requests across user groups, languages, and outcomes. Include complaints, escalations, low-confidence answers, and a random share of ordinary successes. Otherwise the review queue will tell you only about visible failures.
Capture enough context to investigate a sampled answer: the release identifiers, retrieved source IDs and versions, validation results, and outcome. Keep sensitive prompt or customer content only where access, redaction, and retention rules allow it. Full-payload logging is not a prerequisite for useful evaluation.
When a live answer fails, write down the expected behavior and add a sanitized case to the regression set. Recheck the live sample separately. The fixed set tells you whether you broke something known; the live sample tells you whether the work itself has changed.
Set a review cadence and assign the person who can act on the findings. A weekly sample review might surface a new product category that the policy does not yet cover. The fix could be a policy clarification, a retrieval update, or a handoff rule. Label the cause before changing the prompt. Otherwise each unfamiliar case becomes another instruction in a growing prompt, while the underlying source problem remains.
Look at the denominator too. If only escalated conversations are reviewed, the team cannot estimate how often routine answers go wrong. If only completed conversations are reviewed, abandoned requests disappear. Sample across completed, escalated, and abandoned work, then compare the groups separately. The sampling plan should be simple enough to repeat and specific enough that next month's score means the same thing as this month's.
Pick one customer-facing decision, review 30 to 50 real or realistic failure cases alongside its decision owners, and agree on the failures that block release. Run that set against the current system before proposing a change. A focused first pass can be built in about a week when the source material and reviewers are available. The score is useful only when the team can point to the answers behind it.