01
Short answer
Production readiness includes model quality and the operating conditions that a demo may not test. Production means load, bad inputs, cost discipline, failure recovery, auditability, and a team that owns it after launch week. Missing ownership, evaluation gates, or a budget can block launch, alongside technical failures. The teams that ship treat graduation as a checklist with named owners and dates, not as a vibe that the system seems ready.
02
Why pilots die in the graduation gap
A pilot runs on a developer's laptop, on twenty friendly test cases, with the team that built it watching every output. Production runs at 3 a.m. on traffic nobody predicted, while the original developer is on another project. The failure modes are entirely different. Test accuracy and operational failure modes before launch: concurrency limits, prompt injection, long-session context handling, and cost under the expected traffic mix. A successful demo does not establish production readiness. Require evaluation results, monitoring, and a tested rollback plan before release.
03
The graduation checklist
Write down what production means before you build toward it, because "seems ready" is not a standard. The categories: quality, with an eval dataset and thresholds that the current system provably meets; reliability, including load testing at expected peak, retry behavior, and degradation paths when providers or models have incidents; cost, with a modeled per-user-action cost and a budget owner; security, covering prompt injection, data leakage, and access controls on the retrieval corpus; observability, with traces and dashboards from the monitoring post earlier in this series; and operability, meaning rollback, versioning, and a named on-call. Any item without an owner and a date is a wish, not a plan. Record the acceptance threshold, test result, owner, and unresolved risk for each release gate. Specify what happens when a gate fails.
Sequence the work by risk, not by ease. Cost modeling and load testing are boring and find problems that are painful to fix late, so teams defer them. Defer them and you will discover the problems in week one of production, when fixing them is an incident instead of a task. Define evaluation criteria and monitoring early so that subsequent changes can be measured. Develop security, reliability, and cost controls alongside them.
04
Load and failure testing are different for LLM systems
Load testing covers your serving path, can you handle the request rate, and what happens to latency under it. Failure testing also covers model and provider behavior, including rate limits, output failures, and unavailable dependencies. Provider outages happen, and your system needs a defined behavior: fail closed with a graceful message, fail over to a secondary provider, or degrade to cached and templated responses. Test rate limits deliberately. Providers throttle, and unbounded retries can amplify load and prolong an incident. Test bounded retries, backoff, request deadlines, and a defined fallback. Exercise the long-session case, because context window exhaustion on hour-long conversations is a failure mode no short demo session will ever surface.
05
Plan the organizational handoff
An ownership gap can undermine a technically successful launch. The builders moved to the next project, support does not understand it, and the first quality regression goes unnoticed for weeks. Fix this before launch by naming an owner, one person accountable for quality, cost, and incidents, and giving them the eval, monitoring, and rollback tools to actually operate it. Write the one-page runbook: what this does, what normal looks like, what to check when users complain, when to roll back. If the team that built it cannot write that page, they do not understand the system well enough to hand it off, and launching anyway transfers the risk to whoever answers the support queue first.
06
Keep the pilot mindset where it belongs
One thing pilots do well is iteration speed, and you lose it at your peril. Production discipline and fast iteration are not opposites if the machinery is right, small eval datasets, fast regression gates, flag-based rollouts, and configuration-driven prompts exist precisely so that iteration continues safely after launch. Production release should preserve the ability to make controlled changes. Use production failures and changing requirements to decide which improvements to release next.
07
Turn the release checklist into test results
Consider a knowledge assistant moving from a staff pilot to a customer-facing workflow. This is an illustrative scenario. The pilot establishes that the assistant can answer selected questions. The release decision also needs to establish which customers may see which documents, how the application responds when a source is missing, and who handles a failure after launch.
For each requirement, record a testable threshold and the result against it. Quality thresholds should cover supported answers, important failure categories, and appropriate abstention. Reliability criteria should describe the expected workload and acceptable degradation. Cost criteria should include the calls and supporting work needed to complete a user task. Avoid a single readiness score that lets strength in one area compensate for an unresolved access-control defect.
Use production-like boundaries in the test environment. Check authentication, document permissions, configuration loading, and the paths that call external services. Confirm that the logs contain enough permitted information to investigate a complaint. Remove or mask sensitive content before export, and test who can access the retained records.
08
Exercise the fallback as part of the workflow
A fallback needs an owner and a defined user outcome. If the provider is unavailable, the application may hold the request, offer a limited response, or route it to a person. State which tasks remain available and which actions are blocked. A fallback that returns plausible but unverified content can convert an availability incident into an answer-quality incident.
Test a slow dependency as well as a failed one. Confirm that request deadlines, bounded retries, and cancellation work together. Check whether abandoned requests continue consuming resources. Under load, repeated retries can increase the work arriving at an already constrained service. Record the traffic and failure conditions used in the test so the team knows its limits.
A secondary provider also needs evaluation. Confirm that it supports the required tools, output format, access boundaries, and failure behavior. Passing requests to another model changes the system; it does not automatically preserve the quality accepted for the primary path. Define when failover is allowed and when the application must stop.
09
Limit the first production release
Start with a scope the operating team can observe and support. Define the included user group, workflows, traffic limits, and release duration. Check the important quality and operational signals before expanding. Where practical, compare the new configuration with the existing path on the same task categories.
The rollback plan should name the trigger, decision owner, previous configuration, and verification step. Rehearse it. Confirm that the previous source versions and settings are available and that restoring them does not expose obsolete or unauthorized data.
10
Make the handoff observable
Ask the receiving operator to investigate a representative failure using the runbook and retained records. Have them locate the active model and prompt versions, identify the affected workflow, and select the permitted containment action. This checks whether the handoff materials support the work they must perform.
Close launch tasks with recorded results and owners for any remaining risk. Schedule a review after the first production period to compare actual traffic, cost, review load, and failures with the assumptions used for release. The purpose is to decide what needs to change before the next expansion, while the release remains small enough for the team to understand its behavior.
11
Related
Where to read next on the same subject.