01
Related
Where to read next on the same subject.
02
Short answer
Give each customer-facing AI workflow a release record that names the prompt, model, retrieval and tool settings, evaluation result, and rollback target. Test the combination that will actually run. Promote it deliberately, watch the first live traffic, and make the previous working combination available if the new one fails.
03
The incident starts with a simple question
A support lead asks why yesterday’s assistant told customers they had 60 days to return an item. The prompt editor shows several revisions. The provider model name is an alias. The policy index was refreshed overnight. A trace contains the answer but no prompt or source version. The team can see the mistake; it cannot tell which change caused it or which state to restore.
Versioning should tell the team which prompt, model, sources, tools, and settings produced the answer, so they can investigate the mistake and restore a known release.
04
Give the workflow one release record
For a policy-answer assistant, a release record might look like this:
``text workflow: support-policy-answer release: support-policy-answer-004 prompt: support/answer version 7 model: provider and exact model identifier used parameters: temperature, output limit, response format retrieval: policy collection and index or source version tools: allowed operations and authorization policy version evaluation: dataset version, result, reviewer, decision date rollback target: support-policy-answer-003 ``
If the provider offers a fixed model version, record it. If it serves a moving alias, record the resolved identifier when available and note that the provider can change behavior under the alias. The release record must also say who owns promotion and what traffic it reached. A timestamp alone is a poor substitute for that history.
Keep the release record close to the decision that approved it. Link the evaluation result to the cases and actual answers that were reviewed. Note any accepted failure and the person who accepted it. If a customer reports a bad answer two weeks later, the team needs to know whether that behavior was missed, accepted, or introduced after the release. A release name without this context is just a label.
Name the versions at the boundaries where they can change independently. The prompt registry can identify the prompt, while the application records the retrieval index and tool permissions. Record the combined workflow release in each trace so those separate identifiers can be joined later. That is particularly useful when a source refresh happens between two application deployments.
Store prompt text and configuration in a place your team can review and recover. That may be a repository, a prompt registry, or both. BluePi’s Langfuse observability guide describes runtime prompt versions and labels; a reviewed repository record can capture the intended release while the registry delivers the immutable prompt version. Whichever arrangement you choose, make the deployed version and the reviewed record agree. Do not let a production label move without updating the release history.
05
Test the combination, then expose it gradually
A prompt that works with one model can behave differently with another. Retrieval and tool changes can move the answer just as much as prompt wording. Run the candidate combination against the same cases as the current release. Inspect results by failure severity and user segment, along with latency and cost. Keep the actual responses available for review; one score cannot explain a policy error.
Add a case for the change that prompted the release. If the new prompt is meant to handle damaged goods after 30 days, test that exception and nearby ordinary returns. If a source document changed, test the boundary between old and new policy. Review the cases with the owner of the underlying policy, not only the engineer who wrote the prompt.
After the offline check, send a controlled share of appropriate traffic to the candidate where the application supports it. Compare the live outcomes with the baseline. Watch for wrong answers, escalations, retrieval misses, and latency changes. A release that passes a fixed test set can still fail on the requests customers actually make.
Define the live stop rule before the canary begins. For the policy assistant, a new unsupported refund promise might stop the rollout immediately, while a small rise in handoffs might call for investigation. Assign someone to watch the first sample and someone who can move the release back. A canary with no decision owner merely exposes fewer users to the problem before the next scheduled review.
Make the comparison fair. If the candidate sees a new policy index while the current release sees the old one, a result change cannot be attributed to the prompt. Either test the whole new combination as one release or hold the other components fixed. The release record should say which choice was made so later readers do not mistake a mixed comparison for a prompt result.
06
Make rollback specific
Write down the exact switch and the person allowed to use it. In a prompt registry, rollback may mean moving a production label to an earlier prompt version. With that switch already in place, we have rolled back prompt changes in seconds. A model, retrieval index, tool policy, or application change may require a separate switch or deployment. Test the whole rollback path, including caches and clients that hold prompt versions locally. The label may move in seconds while some running clients continue serving the cached version until they refresh.
Practice rollback with a harmless change before an incident. Start a request against the candidate, move the pointer back, and inspect traces from requests immediately before and after the switch. Check whether a worker had cached the candidate and for how long. If the service can pin a prompt version for the life of a conversation, decide whether existing conversations finish on that version or are restarted. Both approaches have user-visible consequences; the incident playbook should state the chosen behavior.
Where the change included a retrieval index or tool permission, restore those separately and verify the combination that is actually serving. Rolling back only the prompt may leave the original fault in place. Keep a smoke test with a known answer and a refusal case so the on-call person can confirm both ordinary and risky behavior after the switch.
Rollback also has a limit: it stops new requests from using a bad combination, but it does not undo answers already given or actions already taken. Decide who investigates and corrects those outcomes. Keep the previous release’s sources and configuration long enough to explain the incident, subject to retention and access rules.
07
Reconstruct the answer without exposing the customer
For each request, retain a trace ID, time, workflow release, prompt and model identifiers, retrieval source IDs and versions, validation result, and final disposition. Where an investigation requires payloads, use controlled, redacted samples with defined retention. Version metadata often explains which configuration ran without copying every customer message into an observability system.
Exact replay may be impossible when a provider changes an alias, a source disappears, or model output is nondeterministic. The practical standard is to identify the configuration and evidence used, replay comparable conditions where possible, and explain the remaining uncertainty.
When a bad release has already answered customers, preserve a list of affected trace IDs under the access controls used for incident work. Compare the exposed traffic with the failure condition and decide whether anyone needs a corrected answer. The rollback restores future behavior; the incident owner handles the already delivered output. Record that decision beside the release so the next review has a complete account.
Start with the workflow that carries the most customer risk. Record its current configuration as release one, run a small regression set against it, name a rollback target, and practice the switch before the next prompt or model change reaches users.