01
Related
Where to read next on the same subject.
02
Short answer
Capture what the system proposed, what people did instead, and what followed. Attribute each outcome to the decision that caused it, allowing for delay. Evaluate on a cadence against the accepted baseline, per segment. Update rules and models through the same evaluation, with a review record on every promotion. A system without the loop is guessing with extra steps.
The best forecast model I ever saw deployed produced its own demise in the first month, and the mechanism was instructive. Planners overrode about a fifth of its recommendations. The overrides were logged, because logs are cheap, but nothing read them. The model owners read the accuracy report, saw a fine number, and shipped a retrain. The retrain learned from history that included the overrides' effects, politely reproduced the planners' corrections at slightly worse timing, and everyone spent six months arguing about whether the system had gotten worse. The loop was never closed. Everyone was guessing, expensively.
Every decision system needs the same four stages, and they cost less than one retrain done blind.
03
Capture: the part that must start on day one
Three records cover it. What the system proposed, with its version and its inputs. What the person did instead, including the override with a reason field, one line, free text, nothing fancier. And the outcome that followed, in the units the business uses: filled or lost, margin, days late, accepted or rejected.
Start before the model is good. The record you will wish you had in month six is the one you can only build in month one, and no backfill can invent an override reason. The capture cost is a table and a habit. The habit is the hard part: overrides that get logged lazily produce labels that poison the next stage, so the review cadence includes a look at the capture quality itself.
04
Attribute: the honest hard part
Which outcome belongs to which decision, and how long does the cause take to show? Sales decisions show effects in days to weeks; credit decisions in quarters; retention decisions across a year of churn curves. There is no correct answer in general. There is only the answer your system declares: a delay model, written down, applied identically at every evaluation.
Declare the attribution window with the business, and resist the temptation to make it flattering. A short window makes the system look responsive and punishes it on real effects. A long window makes everything look like the system's doing, including the promotion the marketing team ran. Both distortions get discovered, eventually, at the review with the most attendees.
05
Evaluate: replay, segment, baseline
The evaluation replays the captured period through the current system and compares it against the accepted baseline on the same data. Per segment, never on average, because the average is where the failing segment hides. The baseline was agreed when the system was accepted, and it stays the judge until a written decision replaces it.
Cadence matters more than depth. A weekly evaluation that people actually read beats a quarterly masterwork that arrives after the market moved. The cadence also bounds the damage: a regression caught in a week costs a week, and one caught in a quarter has usually already been promoted downstream.
06
Update: same gate, every time
Rules, thresholds, and models all change through the same gate. A retrained model replays the captured period. A threshold tweak replays the captured period. The promotion record carries the period, the segments, the result, the reviewer, and the date. No record, no promotion, including for the small changes, especially for the small changes, which is how systems quietly drift while everyone remembers them as stable.
07
The guardrails that keep the loop honest
Never train on what you cannot see. Outcome data arrives with gaps, and the gaps have shape: the orders that failed fastest are the ones least likely to be fully recorded. Feeding a model that biased record without naming the bias produces a system confident in exactly the places it is blind.
Overrides are signal, not noise. A planner refusing a bad order is a label the business paid to generate. Record it, review it, and let it improve the system. The opposite habit, penalizing overrides until planners stop making them, produces compliant silence and a model nobody trusts, which is the failure mode dressed as success.
Watch the drift between proposed and done. When the override rate rises on an unchanged model, the world moved, not the code. That signal routes to the same owner as the data quality alerts, and it arrives weeks before the accuracy report would have noticed.
08
What good looks like
The monthly review has the loop on one page: capture rates, override themes, evaluation results per segment, and the promotions with their records. A newcomer can trace any current threshold to the evaluation that set it. Nobody argues from memory, because nobody has to. The system got that way one captured outcome at a time, starting on day one.
09
Review the capture, not just the model
The loop is only as good as its record, so the capture gets its own review. Monthly, ten minutes: what share of decisions carry a complete record, what share of overrides carry a reason, and what the reasons look like this month. The numbers are humbling early. A capture rate of sixty percent with half the reasons empty is common in the first quarter, and it is a finding about the operating workflow rather than a scolding about the team: if entering a reason takes four clicks, the clicks will lose.
The reason field deserves qualitative reading, not just a rate. Overrides cluster, and the clusters are the product roadmap. "Promotion not in the model" is a data-path task. "Truck already full" is a constraint the system should know. "Supplier changed the lead time" is a master-data task. Reading fifty reasons in a month gives the team a ranked, grounded backlog that no amount of model tuning would have produced, and it gives the planners visible proof that their corrections are being heard, which raises the capture rate more than any exhortation.
Bound the review's ambition. The capture review does not redesign the model, and it does not relitigate individual overrides. It checks the health of the record, harvests the clusters, and routes the findings to owners. The discipline keeps the review fast enough to survive a busy quarter, which is when it is most needed and most often skipped.
The cost accounting helps the case for the whole loop. Capture costs minutes per day, attribution design costs one written decision, evaluation on a cadence costs compute that is nearly free next to a wasted retrain, and the update gate costs a review. Measure that stack against the single blind retrain from the opening story, and the loop pays for itself the first time it prevents one.
10
Related
The evaluation guide details the replay method. The guardrails guide covers the serving-side controls. The dealer recommendation case study shows a recommendation system whose feedback record began before the first model shipped.