01
Related
Where to read next on the same subject.
02
Short answer
A production AI system draws on four kinds of operational data: the request context that frames the decision, the event stream that shows what is happening now, the governed reference data that gives records a shared meaning, and the captured outcomes that show what happened after earlier decisions. The evaluation set is built from the same path the decision used, so a review reflects what the system actually saw.
The first deployment I reviewed on a dealer network had a good model and a silent gap. The system scored retailer credit risk daily. Nobody had wired in the one field that changed the answer most: whether a distributor had already extended terms outside the system of record. The model read the warehouse and the warehouse read the ERP, and the ERP read a process that people ran by phone. Everyone downstream trusted their input. The decision sat on a hole.
That pattern repeats. Teams ask "where is the model getting its data" and get a lineage diagram that ends at a table. The better question is operational: for this decision, what must the system know, and does each input arrive, governed, and recorded?
03
The four inputs
Request context comes first. It answers who asked, for what decision, and under what constraints. Channel, entitlement, service level, and the current state of the customer all belong here. A replenishment order for a store closing next week differs from one for a store opening a second warehouse, and the difference lives in context, not in the sales history. When context is missing, the model fills the gap with the average, and the average is wrong in exactly the cases that matter.
The event stream is what is happening now. Orders, payments, readings, clicks, and cancellations arrive continuously, late, and out of order. The engineering questions are concrete: how much lateness is acceptable, what watermark applies, and what happens when a source replays a week after the load window. Teams that answer these in writing before launch spend their first quarter on improvements. Teams that answer them during an incident spend that quarter on apologies.
Reference data gives records a shared meaning. Products, customers, sites, cost, and hierarchy all change, and each change needs a date, a reason, and an owner. A model trained on last year's store hierarchy produces numbers nobody can match to a paycheck. The fix is dull and effective: one governed definition, a change record with effective dates, and consumers who read the version in force at the time of the decision.
Captured outcomes close the set. What did the system propose, what did a person do instead, and what followed? Overrides are kept with a reason, because a planner who refuses an order the model got wrong is a label, not a complaint. This record feeds the evaluation set, and the evaluation set decides whether the next version earns promotion.
04
The serving path ties them together
Assembly happens at read time. The features the decision needs are pulled from each input at the moment of the request, not precomputed days earlier and hoped current. Freshness is shown beside the answer: the age of each contributing source. A recommendation can carry its own provenance in a footer the way a report does.
Versioning comes with the assembly. The record shows which model version ran, against which data versions, at which timestamp. When someone asks in March why the system ordered forty units in January, the answer takes a minute instead of a meeting. Teams underestimate how much trust this single habit buys.
The path needs an owner for each input, not a general owner for the whole. The source team owns arrival, the pipeline team owns load and reject handling, the steward owns reference definitions, and the operating team owns the decision. Names on a page, not roles in a slide.
05
Build the evaluation set from the same path
An evaluation set that samples clean historical data tests the model on a world that no longer exists. Build it from the serving path instead: replay the last week or month of real requests, score the model against the outcomes that were captured, and review per segment. The differences between that result and the offline benchmark are themselves information, because they measure how much of production the clean sample never contained.
Replay also exposes the context problem directly. When the request context was missing in January, replay from March cannot reconstruct it. The evaluation shows the gap as variance the model cannot explain, which points the fix at the data path rather than at another round of tuning.
06
What breaks when an input is missing
The failure modes have a shape. Without request context, the system answers the average question accurately and the specific one badly. Without an event stream, the system acts on a world that has already moved, and the error grows with the tempo of the business. Without governed reference data, two teams defend different numbers, and the argument outlives the project. Without captured outcomes, nobody can say whether the system helped, and the budget conversation turns philosophical.
Each gap also has a cheap first fix. Context: log the request as it arrives, even before the model uses it. Event stream: record arrival times and volumes, and alert on the window. Reference data: version the hierarchy and date the changes. Outcomes: capture proposals and overrides in a table, with a reason field and nothing else to start.
07
What the inputs cost
Each input carries a different price tag, and the mix explains most budget arguments. Event streams cost continuously: ingestion infrastructure, backfill storage, and the on-call rotation that answers when the window slips. Reference data costs once per change if the governance is real, and once per incident if it is not. Request context costs almost nothing to collect and a great deal to reconstruct after the fact, which is why the logging habit pays for itself in the first quarter. Captured outcomes cost attention: someone must review overrides weekly or the record decays into a log nobody reads.
Teams that cost these inputs up front make better scope decisions. A deployment that drops captured outcomes to hit a date saves a little compute and loses the only mechanism that can prove value later. That trade is worth making explicitly, with a date to revisit, rather than silently, with a shrug.
08
Where to start
Take one live decision and list the four inputs against it. Mark each as present, present but unowned, or missing. That table, done in an afternoon, tells you more than a data audit would. It names the next three engineering tasks in priority order, and it gives the operating team a way to see progress against something they understand.
09
Related
The event path that carries these inputs is described in the ingestion guide. The quality checks that keep each input honest live at the boundaries where data changes hands. The dealer-network work in the two-wheeler case study shows the reference-data problem in a live migration.