01
Related
Where to read next on the same subject.
02
Short answer
Monitor the path, not the jobs. Five signals cover a data pipeline from source to decision: source arrival, ingestion health, data quality, serving freshness, and the state of the operating decision itself. Each signal carries a named owner and a response defined in advance. The dashboard answers one question in one minute: is the decision safe to make today.
A pipeline team once showed me their monitoring with real pride: forty dashboards, every job a tile, everything green. I asked one question, the same one their users would ask: "can we trust yesterday's numbers?" Silence, then forty minutes of tile-by-tile archaeology. The jobs were monitored. The path was not. Nothing on those forty dashboards could answer a question anyone would actually ask.
Job monitoring and path monitoring answer different questions. A job that succeeds is a fact about a job. A decision that can be trusted is a fact about a path, and paths fail differently: quietly, between the green tiles.
03
The five signals
Source arrival comes first, because everything downstream inherits its failures. Did the file, feed, or stream arrive, and did it arrive inside the window the decision allows? The measure is arrival time against the expected window, and the owner is the source team, which is a statement of fact rather than an insult: they run the system that emits the data.
Ingestion health is what most teams monitor today: the job ran, rows loaded against recent volume, rejects handled with reasons, schema drift caught at the door. Keep it. Extend it with the volume history, because a job that succeeds while loading forty percent fewer rows has not succeeded in any sense that matters to the business.
Data quality is the layer that turns loads into trust: freshness, completeness, and validity test results, plus the null-rate trend on critical columns. Nulls creep up for weeks before anyone notices a join silently dropping matches. The trend is the signal; the state is already the incident.
Serving freshness measures what consumers actually see: the age of the data on every dashboard and API response. A stale view that looks identical to a fresh one is a trap, and the footer that says "source: two days old" turns the trap into information. This signal belongs to the platform team, and its appearance on consumer surfaces is what makes the whole path honest.
The decision itself is the fifth signal, and the one teams skip. Did the replenishment run, did the forecast publish, did the report go out, and did all of them run on trusted data? This is the signal the operating team cares about, and it is computed from the other four rather than measured separately. Its owner is the operating team, which is fitting: they hold the question the other four signals exist to answer.
04
Give each signal a response in advance
Three severities cover the field. A warning records a condition safe to serve, with the flag visible to consumers. A hold excludes affected records while unaffected work continues. A stop blocks publication and pages, because consumers cannot make the decision safely on what is there.
Write the severity beside each signal before release, and name the owner in the same line. If severity is decided during an incident, two similar failures get different treatment and everyone learns to distrust the page. The worst arrangement ever inflicted on a pipeline is the shared mailbox: it owns everything and resolves nothing.
05
Drill the alerts, or decorate with them
An alert that has never fired is either perfect or dead, and the way to tell is to drill. Inject a late file once a quarter. Prove the page reaches the named owner, and time how long the response takes. Track time-to-resolution per signal, and review the trend: a resolution time that creeps up on an unchanged pipeline means the ownership has quietly gone vague, and the fix is a conversation rather than a tool.
Delete alerts nobody acts on. Noise trains people to ignore signal, and the training is cumulative and permanent. A small set of signals everyone respects will outperform a rich set everyone has learned to dismiss.
06
The one-minute dashboard
Build the consumer-facing view in business terms. A header that states the decision state: yesterday's sales, complete, arrived 05:42, trusted. Tiles underneath that show the evidence for the header: the five signals with their values, their owners, and their severity states. Percentages and trends live on a second layer for the engineers; the first layer belongs to the planner who asked the question.
The test for the dashboard is the meeting. Someone asks whether the numbers can be trusted, and instead of a discussion, someone puts the dashboard on the screen and the meeting moves on in under a minute. When that happens, the path is monitored. Until it happens, the jobs are monitored, which is a start and only a start.
07
A morning on the five signals
Concrete helps, so here is the path on a normal Tuesday for a retailer running daily replenishment. At 05:00 the source signal fires: the store sales file from the POS vendor arrived at 04:52, inside the window, green. The ingestion signal follows at 05:20: rows loaded within two percent of the recent volume trend, rejects at the usual rate with their usual reasons, schema unchanged. Both signals are computed by the pipeline, not watched by a person.
At 05:45 the quality signal evaluates: freshness measured as event time to availability, completeness against the store coverage list, validity checks on the price and quantity columns. Today, one region shows a completeness shortfall: 14 of 200 stores missing, traced within minutes to a store-system upgrade at a regional hub. The severity table decides for the team, because it was written on a calm day: a hold for that region, an automatic page to the inventory-data owner, replenishment for the other 186 stores proceeds.
At 06:00 the serving signal reflects the hold: the regional dashboard footer reads "14 stores excluded, data as of yesterday," and the API returns the same flag. Nobody discovers the gap by noticing wrong numbers, because the numbers are labeled. At 06:30 the decision signal reports the day's state: replenishment ran for 186 stores on trusted data, one region held pending the backfill, and the estimated recovery time is 09:00.
That is the whole system doing its job: no heroics, no archaeology, no meeting to determine whether the numbers are real. The owner of the missing feed works the backfill with a named escalation path into the vendor, the operating team plans around a stated recovery time, and the review of the incident next week is ten minutes long because the signals already told the story. Every part of that morning was designed in advance, which is the point of giving each signal an owner and a response before anything fails.
08
Related
The quality guide details the tests behind signal three. The ingestion guide covers the mechanics of the event path. The post-trade case study shows monitoring wrapped around a settlement workflow where the decision signal had regulatory teeth.