01
Short answer
Measure data quality at the boundary where bad data can change a report, model, or operating decision. Define the accepted condition, owner, alert, and response for freshness, completeness, validity, and consistency before the pipeline runs.
A few years ago I sat in a planning review where two dashboards disagreed about the previous week's sales by about 12 percent. Both had green checkmarks. Both jobs had succeeded. The pipeline had run on time. What had actually happened was that one source system had backfilled a week of records after the load window closed, and the dashboard that looked wrong was the one telling the truth.
That is the failure mode data quality work exists to prevent. Not the loud failure, where a job dies and someone gets paged. The quiet one, where a report keeps rendering from stale or partial data and a decision rides on it.
02
What to measure
Start from the decision, not from a checklist. For every report, model, or workflow that consumes data, ask what would make its input wrong, and measure that. Usually the answer falls into a handful of dimensions.
Freshness comes first for operational work. Data arrives inside the window the decision allows. The measure that matters is event time to availability, not whether the job succeeded. A job that completes on schedule from a two-day-old source is a freshness failure that looks like a success.
Completeness means the expected records arrived. Compare record counts against source counts and against volume history. A sudden 40 percent drop in rows usually means an upstream filter changed or a source went quiet, and you want to know before the month closes on it.
Validity is about values conforming to rules: types, ranges, business logic. Watch the null rate on critical columns. It is a leading indicator. Nulls creep up for weeks before someone notices that a join has been silently dropping matches.
Consistency means the same fact agrees across systems, or the disagreement is known and documented. If the warehouse says 8,200 active customers and the CRM says 7,900, either one is wrong or the definition differs, and both possibilities need an owner.
03
Where the tests run
I have seen teams run data quality checks the way they run security scans: periodically, from outside, after the fact. It does not work. By the time a nightly audit notebook flags a problem, the bad data has been served to every consumer who opened a dashboard that morning.
Put the checks inside the pipeline, at the boundaries where data changes hands.
At ingestion, enforce contracts. The source declares what the schema, timing, and volume expectations are, and violations either stop the load or flag it with a specific reason. That check catches the backfill-after-cutoff problem from my opening story.
After transformation, declare expectations on each model, in code, versioned with the model itself, and run them on every build. If a model says "customer_id is never null and revenue is never negative," then every pipeline run proves it. These are cheap tests. Most teams find the first five serious bugs in the first month and never go back.
At serving, record what the consumer actually saw. The dashboard should display the freshness and completeness of its own data. A stale view that looks identical to a fresh one is a trap; a footer reading "source: 2 days old" turns the trap into information.
04
Who gets paged
Every check needs a named owner and a defined response before it fails. During the failure is the wrong time to figure out routing.
Freshness failures belong to the pipeline owner. Validity failures belong to whichever team produces the data. Consistency failures need a data steward with the authority to say which system holds the truth, because that call is sometimes political and someone has to be able to make it.
You also need to decide, per check, whether it blocks, pages, or annotates. Some data should stop the pipeline cold. Some should flow through with a quality flag that consumers can see. Deciding that mapping in advance, in writing, is most of the work. The worst pattern is a shared mailbox that owns everything and resolves nothing.
05
Keeping the measures honest
A check that has never fired is either perfect or dead. The way to tell the difference is to test it deliberately. Inject a known bad record once a quarter, confirm the check catches it and the page reaches the right person, and write down that you did it. Teams that drill their checks trust them. Teams that do not, eventually ignore them, and then the checks are decoration.
Watch the trend, not just the state. Track failure rate and time to resolution per check. A failure rate that creeps up on an unchanged pipeline means the world moved, not the code, and the matching rules need a look.
Review the suite quarterly and delete checks nobody would act on. Noise trains people to ignore signal. A small suite that everyone respects beats a large one they have learned to dismiss.
06
Build a scorecard around one decision
Take one operating decision, such as a daily replenishment order, and list the fields that can change it. For each field, define the expected condition, where the check runs, who owns the source, and what happens when the condition fails. This keeps the first scorecard small enough to use.
A useful entry looks like this:
Report the scorecard in business units. “Four stores missing at cutoff” tells an operator what is at risk; “pipeline completeness 98.7 percent” asks them to perform another calculation. Keep the technical measure underneath for trend analysis, but make the alert state the affected decision and deadline.
The scorecard also needs an exception rule. A planned store closure, late supplier file, or newly introduced identifier may be valid even though it breaks the default check. Exceptions need an owner, reason, affected scope, and expiry date. Permanent exceptions are usually missing requirements wearing a different name.
Review the scorecard with the people who make the decision. Remove checks that never change an action. Add cases found during incidents. Change thresholds when seasonality or operating hours change. The scorecard stays useful because it follows the workflow, not because the original design anticipated every failure.
- Store inventory must arrive by 05:30 for every open location. The ingestion job checks arrival time and store coverage. Missing locations stop automatic replenishment for that region and page the inventory-data owner.
- Product identifiers must match the active product table. The transformation job rejects unknown values into a review table. Merchandising owns the mapping decision before the next run.
- Sales quantities must stay within an agreed physical range. Values outside it are held from the forecast and shown to the store-operations analyst with the source row attached.
07
Separate warning, hold, and stop
Not every failure should stop the pipeline. A warning records a condition that remains safe to serve, such as a small delay inside the business tolerance. A hold excludes affected records while unaffected work continues. A stop blocks publication because consumers cannot make the decision safely.
Write the response beside each threshold before release. If engineers decide severity during an incident, two similar failures will receive different treatment and operators will lose confidence in the signal. Run one exercise for every severity so the team sees the exact alert, owner, and recovery path.
08
What good looks like
After a quarter or two of this work, the planning review changes character. Someone asks "can we trust yesterday's numbers," and instead of a discussion, someone puts the freshness, completeness, and validity test results on screen and the meeting moves on. Failures page a specific person with a specific reason. The number of silent surprises, decisions later found to have ridden on bad data, heads toward zero.
That is the whole program. Not a data quality dashboard. An answer you can give in one minute, with the test results behind it, when the question matters.
09
Related
Where to read next on the same subject.