how-to-data-quality

Input - the raw dump, before anyone has cleaned anything

This gate sits at stage 0 of the lineage: the same four raw files the warehouse build consumes. Quality problems are ingestion problems - catch them where they enter, with a ledger, not downstream where they surface as a wrong board number.

Sample data - six planted defects, one per quality dimension

The raw generator plants six defects on purpose (seed 42, each documented in its docstring), and they map one-to-one onto the classic data-quality dimensions - so the gate is a complete tour of the discipline on data where the ground truth is known and recall can be verified.

Honesty note: the event-order defect was originally filed under "timeliness" - the expert panel corrected it to temporal consistency, and timeliness got a real freshness-SLA check instead. The page you are reading reflects the reviewed mapping.

Objective - does Monday's load ship?

Not "run some checks". A gate answers a decision question with a verdict, and accounts for every row it touched on the way.

Find-skills - the quality-gate toolbox

A gate needs declarative structural checks, custom forensic scans, a ledger discipline, and one self-contained artifact an executive can open anywhere.

Build - FAIL, fix, PASS, and account for every row

One script runs the gate on the raw dump (it genuinely blocks), applies the ingestion fixes with a ledger, re-runs the same check functions on what ships, and generates the scorecard from the results dict. Grab the real code below.

The gate at a glance - every check, before and after

Red = blocking failure, amber = warning, teal = pass. The PASS panel is genuinely re-executed on the fixed tables - only the fabrication hold survives, by design.

What tripped - the evidence, one chart per defect family

Top-left: 14 labels arrive for 8 categories - the fix is an explicit reviewed mapping that raises on anything unmapped, never fuzzy matching. Top-right: 3,798 payments precede their own orders, every one in a single region: a timezone diagnosis, so the whole region cohort is shifted to UTC and any residual negative latency is quarantined instead of blanket-"corrected". Bottom-left: two orphan populations, quarantined with reasons. Bottom-right: one merchant's amounts land on round hundreds 9,184x the peer rate (suspect excluded from its own baseline) - held, not edited.

Every row accounted for

Transactions: 107,841 in = 104,099 loaded + 3,742 quarantined. Payments: 46,007 in = 45,081 loaded + 926 quarantined - those 926 followed their quarantined orders out, because a fix that strands orphan payments is itself a defect (a reviewer caught v1 doing exactly that). Corrected (3,754), merged (662) and held (83) rows live in the ledger.

The scorecard - live below, every number computed

Generated by the script from the results dict - zero hand-typed figures. Badge language: PASS = was never dirty, FIXED = repaired at ingestion, HOLD = routed to investigation, still open.

Expert review - the panel that found the gate's own defects

Four senior reviewer agents - data-quality governance, analytics engineering, payments fraud, and board-facing insights, each 10+ years - reviewed the real code and outputs. They found defects in the gate itself. Every fix below is in the shipped code, and the numbers on this page are from the re-run.

# v1 (before): quarantine transaction lines... and strand their payments
tx = tx[~orphan_cust]
tx = tx[~orphan_prod]        # 926 payments now reference orders that no longer ship -
                             # they load anyway, in no ledger. "Every row accounted for" = false.

# v2 (after): payments follow their orders out, with reasons
orphan_pay = ~pay.order_id.isin(tx.order_id)
for r in pay[orphan_pay].itertuples():
    ledger_rows.append({"table": "payments", "row_id": r.pmt_ref,
                        "action": "quarantined",
                        "reason": f"order {r.order_id} was quarantined upstream"})
pay = pay[~orphan_pay]       # 46,007 = 45,081 loaded + 926 quarantined - printed every run