Everrest warehouse load - data-quality gate

Raw dump → warehouse ingestion · seed 42 · every figure computed by dq_gate.py, none typed

BLOCKED as received · 5 blocking failures SHIP WITH HOLDS after fixes · 83 payments on hold

80.49DQ score in

99.97DQ score out - not 100 while a hold is open

104,099lines loaded

completeness FIXED

99.0 → 100.0

customer attributes present

1,074 lines arrived without customer master records - quarantined with reasons

validity FIXED

91.1 → 100.0

category in canon set

14 category spellings arrived for 8 real categories - mapped through a reviewed dictionary

consistency FIXED

0.0 → 100.0

merchant key formats agree across systems

orphan references, a cross-system key split and one region's clock bug - quarantined, conformed and corrected

timeliness PASS

100.0 → 100.0

every feed fresh within SLA

no defects found in this load

uniqueness FIXED

93.0 → 100.0

one identity, one customer id

1,324 customer ids belonged to people who already had one - merged, earliest id kept

accuracy HOLD

99.8 → 99.8

reported amounts free of round-number heaping

one merchant reports only round amounts - suspected fabrication, held out of finance rollups

What the bad data was about to cost

Exposure	Amount	Where it would have landed
GMV on quarantined lines	$215,076	revenue-by-product and finance reconciliation would disagree by this much
Reported amounts on finance hold	$46,200	suspected fabrication that would have been booked as clean revenue
Active-buyer overstatement	3.5%	662 duplicate ids inflating every buyer and repeat-rate metric
Settlement-lag distortion	-8h on 8.2% of payments	one region's settlement SLA reporting would have been fiction

Every check, before and after

Dimension	Check	Table	Rows checked	Failed (in)	Error rate	Failed (out)	Status
validity	structural schema (pandera)	transactions	107,841	0	0.00%	0	PASS
completeness	customer attributes present	transactions	107,841	1,074	1.00%	0	PASS
validity	category in canon set	transactions	107,841	9,572	8.88%	0	PASS
consistency	product_id exists in catalog	transactions	107,841	2,696	2.50%	0	PASS
consistency	payment's order exists in transactions	payments	46,007	0	0.00%	0	PASS
consistency	merchant key formats agree across systems	merchants	107,841	107,841	100.00%	0	PASS
consistency	paid_at is not before order_ts	payments	46,007	3,798	8.26%	0	PASS
timeliness	every feed fresh within SLA	payments	2	0	0.00%	0	PASS
uniqueness	one identity, one customer id	customers	18,888	1,324	7.01%	0	PASS
accuracy	reported amounts free of round-number heaping	payments	46,007	85	0.18%	83	HOLD

Quarantine ledger - where every row went

Action	Rows	Example reason
quarantined	4,668	customer C800303 has no master record
corrected	3,754	paid_at shifted +8h to UTC - region PH diagnosed as a local-time export (100% impossible latencies)
merged	662	duplicate identity of C011073 (same signup second, channel, region; earliest id survives)
finance_hold	83	merchant in ['M0333'] under round-amount investigation - excluded from finance rollups, amount untouched; evidence snapshot sha256 241e8a5c7b24a2ec

How the scores are computed. check error rate = rows failing / rows checked on the check's own grain · dimension score = 100 x (1 - worst check's error rate) - a dimension is as good as its worst defect · overall = mean of the six dimension scores · the verdict ignores the average: any BLOCKER failure blocks the load, and fabrication suspicion (83 payments) HOLDS finance rollups rather than editing amounts. Reconciliation: 107,841 lines in = 104,099 loaded + 3,742 quarantined.