Run automated checks after each load and fail the pipeline (or alert) when they break. Check Example rule Completeness email IS NOT NULL for ≥ 99% of rows Uniqueness no duplicate orderid Validity amount > 0, status IN (...) Referential integrity every fact.customersk exists in the dimension Freshness newest loadedat < 2 hours old Volume row count within ±30% of the 7-day average Reconciliation source total = target total Tools: Great Expectations, dbt tests (unique, notnull, relationships), Soda, or plain SQL like below.
Key Takeaways
Express each rule as a query that counts violating rows.
Check uniqueness, nulls, ranges, referential integrity, freshness and volume.
Reconcile totals between source and target for financial data.
Fail or alert the pipeline — a silent bad load is worse than a late one.