QuickDevGuide

Home › ETL - Talend & SSIS

Data Quality Checks — ETL - Talend & SSIS

Run automated checks after each load and fail the pipeline (or alert) when they break. Check Example rule Completeness email IS NOT NULL for ≥ 99% of rows Uniqueness no duplicate orderid Validity amount > 0, status IN (...) Referential integrity every fact.customersk exists in the dimension Freshness newest loadedat < 2 hours old Volume row count within ±30% of the 7-day average Reconciliation source total = target total Tools: Great Expectations, dbt tests (unique, notnull, relationships), Soda, or plain SQL like below.

Key Takeaways