Bad data does not announce itself. It circulates quietly through dashboards, machine learning training sets, and financial reports until a stakeholder catches a number that cannot be right. By then, the bad data has influenced decisions for weeks or months. Gartner estimates that poor data quality costs organizations an average of $12.9 million per year — not in dramatic system failures, but in the accumulated cost of wrong inventory forecasts, duplicate customer records, and revenue figures nobody trusts.
Data quality management is the practice of preventing bad data from entering analytical systems and detecting it quickly when prevention fails. This guide covers the six dimensions of data quality, the toolchain for automated enforcement (Great Expectations, dbt tests, Monte Carlo), the governance structures that give those tools organizational backing, and a realistic implementation roadmap. By the end, you will understand what a mature data quality program looks like and how to build toward it incrementally.
Data Quality Management: Six Dimensions
Accuracy
Data accurately represents the real-world entity it describes. A customer record with the wrong shipping address is inaccurate, regardless of how complete or consistent the remaining fields are. Accuracy failures are the hardest dimension to detect automatically because they require comparison with an authoritative source — often a system outside the data pipeline.
Detection approaches: Cross-system reconciliation (comparing CRM records against order system records), sampling with manual verification, and integration testing that validates source-to-warehouse consistency at a representative sample rate.
Cost of failure: Marketing campaigns sent to wrong addresses, shipments to incorrect locations, clinical records misattributed to wrong patients. Accuracy failures in regulated industries carry compliance risk beyond operational cost.
Completeness
Required fields contain values. A product catalog with 200 missing prices out of 5,000 records is incomplete; the revenue forecast built on that catalog will understate actual inventory value. Completeness is the easiest quality dimension to measure automatically.
Detection approach: Calculate the null ratio for every required field. A completeness SLA of 99% for critical fields (product_id, order_amount, customer_id) is a reasonable baseline. Fields below threshold trigger pipeline alerts.
dbt test: not_null applied to required columns catches completeness failures at transformation time before incomplete records reach reporting tables.
Consistency
The same entity is represented identically across systems. When a customer's country is "UK" in the CRM, "United Kingdom" in the billing system, and "GB" in the analytics warehouse, aggregations break. The three values represent the same fact, but they count as three distinct groups in a GROUP BY query.
Detection approach: Cross-system attribute comparison for shared entities. Standardization rules in the transformation layer (dbt silver-layer models) enforce consistent representation before data reaches reporting tables.
Example: Standardizing country codes to ISO 3166-1 alpha-2 format across all source systems eliminates consistency failures caused by free-text country entry in upstream forms.
Timeliness
Data is available when users need it. A sales dashboard that refreshes every 24 hours fails timeliness requirements for a sales team making intra-day decisions. A fraud detection system that processes transaction features with a 10-minute lag fails when the fraud window is 30 seconds.
Detection approach: Measure the age of the most recent record in each table (MAX(updated_at) relative to current time). Pipeline SLA monitoring tracks whether each DAG run completes within the agreed freshness window.
Failure pattern: Pipelines that run on fixed schedules but take longer than the schedule interval create timeliness cascades. A 4-hour daily pipeline on a 6-hour schedule works until the pipeline grows to 7 hours — at which point every run starts before the previous one finishes.
Validity
Data conforms to defined format and value rules. Age = -50, phone_number = "abc123", email = "not_an_email", order_status = "COMPLETED_PARTIALLY" (when valid values are only "COMPLETE" and "CANCELLED") are all validity failures.
Detection approach: Validation rules applied at ingestion. Great Expectations expect_column_values_to_be_between, expect_column_values_to_match_regex, and expect_column_values_to_be_in_set cover the common validity patterns. dbt's accepted_values test catches values outside the allowed set.
Uniqueness
Records are not duplicated. The same customer appearing twice in the customer table — with slightly different name spellings or email addresses — causes double-counting in metrics, duplicate marketing sends, and inflated acquisition cost calculations.
Detection approach: Primary key uniqueness checks on every table. dbt's unique test validates that identifier columns contain no duplicates. Probabilistic deduplication (fuzzy matching on name + email + phone) catches near-duplicate records that exact-match uniqueness checks miss.
Automated Toolchain for Data Quality Management
Great Expectations
Great Expectations is a Python framework for defining and enforcing data quality rules. An "expectation" is a testable assertion about data: "the age column contains values between 0 and 120," "the email column has a null rate below 2%," "the order_status column contains only values in {'COMPLETE', 'CANCELLED', 'PENDING'}."
Expectations are organized into Expectation Suites — collections of tests for a specific data asset. When data arrives in the pipeline, Great Expectations runs the suite and produces a validation report: which expectations passed, which failed, and what the actual values were.
Integration patterns: Great Expectations runs as a step in Airflow DAGs, after data is extracted but before it is loaded into the warehouse Bronze layer. Failed validations halt the pipeline and trigger alerts — preventing bad data from propagating downstream.
Data Docs: Great Expectations generates HTML documentation from the Expectation Suites, giving non-technical stakeholders a readable description of what the data quality rules are and whether they passed last run.
dbt Tests
dbt's built-in testing framework applies quality checks at transformation time, after raw data enters the warehouse but before it is published to reporting tables. Four standard tests cover the core quality dimensions:
unique— no duplicate values in the columnnot_null— no null valuesaccepted_values— only values from an explicit listrelationships— foreign key integrity between tables
Custom dbt tests extend beyond these defaults. A custom test can assert that a revenue column never decreases week-over-week, that a count column is always positive, or that a percentage field is always between 0 and 1. These are defined as SQL models that return failing rows — zero rows means the test passes.
dbt test failures are surfaced in CI/CD pipelines, blocking deployment of transformation changes that would introduce quality regressions. This makes data quality a development-time concern, not a production incident.
Monte Carlo (Data Observability)
Monte Carlo is a data observability platform that learns what "normal" looks like for each table in the warehouse and alerts on anomalies — without requiring manually defined thresholds.
Rather than specifying that the orders table must have between 4,000 and 6,000 rows per day, Monte Carlo learns from historical distributions and alerts when today's row count falls outside the expected range. This covers the failure class that rule-based tools miss: gradual drift, seasonal variation, and black-swan events that nobody thought to write a rule for.
Value proposition: Great Expectations and dbt tests catch the failures you anticipated when writing the rules. Monte Carlo catches the failures you did not anticipate.
Integration: Monte Carlo connects to the warehouse via read-only access and operates without changes to existing pipelines. Most organizations deploy it alongside (not instead of) Great Expectations and dbt tests.
Data Governance Framework
Data quality tools without governance structures produce alerts that nobody acts on. Governance defines who owns each data asset, who is responsible for quality, and what the escalation path is when problems occur.
Data Ownership
Every table in the warehouse has a named owner — not a data engineering team, but a specific business stakeholder in the originating system. The customer table is owned by the VP of Sales Operations. The order table is owned by the Head of Commerce. The appointment table is owned by the Clinical Operations Director.
Data owners are accountable for the accuracy and completeness of their source system's data. When a quality alert fires for their table, they are the first call. This model prevents quality issues from falling into a gap between the data engineering team ("not our source system") and the business team ("not our data pipeline").
Data Catalog
A data catalog (Collibra, Alation, Atlan, Apache Atlas) provides a searchable inventory of every table, column, and metric in the warehouse. Each entry includes: the business definition (not just the technical column name), the data owner, the freshness SLA, quality metric history, and lineage (which upstream tables feed this one).
Operational value: Analytics engineers self-serve on data discovery — they find the table they need without asking the data engineering team. Data owners receive automated notifications when their table's quality metrics drop below threshold. New hire onboarding time for analytics roles decreases significantly when a catalog exists.
Data Stewardship
A data steward is the operational counterpart to the data owner. Owners define what the data should be; stewards monitor whether it actually is. Stewards review quality dashboards, investigate alert patterns, coordinate with data engineering when pipeline changes are needed, and maintain the catalog entries for their domain.
In most organizations, the data steward role is assigned to a senior analyst or analytics engineer within a business unit — not a separate data governance team. The governance committee (cross-functional, monthly) reviews quality trends, approves new data source onboarding, and updates access policies.
Data Lineage
Data lineage tracks the path from every source record to every downstream dashboard or report. dbt generates column-level lineage automatically within the transformation layer. OpenLineage (an open standard) captures lineage across tools (Airflow, dbt, Spark) into a unified graph.
Why lineage matters for data quality management: When a dashboard shows wrong numbers, lineage shortens incident response from "dig through all the pipeline code" to "navigate the lineage graph to the broken node." For organizations with hundreds of tables, lineage is not a nice-to-have — it is the difference between a 2-hour incident and a 2-day incident.
Implementation Roadmap
Data quality management is not a project — it is a program that builds capability incrementally.
Phase 1 (0–3 months): Foundations
Profile existing data sources to understand current quality levels. Identify the top 10 most-used tables and define SLAs for completeness and uniqueness. Add not_null and unique dbt tests to all primary key columns in the top 10 tables. Implement pipeline alerting in Airflow so failures surface to the data engineering team within 15 minutes.
Phase 2 (3–6 months): Automated Enforcement
Deploy Great Expectations on the highest-risk ingestion pipelines (financial data, customer records). Add accepted_values and range tests for critical business fields. Implement a basic data catalog (even a documented Confluence space is better than nothing at this stage). Assign data owners for the top 20 tables.
Phase 3 (6–12 months): Governance Structure
Stand up a formal data governance committee with monthly cadence. Deploy a purpose-built data catalog tool (Atlan or Collibra). Implement data lineage tracking across dbt and Airflow. Add Monte Carlo or Soda for anomaly detection. Define quality SLA targets for each business domain and track them in quarterly reviews.
Phase 4 (12+ months): Continuous Improvement
Expand Great Expectations coverage to all production pipelines. Use quality metrics as input to a "data trust score" shown to BI consumers for each table. Automate incident routing: quality alerts for a given table auto-assign to the correct data steward.
Measuring Data Quality
Data quality management requires quantitative targets. Key metrics:
| Metric | Formula | Typical target |
|---|---|---|
| Completeness rate | Non-null rows / total rows | >99% for critical fields |
| Uniqueness rate | Distinct PKs / total rows | 100% for primary keys |
| Consistency rate | Records with consistent values / total | >99.5% |
| Validity rate | Records passing format/range rules / total | >99% |
| Pipeline timeliness | % of DAG runs completing within SLA | >99% |
| MTTR (mean time to resolve quality incidents) | Incident open to close in hours | <4 hours for P1 |
Tracking these metrics weekly and reviewing them in governance committee meetings transforms data quality from an invisible background concern into a managed operational capability.
Conclusion
Data quality management is not a technology problem — it is an organizational capability built on tooling (Great Expectations, dbt tests, data observability) and governance (data ownership, catalog, stewardship, lineage). The tools catch failures; the governance structures ensure someone acts on them. Organizations that invest in both see measurable improvements in decision-making speed and stakeholder trust in analytics — and avoid the compounding cost of bad data propagating silently through their most important reports.
Related Articles
MLOps Guide: Taking Machine Learning Models to Production [2026]
87% of machine learning models built by data science teams never reach production. The models work — they pass cross-validation, they score well on holdout sets, they demonstrate genuine predictive value. The problem is not the modeling. The problem is everything that happens between a notebook experiment and a reliable, monitored, production system. MLOps is the discipline that closes that gap. This guide covers the full MLOps stack: maturity levels, tooling choices (MLflow, DVC, Kubeflow
Read More: MLOps Guide: Taking Machine Learning Models to Production [2026]LLM Fine-Tuning Guide: Custom Model Training with LoRA and QLoRA [2026]
General-purpose LLMs are impressive. They can write code, summarize documents, answer questions, and translate between languages with reasonable accuracy. But "reasonable" is not good enough when your application requires consistent output format, domain-specific terminology, a particular tone, or behavior that the base model was never trained to exhibit. That gap is where fine-tuning matters. Fine-tuning updates a model's weights on your specific data, changing how the model behaves — not
Read More: LLM Fine-Tuning Guide: Custom Model Training with LoRA and QLoRA [2026]Computer Vision Applications: Object Detection, OCR, and Industrial AI [2026]
Computer vision has moved well past the research phase. The models are trained, the frameworks are mature, the hardware is accessible, and the use cases are generating measurable returns. What was a specialized capability requiring deep expertise in 2018 is now deployable infrastructure — if you know which component to reach for and where the real complexity lives. This guide covers computer vision applications across industrial, medical, logistics, and document processing domains. It expl
Read More: Computer Vision Applications: Object Detection, OCR, and Industrial AI [2026]