What Should a dbt CI Pipeline Check Beyond Tests?

What Gap Do dbt Tests Leave in CI?

Most dbt CI pipelines follow the same pattern: run dbt build, execute tests, and merge if everything is green. The problem is that dbt tests validate constraints — not-null, uniqueness, accepted values, referential integrity — but they do not validate data output.

A model can pass every test and still produce wrong results. An incorrect filter silently drops records. A bad JOIN fans out rows. A calculation uses the wrong column. The tests pass because the output is structurally valid. The data is just wrong.

This is the gap: dbt tests check structure, but your CI pipeline needs to check data impact.

What Additional Checks Should CI Include?

Beyond dbt tests, a robust CI pipeline should compare PR branch output against the production baseline using four types of data validation checks:

Check Type	What It Detects	When It Matters
Schema diff	Column additions, removals, type changes	Every PR — schema changes break downstream consumers
Row count diff	Data loss, duplication, unexpected growth	Every PR — row count shifts signal logic errors
Profile diff	Distribution changes in column values	PRs touching business logic or filters
Value diff	Specific value-level differences	PRs affecting critical models

Schema Diff

A schema diff compares the column structure of a model between your PR branch and production. It catches added columns, removed columns, renamed columns, and type changes. Schema changes are especially important because they can silently break downstream models, dashboards, and reverse ETL pipelines that depend on specific column names or types.

Row Count Diff

A row count diff compares the number of rows in a model between environments. A significant drop often indicates a filter bug or failed JOIN. A significant increase may indicate duplication or an overly permissive filter. Either way, it is a signal that something changed beyond what the code alone would suggest.

Profile Diff

A profile diff compares statistical summaries of columns — min, max, mean, median, null percentage, distinct count — between environments. It catches distribution shifts that row counts miss. For example, if the average order value drops by 30% because a filter excluded high-value transactions, the row count might barely change, but the profile diff will flag it immediately.

Value Diff

A value diff compares actual row-level data between environments. This is the most granular and most expensive check. Reserve it for critical models where being wrong has direct business impact — revenue tables, customer-facing metrics, models feeding ML pipelines.

What Are Critical Models and How Do You Identify Them?

Not every model in your dbt project needs the same level of CI validation. Critical models are the ones where being wrong has significant business consequences.

Critical models typically share one or more of these characteristics:

• Customer-facing — powers dashboards, reports, or APIs that stakeholders see directly
• Revenue-related — feeds billing, financial reporting, or pricing logic
• ML pipeline input — serves as a feature table for machine learning models
• High fan-out — has many downstream dependencies, amplifying the impact of errors

Identify them through domain knowledge. Ask your team: “Which models trigger a stakeholder call when something goes wrong?” Those are your critical models.

How Do You Automate Data Checks in CI?

Automation turns data checks from a manual practice into a repeatable process. The key concept is preset checks — a configuration file that defines which models to check and what types of diffs to run.

A typical preset configuration (in a file like recce.yml) specifies:

• Target models — which models to run checks on (usually critical models)
• Check types — schema diff, row count, profile diff, or value diff for each target
• Thresholds — what constitutes a blocking difference (e.g., row count change > 5%)

A Practical CI Workflow

Here is the end-to-end workflow for a dbt CI pipeline with data checks:

1. dbt build — build and test the PR branch against a CI environment
2. Run preset checks — execute the configured data checks, comparing PR output against the production baseline
3. Compare to baseline — evaluate results against defined thresholds
4. Post summary to PR — format the results as a structured PR comment so reviewers have immediate context
5. Block on mismatches — if critical model checks detect unexpected differences, mark the CI check as failed

This workflow integrates naturally with GitHub Actions or any CI provider. The data checks run after dbt build completes and report results back to the pull request.

What Does “All Signal, No Noise” Mean for CI Checks?

A common failure mode for data checks is alert fatigue. If every PR generates a wall of “check passed” messages, reviewers stop reading them. The all signal, no noise philosophy means: only report when something is different.

In practice this means:

• Do not report passing checks — if the schema matches, the row count is stable, and the profile is unchanged, there is nothing to report
• Report only mismatches — surface the specific models and columns where differences were detected
• Provide context with the signal — show the actual values (e.g., “row count changed from 142,387 to 138,201”) so reviewers can assess severity immediately
• Distinguish blocking from informational — some checks should block the merge; others are advisory

This approach keeps the CI output actionable. When a reviewer sees a data check result, it means something actually changed and needs attention.

How Do CI Checks and PR Review Work Together?

CI checks and human PR review serve complementary roles:

	CI Checks	Human Review
Speed	Runs in minutes on every PR	Requires scheduling and availability
Scope	Predefined checks on critical models	Can explore any model or question
Strength	Catches obvious regressions consistently	Evaluates intent, context, and business logic
Limitation	Cannot judge whether a change is correct for the business	Cannot run on every PR at full depth

CI catches the obvious and repetitive — schema breaks, row count drops, distribution shifts. Human review handles the nuanced — “Is this metric change expected given the business context?” Together, they form a layered defense against data errors.

How Does This Prevent “Tests Pass but Data Is Wrong”?

The tests-pass-data-wrong scenario happens when a change is structurally valid but semantically incorrect. dbt tests confirm the structure is fine. CI data checks close the gap by comparing actual output:

• A buggy filter that drops records? Row count diff catches it.
• A calculation error that shifts averages? Profile diff catches it.
• A column rename that breaks downstream? Schema diff catches it.
• A JOIN change that subtly inflates metrics? Value diff on the critical model catches it.

Without these checks, the only defense is a human reviewer manually running queries — which happens inconsistently at best. Automated CI checks make this validation happen on every single PR.

Summary

dbt tests validate structural constraints, but your CI pipeline needs to go further. Add data-level checks — comparing structure, volume, distributions, and values between environments — to catch issues that tests miss. Focus validation on critical models — the ones where being wrong triggers stakeholder calls. Automate checks using preset configurations that define what to check and what thresholds to enforce. Follow the “all signal, no noise” philosophy: only report differences, not passing checks. CI handles the obvious regressions; human review handles the nuanced judgment calls. Together, they close the gap between “tests pass” and “data is correct.”