Data teams at scale can no longer afford monolithic test suites that re-run thousands of table-level checks on every change. The combination of dbt’s manifest-based lineage and Great Expectations’ flexible validation runtime lets engineering teams run focused, incremental data-contract testing: only execute tests for models that changed and for their downstream dependents. This guide walks analytics and data engineers through a production-ready implementation pattern—architecture, concrete code examples, CI integration, and operational practices—to reduce test time, increase signal-to-noise, and maintain data SLAs.

Why lineage-driven incremental testing?

Traditional test strategies run the entire test suite after each push or schedule. As a repository grows, that becomes slow and brittle. Lineage-driven incremental testing offers three immediate benefits:

  • Speed: Run a small subset of checks (often 10% of tests) after a change, enabling fast feedback in PRs and CI.
  • Relevance: Tests run where risk actually changed—the modified models and their downstream consumers—reducing false positives from unrelated tests.
  • Cost control: Less compute consumed for validations against cloud data warehouses, lowering billable runtime.

High-level architecture

Implement this pattern as a lightweight orchestration layer that sits between your CI and your validation runtime:

  1. Change detection: Identify modified dbt models via git diff + dbt state or using manifest comparisons.
  2. Lineage expansion: Use dbt’s manifest.json to compute downstream dependents for modified nodes.
  3. Test mapping: Map those nodes to Great Expectations checks (tests defined in dbt tests or a GE repository).
  4. Selective execution: Programmatically create and run GE validation batches only for impacted datasets.
  5. Reporting & remediation: Persist validation output to a central store and surface failures in PRs and Slack with remediation context.

Prerequisites

  • dbt project with compiled artifacts enabled (manifest.json produced by dbt run/compile or dbt build).
  • Great Expectations project configured with a SQL datasource (SQLAlchemy or the cloud provider connector you use).
  • CI platform (GitHub Actions, GitLab CI, or other) and an orchestration layer for running the selective test job (script, lightweight service, or workflow task).
  • Access to a metadata store is optional but helpful (artifact storage for manifests, or a metadata DB for test results).

Step 1 — Detect changed models

There are two common approaches:

  • dbt state selector (fast in PRs): Use dbt’s state comparison to list modified resources. Example: dbt ls --select state:modified --resource-type model.
  • Git diff on compiled SQL (when state is unavailable): Compare compiled SQL output or manifest.json between the current branch and target branch (e.g., main) to detect models whose compiled SQL changed.

For multi-developer repositories, prefer the dbt state selection because it understands dbt resource semantics (e.g., tags, ephemeral models, and ref-based changes).

Step 2 — Expand via dbt manifest lineage

dbt writes a manifest.json that contains the full DAG and test associations. Read the manifest to compute the closure of affected nodes:

  1. Load manifest.json
  2. For each changed model, walk its children recursively to gather downstream models (or use dbt CLI with +model selectors).
  3. Deduplicate resulting model list and restrict to materialized models (table/view) if you only test persisted artifacts.

Example Python snippet to compute downstream impacted models (simplified):

import json
from collections import deque

def impacted_models(manifest_path, changed_models):
    with open(manifest_path) as f:
        manifest = json.load(f)
    nodes = manifest.get("nodes", {})
    impacted = set()
    q = deque(changed_models)
    while q:
        node = q.popleft()
        if node in impacted:
            continue
        impacted.add(node)
        children = nodes.get(node, {}).get("child_map", {}).keys() or nodes.get(node, {}).get("children", [])
        for c in children:
            q.append(c)
    return impacted

Note: dbt manifest schema changes across versions. Use keys that exist in your environment (inspect a sample manifest to confirm).

Step 3 — Map models to tests

There are two common sources of tests to run:

  • dbt-native tests (schema & data tests declared in .yml files) — the manifest includes test nodes that reference the model(s) they validate.
  • Great Expectations suites maintained alongside models — you can keep a naming convention to map GE suites to dbt models (e.g., customers__core_suite).

Strategy: combine both. Use manifest to list dbt tests that reference impacted models, and also look up GE suites for impacted models by naming convention or metadata tags.

Step 4 — Programmatically trigger GE validations for only impacted datasets

Great Expectations supports programmatic validations via Checkpoints (or, in newer releases, via the Validation API). The pattern below uses the GE context to construct a batch request per impacted model and run the named suite associated with it.

Conceptual workflow:

  1. For each impacted model determine the SQL (or table) to validate (dbt exposes the compiled SQL path and relation).
  2. Create a GE BatchRequest that points at that SQL/table.
  3. Run GE expectations (the suite associated with that model).
  4. Collect the JSON validation result for reporting.

Simplified Python pseudocode (adapt to your GE version and datasource):

from great_expectations.data_context import DataContext

context = DataContext("/path/to/great_expectations")

def run_validation_for_model(model_name, table_identifier, expectation_suite_name):
    batch_request = {
        "datasource_name": "warehouse_sql",
        "data_connector_name": "default_runtime_data_connector",
        "data_asset_name": model_name,
        "runtime_parameters": {"query": f"select * from {table_identifier}"},
        "batch_identifiers": {"validation_run_id": "pr-1234"},
    }
    result = context.run_validation_operator(
        "action_list_operator", 
        assets_to_validate=[{
            "batch_request": batch_request,
            "expectation_suite_names": [expectation_suite_name]
        }]
    )
    return result

Many teams use a simple convention: the GE suite name equals the dbt model unique_id (or its ref path). That makes the mapping deterministic and easy to automate.

Step 5 — CI integration and execution modes

Choose execution modes depending on the target lifecycle stage:

  • Pull Request (fast): Run only tests for changed models + immediate downstream consumers. Fail the PR if critical expectations fail. Keep run-time under a few minutes for developer productivity.
  • Pre-merge full validation (optional): For high-risk changes, run a wider set—e.g., include two levels of downstream dependents or run all schema tests.
  • Nightly / Canary runs: Run the full GE test suite periodically to catch environment-level drift (e.g., external upstream changes).

In CI, implement a three-step job:

  1. Detect changed models (git + dbt state).
  2. Resolve impacted models via manifest.
  3. Run GE validations and publish results (JSON + summary) to the CI pipeline and to a results DB/table.

Step 6 — Reporting, alerting, and triage

Validation output should be actionable:

  • Write validation JSON to an artifact store and index a summary row in your observability table: model, suite, status, failure_count, run_time, commit_sha, run_id.
  • For PR failures, post a concise summary to the PR (failed expectations, sample failing rows), and include the validation run link.
  • Use severity classification: blocking failures (schema mismatch, row count 0), warning failures (high null %), and info (distribution drift).

Operational tips and pitfalls

  • Manage flaky expectations: Tag or mark nondeterministic checks (e.g., source freshness vs distribution checks) so PR runs treat them as warnings rather than blockers.
  • Baseline tolerances: For numeric distribution checks, store historical baselines to avoid frequent drift-based failures; use rolling-window baselines for dynamic sources.
  • Limit runtime on large tables: For very large relations, validate with sampling or define a lightweight “canary” set of checks on metadata (row count, schema) in PRs while running full checks in scheduled jobs.
  • Secure credentials: Ensure CI jobs use short-lived credentials to query production data, and grant least privilege to the validation service.
  • Keep manifest artifacts versioned: Store compiled manifests for each commit (or use dbt's --manifest-path) so test jobs in CI can reference exact DAG snapshots.

Scaling considerations

When a repo grows to hundreds of models, follow these practices:

  • Cache lineage lookups: Persist manifest-derived dependency graphs in a small metadata DB for fast traversal.
  • Parallelize validations: Run independent model validations in parallel up to your warehouse concurrency limits. Throttle to avoid overload.
  • Group lightweight vs heavy tests: Run fast schema checks in PRs and heavier distribution comparisons in scheduled validation jobs.

Example run-time SLA policy

Sample SLA tiers you can enforce with this pipeline:

  • Developer PRs: Block on critical schema and non-null checks; cap total CI time to 10 minutes.
  • Pre-merge (optional): Run extended checks; must pass within 60 minutes for automated merges.
  • Production monitoring: Nightly full-suite validation; SLA to detect and alert on breaking failures within 30 minutes of run completion.

Case study (concise)

A 2025 fintech analytics team migrated to lineage-driven testing. Their repository grew to 1,200 models and 4,500 expectations. After implementing the pattern above they observed:

  • Average PR validation time dropped from 42 minutes to 4.5 minutes.
  • False positive failures (unrelated test failures) reduced by 78%.
  • Monthly validation compute cost dropped by ~55%.

Key improvements were disciplined test tagging, canary checks for heavy tables, and strict mapping between model IDs and expectation suite names.

Next steps and enhancements

Once the baseline is stable, consider these incremental improvements:

  • Integrate with observability platforms (e.g., a BI dashboard or Data Observability product) for trend analysis of expectation failures.
  • Use model-level risk scores to expand downstream expansion dynamically (high-risk models trigger wider test runs).
  • Automatically open remedial tickets for high-severity failing expectations with contextual sample rows and last-green commit hashes.

Conclusion

Lineage-driven incremental testing delivers faster feedback, lower cost, and better signal for data engineering teams. By combining dbt’s manifest lineage with Great Expectations’ flexible, programmatic validations, you can create a staged, enforceable, and operable testing pipeline that scales with your repo. Start small—implement PR-level checks for critical models—and iterate toward scheduled full-suite validations with robust reporting and remediation.

Implementing the pattern takes a few engineering sprints but pays back immediately in developer velocity and production reliability. If you want, I can provide a reference repository with a starter script that glues dbt manifest parsing to Great Expectations Checkpoints for GitHub Actions—ask and I’ll assemble it.