Learn how to ensure data quality with advanced testing techniques in dbt. Explore built-in, singular, and reusable tests to validate models, sources, and seeds. Understand how to define custom tests using Jinja, troubleshoot failures, and optimize your validation workflow to catch inconsistencies before they impact downstream analysis.

Introduction to testing in dbt

Options

Built-in

Not built-in

Built-in tests

nyc_yellow_taxi:
  outputs:
    dev:
      type: duckdb
      path: dbt.duckdb
  target: dev



# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'taxi_project'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view


{{ config(materialized='view')}}


with source_data as (
    select * from read_parquet('yellow_tripdata_2023-01-partial.parquet')
)

select * from source_data


version: 2

models:
- name: taxi_rides_raw
  columns:
    - name: fare_amount
      tests:
        - not_null
    # Enter the name of the column
    - name: ____
    # Define the two tests
      tests:
        - ____
        - ____
            ____: [1, 2, 3, ____]


Run the following commands in the terminal:
  
dbt run
dbt test

version: 2

models:
- name: taxi_rides_raw
  columns:
    - name: fare_amount
      tests:
        - not_null
    # Enter the name of the column
    - name: payment_type
    # Define the two tests
      tests:
        - not_null
        - accepted_values:
            values: [1, 2, 3, 4, 5, 6] 


Defining tests on a model

{{ config(materialized='view')}}


with source_data as (
    select * from read_parquet('yellow_tripdata_2023-01-partial.parquet')
)

-- Add a filter / WHERE clause to the line below
select * from source_data ____


version: 2

models:
- name: taxi_rides_raw
  columns:
    - name: fare_amount
      tests:
        - not_null
    - name: payment_type
      tests:
        - not_null
        - accepted_values:
            values: [1, 2, 3, 4, 5, 6]


Run the following commands in the terminal:
  
dbt run

{{ config(materialized='view')}}


with source_data as (
    select * from read_parquet('yellow_tripdata_2023-01-partial.parquet')
)

-- Add a filter / WHERE clause to the line below
select * from source_data where payment_type != 0


Run the following command in the terminal:
  
dbt run

Finding bad data

Creating singular tests

Singular test development process

Steps to develop a singular test

select * 
from taxi_rides_raw
-- Complete the test on the following line


Run the following command in the terminal:
  
dbt test --select assert_trip_duration_gt_0.sql

select * 
from taxi_rides_raw
-- Complete the test on the following line
where tpep_pickup_datetime = tpep_dropoff_datetime


The appropriate filter has been added to assert_trip_duration_gt_0.sql

Run the following command in the terminal:
  
dbt test --select assert_trip_duration_gt_0.sql

Verifying trip duration

Valid test query

Invalid test query

Verifying test queries

Creating custom reusable tests

Reusable

Singular

Testing, testing, testing

version: 2

# Replace any entry of ____ with the appropriate content
models:
- name: taxi_rides_raw
  columns:
    - name: fare_amount
      tests:
        - ____
    - name: ____
      tests:
        - check_gt_0
        


{% test ____(model, column_name) %}
select * 
from {{ model }}
where {{ column_name }} <= 0
{% endtest %}



version: 2

models:
- name: taxi_rides_raw
  columns:
    - name: fare_amount
      tests:
        - check_gt_0
    - name: total_amount
      tests:
        - check_gt_0


{% test check_gt_0(model, column_name) %}
select * 
from {{ model }}
where {{ column_name }} <= 0
{% endtest %}


- Update the check_gt_0.sql file as shown in the solution. (Note, the last character is a zero)
- Update the model_properties.yml file as shown.
- Run dbt run in the terminal window
- Run dbt test --select taxi_rides_raw or dbt test --select taxi_rides_raw.sql in the terminal window

Implementing a reusable test

version: 2

models:
- name: taxi_rides_raw
  columns:
    - name: tpep_pickup_datetime
      tests:
        - ____:
            ____: ____


select * 
from taxi_rides_raw
where tpep_pickup_datetime = tpep_dropoff_datetime


{% test ____(model, column_name, column_name2) %}
select *
from {{ model }}
____
{% endtest %}


version: 2

models:
- name: taxi_rides_raw
  columns:
    - name: tpep_pickup_datetime
      tests:
        - columns_equal:
            column_name2: tpep_dropoff_datetime


{% test columns_equal(model, column_name, column_name2) %}
select *
from {{ model }}
where {{ column_name }} = {{ column_name2 }}
{% endtest %}


- Edit the columns_equal.sql file as shown.
- Edit the model_properties.yml file as shown.
- Run dbt run in the terminal.
- Run dbt test --select taxi_rides_raw.sql or dbt test --select taxi_rides_raw in the terminal.

Updating from singular to reusable test

Testing & Documentation

Take your dbt skills to the next level by implementing scalable, production-ready workflows. Learn how to use dbt sources and seeds to improve data lineage, implement snapshots for tracking historical changes, and automate your transformations with dbt build. By the end, you’ll be equipped to manage large-scale data pipelines with confidence.

dbt sources

Models, sources, or both?

Adding a source

dbt seeds

Kernels of truth

ZIP is the code

SCD2 with dbt snapshots old

Snapshot process

Snapshot issue

Adding a snapshot

Automating with dbt build

What can't dbt build do?

Helping the intern!

Putting it all together

Implementing dbt in production

Take your dbt skills to the next level with this hands-on course designed for data engineers and analytics professionals. You’ll explore advanced dbt features, including custom testing strategies, reusable tests, sources, seeds, and snapshots to track historical data changes. Learn how to automate dbt pipelines using dbt build, ensuring reliable and efficient transformations in production environments. With interactive exercises and real-world scenarios, you’ll gain the confidence to implement dbt in large-scale data workflows.

<h2>Master dbt Workflows</h2>
<p>Take your dbt skills to the next level and learn how to implement robust, scalable data transformation workflows in a production environment. This course is designed for data engineers, analysts, and analytics engineers who want to move beyond the basics and gain hands-on experience with advanced dbt functionality.</p>

<h2>Ensure Data Quality with Advanced Testing</h2>
<p>Building reliable data pipelines starts with validation. You'll explore advanced testing techniques to catch data inconsistencies, create custom reusable tests to standardize validation across models, and apply tests to sources and seeds for better governance and data lineage tracking.</p>

<h2>Leverage dbt Sources, Seeds, and Snapshots</h2>
<p>Discover how dbt sources can improve documentation and lineage while ensuring traceability of raw data. Learn to use dbt seeds for managing small, static datasets efficiently. Then, master slowly changing dimensions (SCD2) with dbt snapshots, allowing you to track historical changes in your data warehouse with minimal effort.</p>

<h2>Automate and Optimize with dbt Build</h2>
<p>Efficiency is key in production environments. You’ll learn how to streamline workflows with dbt build, automating model execution, tests, and snapshots to ensure reliable transformations. By optimizing your pipeline, you'll enhance performance, maintainability, and scalability of your dbt projects.</p>

<h2>Apply Your Skills in Real-World Scenarios</h2>
<p>Through interactive exercises and hands-on practice, you’ll reinforce your knowledge and gain the confidence to apply dbt in real-world settings. By the end of the course, you'll be equipped to design, test, and automate production-ready dbt workflows, ensuring high-quality and well-documented transformations at scale.</p>


Introduction to dbt

Take your dbt skills to the next level with this hands-on course designed for data engineers and analytics professionals.

Intermediate dbt

dbt Fundamentals

`SELECT * FROM taxi_rides_raw WHERE tpep_pickup_datetime = tpep_dropoff_datetime`

`SELECT * FROM taxi_rides_raw WHERE tpep_dropoff_datetime - tpep_pickup_datetime = 0`

`SELECT * FROM taxi_rides_raw WHERE tpep_dropoff_datetime - tpep_pickup_datetime >= 0`

`SELECT * FROM taxi_rides_raw WHERE tpep_dropoff_datetime > tpep_pickup_datetime`

Verifying test queries

Intermediate dbt

Praktische interactieve oefening