Learn how to ensure data quality with advanced testing techniques in dbt. Explore built-in, singular, and reusable tests to validate models, sources, and seeds. Understand how to define custom tests using Jinja, troubleshoot failures, and optimize your validation workflow to catch inconsistencies before they impact downstream analysis.

Introduction to testing in dbt

Built-in tests

Defining tests on a model

Finding bad data

Creating singular tests

Steps to develop a singular test

Verifying trip duration

Verifying test queries

Creating custom reusable tests

Testing, testing, testing

Implementing a reusable test

Updating from singular to reusable test

Testing & Documentation

Take your dbt skills to the next level by implementing scalable, production-ready workflows. Learn how to use dbt sources and seeds to improve data lineage, implement snapshots for tracking historical changes, and automate your transformations with dbt build. By the end, you’ll be equipped to manage large-scale data pipelines with confidence.

dbt-sources

Opties

Alleen dbt-modellen

Gemeen aan modellen en sources

Alleen dbt-sources

Modellen, sources, of allebei?

nyc_yellow_taxi:
  outputs:
    dev:
      type: duckdb
      path: dbt.duckdb
  target: dev



# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'taxi_project'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view


{{ config(materialized='view')}}

select * from ____
        


version: 2

sources:
  - name: ____
    tables:
      - name: taxi_rides


#!/usr/bin/env python3
import duckdb
con = duckdb.connect('dbt.duckdb', read_only=True)
print(con.sql('select * from taxi_rides_raw'))
print(con.sql('select count(*) from taxi_rides_raw'))
if (con.execute('select count(*) from taxi_rides_raw').fetchall()[0][0] == 300000):
  with open('/home/repl/workspace/successful_data_check', 'w') as f:
    f.write('300000')


Run the following commands in the terminal:
  
dbt run
datacheck

{{ config(materialized='view')}}

select * from {{ source('raw', 'taxi_rides') }}
        


version: 2

sources:
  - name: raw
    tables:
      - name: taxi_rides


Run the following commands in the terminal:
  
dbt run
./datacheck

Een source toevoegen

dbt seeds

Korrels waarheid

{{ config(materialized='view')}}


with source_data as (
    select * from read_parquet('yellow_tripdata_2023-01-partial.parquet')
)

select * from source_data


version: 2

seeds:
  - name: ____
    config:
      column_types:
        ____: ____(5)


#!/usr/bin/env python3
import duckdb
import os.path

try:
  con = duckdb.connect('dbt.duckdb', read_only=True)
  print(con.sql('select * from nynj_zipcodes'))
  if (con.execute('select count(*) as total_zipcodes from nynj_zipcodes').fetchall()[0][0] == 2877):
    print("Your data counts are correct!")
    with open('/home/repl/workspace/successful_data_check', 'w') as f:
      f.write('2877')
  else:
    print("There appears to be an issue with your data counts.")
except duckdb.CatalogException:
  if not os.path.isfile('/home/repl/workspace/nyc_yellow_taxi/seeds/nynj_zipcodes.csv'):
    print("It looks like the nynj_zipcodes.csv file is\n not present in the seeds/ directory.\n\nPlease move the file and try again.\n")
  else:
    print("It looks like your data warehouse doesn't exist yet\n\n Please run the dbt seed command and try again.\n")

version: 2

seeds:
  - name: nynj_zipcodes
    config:
      column_types:
        zipcode: varchar(5)


#!/usr/bin/env python3
import duckdb
con = duckdb.connect('dbt.duckdb', read_only=True)
print(con.sql('select * from nynj_zipcodes'))
if (con.execute('select * as total_zipcodes from nynj_zipcodes').fetchall()[0][1] == 2877):
  with open('/home/repl/workspace/successful_data_check', 'w') as f:
    f.write('2877')


- Run the following command in the terminal window (or drag and drop the nynj_zipcodes.csv file into the seeds folder using the file explorer window.)

mv nynj_zipcodes.csv seeds/

- Modify the properties.yml file as described in the solution.

- Run dbt seed in the terminal window.

- Run the command ./datacheck in the terminal window and verify you see zipcode information in the printed table.

ZIP is de code

SCD2 met dbt snapshots (oud)

Proces voor het maken van een snapshot

Snapshot-proces

Snapshotprobleem

version: 2

sources:
  - name: raw
    tables:
      - name: taxi_rides
      - name: vehicle_list


____

{{
    config(
      target_schema='main',
      unique_key='____',
      strategy='timestamp',
      updated_at='____'
    )
}}

select * from ____
{% endsnapshot %}


{% snapshot vehicle_list_snapshot %}

{{
    config(
      target_schema='main',
      unique_key='license_id',
      strategy='timestamp',
      updated_at='last_updated'
    )
}}

select * from {{ source('raw', 'vehicle_list') }}
{% endsnapshot %}


- Edit the snapshot_vehicle_list.sql file as shown
- Run the command dbt snapshot

Een snapshot toevoegen

Automatiseren met dbt build

dbt build voert deze stap uit

dbt build voert dit niet uit

Wat kan dbt build niet doen?


# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'nyc_yellow_taxi'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view


{{ config(materialized='view')}}

select * from 'taxi_rides'
        


{% snapshot vehicle_list_snapshot %}

{{
    config(
      target_schema='main',
      unique_key='license_id',
      strategy='timestamp',
      updated_at='last_updated'
    )
}}

select * from {{ source('new', 'vehicle_list') }}
{% endsnapshot %}


with avg_daily_fare as (
  select date_part('day', tpep_pickup_datetime) as day,
         avg(total_amount) as avg_amount
  from {{ ref('taxi_rides_raw') }}
  group by day
  order by day
)
select * from avg_daily_fare


version: 2

models:
  - name: taxi_rides_raw
    description: Initial import of the NYC Yellow Taxi trip data from Parquet source
    access: public
  - name: avg_ride_cost_per_day
    description: Avg amount spent per ride, per day
    access: public

sources:
  - name: raw
    tables:
      - name: taxi_rides
      - name: vehicle_list


version: 2

models:
  - name: taxi_rides_raw
    description: Initial import of the NYC Yellow Taxi trip data from Parquet source
    access: public
  - name: avg_ride_cost_per_day
    description: Avg amount spent per ride, per day
    access: public
sources:
  - name: raw
    tables:
      - name: taxi_rides
      - name: vehicle_list


- Run dbt build and look for errors. You should see an issue in the snapshots/vehicle_list_snapshot.sql.

- Edit this file as shown.

- Run dbt build again. You should see errors relating to the models/taxi_rides_raw.sql file.

- Edit this file as shown.

- Run dbt build once more and you should not see any errors.

De stagiair helpen!

Run the following commands in the terminal:

dbt build
dbt docs generate
dbt docs serve

Alles samenbrengen

Implementing dbt in production

Breng je dbt-vaardigheden naar een hoger niveau met deze praktijkgerichte cursus voor data engineers en analytics-professionals. Je verkent geavanceerde dbt-functies, waaronder aangepaste teststrategieën, herbruikbare tests, sources, seeds en snapshots om historische gegevenswijzigingen te volgen. Leer hoe je dbt-pijplijnen automatiseert met dbt build, zodat transformaties in productieomgevingen betrouwbaar en efficiënt verlopen. Met interactieve oefeningen en realistische scenario’s krijg je het vertrouwen om dbt toe te passen in grootschalige dataworkflows.

<h2>Word een expert in dbt-workflows</h2>
<p>Til je DBT-vaardigheden naar een hoger niveau en leer hoe je robuuste, schaalbare workflows voor datatransformatie kunt implementeren in een productieomgeving. Deze cursus is voor data-engineers, analisten en analytics-engineers die verder willen gaan dan de basis en praktische ervaring willen opdoen met geavanceerde dbt-functies.</p>

<h2>Zorg voor datakwaliteit met geavanceerde tests</h2>
<p>Het bouwen van betrouwbare datapijplijnen begint met validatie. Je gaat geavanceerde testtechnieken ontdekken om inconsistenties in gegevens op te sporen, herbruikbare tests op maat maken om validatie tussen modellen te standaardiseren, en tests toepassen op bronnen en seeds voor beter beheer en het volgen van gegevensherkomst.</p>

<h2>Maak gebruik van dbt-bronnen, seeds en snapshots</h2>
<p>Ontdek hoe dbt-bronnen je documentatie en afkomst kunnen verbeteren en tegelijkertijd de traceerbaarheid van ruwe gegevens kunnen garanderen. Leer hoe je dbt seeds kunt gebruiken om kleine, statische datasets efficiënt te beheren. Leer dan langzaam dimensies aanpassen (SCD2) met dbt-snapshots, zodat je met weinig moeite historische veranderingen in je datawarehouse kunt volgen.</p>

<h2>Automatiseer en optimaliseer met dbt Build</h2>
<p>Efficiëntie is superbelangrijk in productieomgevingen. Je leert hoe je workflows kunt stroomlijnen met dbt build, door het uitvoeren van modellen, tests en snapshots te automatiseren om betrouwbare transformaties te garanderen. Door je pijplijn te optimaliseren, verbeter je de prestaties, onderhoudbaarheid en schaalbaarheid van je dbt-projecten.</p>

<h2>Gebruik je vaardigheden in echte situaties</h2>
<p>Door interactieve oefeningen en praktijkervaring versterk je je kennis en krijg je het zelfvertrouwen om dbt in de praktijk toe te passen. Aan het einde van de cursus kun je productieklaar dbt-workflows ontwerpen, testen en automatiseren, zodat je zeker weet dat je hoogwaardige en goed gedocumenteerde transformaties op grote schaal kunt doen.</p>


Introduction to dbt

Verhoog je dbt-vaardigheden met deze praktijkcursus voor data-engineers en analytics-professionals.

Gevorderde dbt

Trek je DBT-vaardigheden naar een hoger niveau met deze praktische cursus, speciaal gemaakt voor data-engineers en analytics-professionals.

dbt-basisprincipes

Kan worden gedefinieerd in `models/model_properties.yml`

Aangeroepen via `{{ source(source_name, table_name) }}`

Modellen, sources, of allebei?

Gevorderde dbt

Praktische interactieve oefening