Learn how to ensure data quality with advanced testing techniques in dbt. Explore built-in, singular, and reusable tests to validate models, sources, and seeds. Understand how to define custom tests using Jinja, troubleshoot failures, and optimize your validation workflow to catch inconsistencies before they impact downstream analysis.

Introduction to testing in dbt

Built-in tests

Defining tests on a model

Finding bad data

Creating singular tests

Steps to develop a singular test

Verifying trip duration

Verifying test queries

Creating custom reusable tests

Testing, testing, testing

Implementing a reusable test

Updating from singular to reusable test

Testing & Documentation

Take your dbt skills to the next level by implementing scalable, production-ready workflows. Learn how to use dbt sources and seeds to improve data lineage, implement snapshots for tracking historical changes, and automate your transformations with dbt build. By the end, you’ll be equipped to manage large-scale data pipelines with confidence.

dbt sources

Opsi

Hanya dbt models

Umum untuk models dan sources

Hanya dbt sources

Model, sumber, atau keduanya?

nyc_yellow_taxi:
  outputs:
    dev:
      type: duckdb
      path: dbt.duckdb
  target: dev



# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'taxi_project'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view


{{ config(materialized='view')}}

select * from ____
        


version: 2

sources:
  - name: ____
    tables:
      - name: taxi_rides


#!/usr/bin/env python3
import duckdb
con = duckdb.connect('dbt.duckdb', read_only=True)
print(con.sql('select * from taxi_rides_raw'))
print(con.sql('select count(*) from taxi_rides_raw'))
if (con.execute('select count(*) from taxi_rides_raw').fetchall()[0][0] == 300000):
  with open('/home/repl/workspace/successful_data_check', 'w') as f:
    f.write('300000')


Run the following commands in the terminal:
  
dbt run
datacheck

{{ config(materialized='view')}}

select * from {{ source('raw', 'taxi_rides') }}
        


version: 2

sources:
  - name: raw
    tables:
      - name: taxi_rides


Run the following commands in the terminal:
  
dbt run
./datacheck

Menambahkan source

dbt seeds

Butir kebenaran

{{ config(materialized='view')}}


with source_data as (
    select * from read_parquet('yellow_tripdata_2023-01-partial.parquet')
)

select * from source_data


version: 2

seeds:
  - name: ____
    config:
      column_types:
        ____: ____(5)


#!/usr/bin/env python3
import duckdb
import os.path

try:
  con = duckdb.connect('dbt.duckdb', read_only=True)
  print(con.sql('select * from nynj_zipcodes'))
  if (con.execute('select count(*) as total_zipcodes from nynj_zipcodes').fetchall()[0][0] == 2877):
    print("Your data counts are correct!")
    with open('/home/repl/workspace/successful_data_check', 'w') as f:
      f.write('2877')
  else:
    print("There appears to be an issue with your data counts.")
except duckdb.CatalogException:
  if not os.path.isfile('/home/repl/workspace/nyc_yellow_taxi/seeds/nynj_zipcodes.csv'):
    print("It looks like the nynj_zipcodes.csv file is\n not present in the seeds/ directory.\n\nPlease move the file and try again.\n")
  else:
    print("It looks like your data warehouse doesn't exist yet\n\n Please run the dbt seed command and try again.\n")

version: 2

seeds:
  - name: nynj_zipcodes
    config:
      column_types:
        zipcode: varchar(5)


#!/usr/bin/env python3
import duckdb
con = duckdb.connect('dbt.duckdb', read_only=True)
print(con.sql('select * from nynj_zipcodes'))
if (con.execute('select * as total_zipcodes from nynj_zipcodes').fetchall()[0][1] == 2877):
  with open('/home/repl/workspace/successful_data_check', 'w') as f:
    f.write('2877')


- Run the following command in the terminal window (or drag and drop the nynj_zipcodes.csv file into the seeds folder using the file explorer window.)

mv nynj_zipcodes.csv seeds/

- Modify the properties.yml file as described in the solution.

- Run dbt seed in the terminal window.

- Run the command ./datacheck in the terminal window and verify you see zipcode information in the printed table.

ZIP adalah kodenya

SCD2 dengan dbt snapshots lama

Proses pembuatan snapshot

Proses snapshot

Masalah snapshot

version: 2

sources:
  - name: raw
    tables:
      - name: taxi_rides
      - name: vehicle_list


____

{{
    config(
      target_schema='main',
      unique_key='____',
      strategy='timestamp',
      updated_at='____'
    )
}}

select * from ____
{% endsnapshot %}


{% snapshot vehicle_list_snapshot %}

{{
    config(
      target_schema='main',
      unique_key='license_id',
      strategy='timestamp',
      updated_at='last_updated'
    )
}}

select * from {{ source('raw', 'vehicle_list') }}
{% endsnapshot %}


- Edit the snapshot_vehicle_list.sql file as shown
- Run the command dbt snapshot

Menambahkan snapshot

Otomatisasi dengan dbt build

dbt build mengeksekusi langkah ini

dbt build tidak menjalankan ini

Apa yang tidak dapat dilakukan oleh dbt build?


# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'nyc_yellow_taxi'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view


{{ config(materialized='view')}}

select * from 'taxi_rides'
        


{% snapshot vehicle_list_snapshot %}

{{
    config(
      target_schema='main',
      unique_key='license_id',
      strategy='timestamp',
      updated_at='last_updated'
    )
}}

select * from {{ source('new', 'vehicle_list') }}
{% endsnapshot %}


with avg_daily_fare as (
  select date_part('day', tpep_pickup_datetime) as day,
         avg(total_amount) as avg_amount
  from {{ ref('taxi_rides_raw') }}
  group by day
  order by day
)
select * from avg_daily_fare


version: 2

models:
  - name: taxi_rides_raw
    description: Initial import of the NYC Yellow Taxi trip data from Parquet source
    access: public
  - name: avg_ride_cost_per_day
    description: Avg amount spent per ride, per day
    access: public

sources:
  - name: raw
    tables:
      - name: taxi_rides
      - name: vehicle_list


version: 2

models:
  - name: taxi_rides_raw
    description: Initial import of the NYC Yellow Taxi trip data from Parquet source
    access: public
  - name: avg_ride_cost_per_day
    description: Avg amount spent per ride, per day
    access: public
sources:
  - name: raw
    tables:
      - name: taxi_rides
      - name: vehicle_list


- Run dbt build and look for errors. You should see an issue in the snapshots/vehicle_list_snapshot.sql.

- Edit this file as shown.

- Run dbt build again. You should see errors relating to the models/taxi_rides_raw.sql file.

- Edit this file as shown.

- Run dbt build once more and you should not see any errors.

Membantu anak magang!

Run the following commands in the terminal:

dbt build
dbt docs generate
dbt docs serve

Menggabungkan semuanya

Implementing dbt in production

Tingkatkan keterampilan dbt Anda dengan kursus praktis ini yang dirancang untuk data engineer dan profesional analitik. Anda akan mempelajari fitur dbt tingkat lanjut, termasuk strategi pengujian kustom, tes yang dapat digunakan kembali, sources, seeds, dan snapshots untuk melacak perubahan data historis. Pelajari cara mengotomatiskan pipeline dbt menggunakan dbt build, sehingga transformasi di lingkungan produksi menjadi andal dan efisien. Melalui latihan interaktif dan skenario dunia nyata, Anda akan memperoleh kepercayaan diri untuk menerapkan dbt dalam alur kerja data skala besar.

<h2>Menguasai Alur Kerja DBT</h2>
<p>Tingkatkan keterampilan DBT Anda ke level berikutnya dan pelajari cara mengimplementasikan alur kerja transformasi data yang andal dan skalabel di lingkungan produksi. Kursus ini dirancang untuk insinyur data, analis, dan insinyur analitik yang ingin melampaui dasar-dasar dan memperoleh pengalaman praktis dengan fitur-fitur lanjutan dbt.</p>

<h2>Pastikan Kualitas Data dengan Pengujian Lanjutan</h2>
<p>Membangun saluran data yang andal dimulai dengan validasi. Anda akan mempelajari teknik pengujian lanjutan untuk mendeteksi ketidakkonsistenan data, membuat tes kustom yang dapat digunakan ulang untuk mengstandarkan validasi di seluruh model, dan menerapkan tes pada sumber data dan benih data untuk meningkatkan tata kelola dan pelacakan jejak data.</p>

<h2>Manfaatkan Sumber, Benih, dan Snapshot dbt</h2>
<p>Temukan bagaimana sumber dbt dapat meningkatkan dokumentasi dan jejak data sambil memastikan jejak data mentah. Pelajari cara menggunakan dbt seeds untuk mengelola dataset kecil dan statis secara efisien. Kemudian, gunakan Master Slowly Changing Dimensions (SCD2) dengan snapshot dbt, yang memungkinkan Anda melacak perubahan historis dalam data warehouse Anda dengan usaha minimal.</p>

<h2>Otomatisasi dan Optimalisasi dengan dbt Build</h2>
<p>Efisiensi merupakan hal yang paling penting dalam lingkungan produksi. Anda akan belajar cara mengoptimalkan alur kerja dengan dbt build, mengotomatisasi eksekusi model, pengujian, dan snapshot untuk memastikan transformasi yang andal. Dengan mengoptimalkan pipeline Anda, Anda akan meningkatkan kinerja, kemudahan pemeliharaan, dan skalabilitas proyek dbt Anda.</p>

<h2>Terapkan Keterampilan Anda dalam Situasi Nyata</h2>
<p>Melalui latihan interaktif dan praktik langsung, Anda akan memperkuat pengetahuan Anda dan memperoleh kepercayaan diri untuk menerapkan DBT dalam situasi nyata. Pada akhir kursus ini, Anda akan mampu merancang, menguji, dan mengotomatisasi alur kerja dbt yang siap produksi, memastikan transformasi berkualitas tinggi dan terdokumentasi dengan baik pada skala besar.</p>


Introduction to dbt

Tingkatkan keterampilan dbt Anda dengan kursus praktis ini untuk insinyur data dan profesional analitik.

dbt Tingkat Menengah

Tingkatkan keterampilan DBT Anda ke level berikutnya dengan kursus praktis ini yang dirancang khusus untuk insinyur data dan profesional analitik.

Menggabungkan semuanya

dbt Tingkat Menengah

Latihan interaktif praktis