Pelajari alasan keberadaan dbt dan bagaimana dbt digunakan untuk membuat proyek transformasi data.

Dasar-dasar dbt

Menjalankan perintah dbt

Pengenalan dbt

Versi dbt

Mendapatkan bantuan

Pengguna dbt

Membuat proyek dbt

Menginisialisasi proyek dbt

Membuat profil proyek

Selamat datang di dbt

Masuk ke inti dbt dengan bekerja menggunakan model data dan membuat model berbasis SQL di dalam dbt. Pelajari model SQL dasar, tetapkan dependensi, lakukan pembaruan, dan selesaikan berbagai masalah pada model.

Mengerjakan proyek pertama

Langkah-langkah alur kerja dbt

Alur kerja proyek dbt

nyc_yellow_taxi:
  outputs:
    dev:
      type: duckdb
      path: dbt.duckdb
  target: dev



# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'nyc_yellow_taxi'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view
    


with source_data as (
    -- Add the query as described to generate the data model
    ____
)

select * from source_data


#!/usr/bin/env python3
import duckdb
con = duckdb.connect('dbt.duckdb', read_only=True)
print(con.sql('select * from taxi_rides_raw limit 10'))
print(con.sql('select count(*) as total_rides from taxi_rides_raw'))
if (con.execute('select count(*) as total_rides from taxi_rides_raw').fetchall()[0][0] == 300000):
  with open('/home/repl/workspace/successful_data_check', 'w') as f:
    f.write('300000')


Run the following commands in the terminal:
dbt run
./datacheck



# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'nyc_yellow_taxi'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
  nyc_yellow_taxi:
  # Config indicated by + and applies to all files under models/example/
    taxi_rides:
    +materialized: view


with source_data as (
    -- Add the query as described to generate the data model
    select * from read_parquet('yellow_tripdata_2023-01-partial.parquet')
)

select * from source_data


Run the following commands in the terminal:
dbt run
./datacheck   


Menjalankan sebuah proyek

Apa itu model dbt?

Opsi

Benar tentang sebuah model data

Salah tentang sebuah model data

Fitur dari sebuah model data


# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'nyc_yellow_taxi'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'nyc_yellow_taxi'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

target-path: "target"  # directory which will store compiled SQL files
clean-targets:         # directories to be removed by `dbt clean`
  - "target"
  - "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
#models:
#  nyc_yellow_taxi:
#  # Config indicated by + and applies to all files under models/example/
#    taxi_rides:
#    +materialized: view
    


Run the following command in the terminal:

dbt run


select * from 'yellow_tripdata_2023-01-partial.parquet'


Run the following command in the terminal:

cd nyc_yellow_taxi
dbt run

Membuat model dbt

Memperbarui model dbt

Proyek

Hanya model

Berkas konfigurasi


-- Update with SQL to return requested information
select 
  date_part('day', tpep_pickup_datetime) as day,
  ____ as total_cc_riders
from taxi_rides_raw
-- Refer to the Payment_type in schema doc
where payment_type = ____
group by day


#!/usr/bin/env python3
import duckdb
con = duckdb.connect('dbt.duckdb', read_only=True)
print(con.sql('select * from total_creditcard_riders_by_day'))
if (con.execute('select * as total_rides from total_creditcard_riders_by_day').fetchall()[0][1] == 54743):
  with open('/home/repl/workspace/successful_data_check', 'w') as f:
    f.write('54743')
    



-- Update with SQL to return requested information
select 
   date_part('day', tpep_pickup_datetime) as day,
   count(*) as total_cc_riders
from taxi_rides_raw
-- Refer to the Payment_type in schema doc
where payment_type = 1
group by day


Proyek dan model dbt

Pelajari cara menggunakan templat Jinja untuk menyederhanakan model dbt dan mendefinisikan dependensi hierarkis dengan fungsi `ref` untuk memastikan eksekusi model yang benar.

Membuat dan menghasilkan dokumentasi dbt

Opsi Perintah dbt docs

Alur dokumentasi dbt

Membuat dokumentasi dbt

Templat Jinja

Apa yang dapat dilakukan Jinja?

Memproses template Jinja

Model hierarkis di dbt

Pembuatan model hierarkis

Memperbarui hierarki model

Lebih lanjut tentang model dbt

Kursus ini memperkenalkan Anda pada dbt (data build tool), sebuah alat populer yang memunculkan analytics engineering untuk pembuatan dan pemeliharaan pipeline extract, transform, and load (ELT). Anda akan mempelajari cara membangun gudang data, melakukan pemodelan dan transformasi data, serta merancang pengujian untuk melakukan validasi data. Anda akan menggunakan SQL dan perintah shell untuk menjalankan berbagai tugas dengan dbt, sekaligus mempelajari cara menghasilkan dokumentasi bagi pengguna gudang data Anda!

dbt, atau alat pembangunan data, telah mengguncang dunia data. Kursus ini memperkenalkan Anda pada konsep, istilah, dan metode penggunaan dbt untuk mengimplementasikan contoh data warehouse. Anda akan memahami apa itu dbt, kapan sebaiknya digunakan, dan praktik terbaik dalam implementasi data warehousing. Anda akan bekerja dengan dataset nyata dan melakukan operasi ekstraksi, pemuatan, dan transformasi yang diterapkan dalam peran analis data, insinyur data, dan insinyur analitik.

Peserta didik akan mengembangkan keterampilan untuk merancang data warehouse dari awal, memodelkan dan mentransformasi data, serta membangun tes! Kursus ini akan memberikan Anda dasar yang kokoh untuk dikembangkan dalam perjalanan DBT Anda, terlepas dari jenis data warehouse yang Anda rencanakan untuk diimplementasikan. Dapatkan kepercayaan diri dalam memahami cara dan waktu yang tepat untuk menggunakan dbt dengan mengerjakan latihan menggunakan SQL dan perintah shell.

Data Warehousing Concepts

Introduction to Shell

Joining Data in SQL

Kursus ini memperkenalkan dbt untuk pemodelan data, transformasi, pengujian, dan pembuatan dokumentasi.

Pengantar dbt

Insinyur Data Profesional dalam Python

Mengklasifikasikan atribut umum tentang data

Fitur dari sebuah model data

Pengantar dbt

Latihan interaktif langsung