Get ready to discover how data is collected, processed, and moved using data pipelines. You will explore the qualities of the best data pipelines, and prepare to design and build your own.

Introduction to ETL and ELT Pipelines

Running an ETL Pipeline

ELT in Action

ETL and ELT Pipelines

Building ETL and ELT Pipelines

Building an ETL Pipeline

The "T" in ELT

Extracting, Transforming, and Loading Student Scores Data

Introduction to Data Pipelines

Dive into leveraging pandas to extract, transform, and load data as you build your first data pipelines. Learn how to make your ETL logic reusable, and apply logging and exception handling to your pipelines.

Extracting data from structured sources

Extracting data from parquet files

Pulling data from SQL databases

Building functions to extract data

Transforming data with pandas

Filtering pandas DataFrames

Transforming sales data with pandas

Validating data transformations

Persisting data with pandas

Loading sales data to a CSV file

Customizing a CSV file

Persisting data to files

Monitoring a data pipeline

Logging within a data pipeline

Handling exceptions when loading data

Monitoring and alerting within a data pipeline

Building ETL Pipelines

Supercharge your workflow with advanced data pipelining techniques, such as working with non-tabular data and persisting DataFrames to SQL databases. Discover tooling to tackle advanced transformations with pandas, and uncover best-practices for working with complex data.

Extracting non-tabular data

Ingesting JSON data with pandas

Reading JSON data into memory

Transforming non-tabular data

Iterating over dictionaries

Parsing data from dictionaries

Transforming JSON data

Transforming and cleaning DataFrames

Advanced data transformation with pandas

Filling missing values with pandas

Grouping data with pandas

Applying advanced transformations to DataFrames

Loading data to a SQL database with pandas

Loading data to a Postgres database

Validating data loaded to a Postgres Database

Advanced ETL Techniques

In this final chapter, you’ll create frameworks to validate and test data pipelines before shipping them into production. After you’ve tested your pipeline, you’ll explore techniques to run your data pipeline end-to-end, all while allowing for visibility into pipeline performance.

Menguji pipeline data secara manual

Menguji data pipeline

Memvalidasi pipeline data pada "checkpoint"

Menguji pipeline data secara end-to-end

Pengujian unit untuk sebuah pipeline data

Memvalidasi data pipeline dengan assert

Menulis unit test dengan pytest

Membuat fixture dengan pytest

Pengujian unit untuk data pipeline dengan fixtures

Menjalankan pipeline data di produksi

Alat orkestrasi dan ETL

Pola arsitektur data pipeline

Menjalankan pipeline data end-to-end

Selamat!

Deploying and Maintaining a Data Pipeline

scores.csv

schools_modified.csv

amazon_sales_cleaned_sql.csv

tax_rate_cleaned.csv

Course Glossary

Pipa data adalah fondasi dari setiap platform data yang kuat. Membangun pipa ini merupakan keterampilan penting bagi data engineer, yang memberikan nilai luar biasa bagi bisnis yang siap melangkah ke masa depan berbasis data. Kursus pengantar ini akan membantu Anda mengasah keterampilan untuk membangun pipa data yang efektif, berkinerja baik, dan andal.

Video berisi transkrip langsung yang dapat Anda tampilkan dengan mengeklik "Show transcript" di kiri bawah video.
Glosarium kursus dapat ditemukan di sebelah kanan pada bagian sumber daya.
Untuk memperoleh kredit CPE Anda harus menyelesaikan kursus dan mencapai skor 70% pada penilaian yang memenuhi syarat. Anda dapat membuka penilaian dengan mengeklik sorotan kredit CPE di sebelah kanan.

Data Warehousing Concepts

Streamlined Data Ingestion with pandas

Pelajari cara membangun pipeline data yang efektif, cepat, dan andal dengan prinsip ETL.

ETL dan ELT di Python

Data Engineer dalam Python

Insinyur Pembelajaran Mesin

Data Engineer

Pola arsitektur pipeline data

Mengorkestrasi pipeline data di produksi

Ayo berlatih!

Deploying and Maintaining a Data Pipeline - Menjalankan pipeline data di produksi

Menjalankan pipeline data di produksi

Create Your Free Account