1. Learn
    2. /
  2. Courses
    3. /
  3. ETL and ELT in Python
  • 1

    Introduction to Data Pipelines

    Free

    Get ready to discover how data is collected, processed, and moved using data pipelines. You will explore the qualities of the best data pipelines, and prepare to design and build your own.

  • 2

    Building ETL Pipelines

    Dive into leveraging pandas to extract, transform, and load data as you build your first data pipelines. Learn how to make your ETL logic reusable, and apply logging and exception handling to your pipelines.

  • 3

    Advanced ETL Techniques

    Supercharge your workflow with advanced data pipelining techniques, such as working with non-tabular data and persisting DataFrames to SQL databases. Discover tooling to tackle advanced transformations with pandas, and uncover best-practices for working with complex data.

  • 4

    Deploying and Maintaining a Data Pipeline

    In this final chapter, you’ll create frameworks to validate and test data pipelines before shipping them into production. After you’ve tested your pipeline, you’ll explore techniques to run your data pipeline end-to-end, all while allowing for visibility into pipeline performance.

Initializing

Exercise

Testing a data pipeline end-to-end

In this exercise, you'll be working with the same data pipeline as before, which extracts, transforms, and loads tax data. You'll practice testing this pipeline end-to-end to ensure the solution can be run multiple times, without duplicating the transformed data in the parquet file.

pandas has been loaded as pd, and the extract(), transform(), and load() functions have already been defined.

Instructions

100 XP
  • Run the ETL pipeline three times, using a for-loop.
  • Print the shape of the clean_tax_data in each iteration of the pipeline run.
  • Read the DataFrame stored in the "clean_tax_data.parquet" file into the to_validate variable.
  • Output the shape of the to_validate DataFrame, comparing it to the shape of clean_tax_rate to ensure data wasn't duplicated upon each pipeline run.