Testing a data pipeline end-to-end

In this exercise, you'll be working with the same data pipeline as before, which extracts, transforms, and loads tax data. You'll practice testing this pipeline end-to-end to ensure the solution can be run multiple times, without duplicating the transformed data in the parquet file.

pandas has been loaded as pd, and the extract(), transform(), and load() functions have already been defined.

Questo esercizio fa parte del corso

ETL and ELT in Python

Visualizza il corso

Istruzioni dell'esercizio

Run the ETL pipeline three times, using a for-loop.
Print the shape of the clean_tax_data in each iteration of the pipeline run.
Read the DataFrame stored in the "clean_tax_data.parquet" file into the to_validate variable.
Output the shape of the to_validate DataFrame, comparing it to the shape of clean_tax_rate to ensure data wasn't duplicated upon each pipeline run.

Esercizio pratico interattivo

Prova a risolvere questo esercizio completando il codice di esempio.

# Trigger the data pipeline to run three times
____ attempt in range(0, ____):
	print(f"Attempt: {attempt}")
	raw_tax_data = extract("raw_tax_data.csv")
	clean_tax_data = transform(raw_tax_data)
	load(clean_tax_data, "clean_tax_data.parquet")
	
	# Print the shape of the cleaned_tax_data DataFrame
	print(f"Shape of clean_tax_data: {clean_tax_data.____}")
    
# Read in the loaded data, check the shape
to_validate = pd.____("clean_tax_data.parquet")
print(f"Final shape of cleaned data: {to_validate.____}")

Modifica ed esegui il codice