Monitoring and alerting within a data pipeline
It's time to put it all together! You might have guessed it, but using handling errors using try-except and logging go hand-in-hand. These two practices are essential for a pipeline to be resilient and transparent, and are the building blocks for more advanced monitoring and alerting solutions.
pandas has been imported as pd, and the logging module has been loaded and configured for you. The raw_sales_data DataFrame has been extracted, and is ready to be transformed.
This exercise is part of the course
ETL and ELT in Python
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
def transform(raw_data):
return raw_data.loc[raw_data["Total Price"] > 1000, :]
try:
# Attempt to transform DataFrame, log an info-level message
clean_sales_data = transform(raw_sales_data)
logging.____("Successfully filtered DataFrame by 'Total Price'")
except Exception:
# Log a warning-level message
____.____("Cannot filter DataFrame by 'Total Price'")