Monitoring and alerting within a data pipeline
It's time to put it all together! You might have guessed it, but using handling errors using try-except and logging go hand-in-hand. These two practices are essential for a pipeline to be resilient and transparent, and are the building blocks for more advanced monitoring and alerting solutions.
pandas has been imported as pd, and the logging module has been loaded and configured for you. The raw_sales_data DataFrame has been extracted, and is ready to be transformed.
Deze oefening maakt deel uit van de cursus
ETL and ELT in Python
Praktische interactieve oefening
Probeer deze oefening eens door deze voorbeeldcode in te vullen.
def transform(raw_data):
return raw_data.loc[raw_data["Total Price"] > 1000, :]
try:
# Attempt to transform DataFrame, log an info-level message
clean_sales_data = transform(raw_sales_data)
logging.____("Successfully filtered DataFrame by 'Total Price'")
except Exception:
# Log a warning-level message
____.____("Cannot filter DataFrame by 'Total Price'")