Monitoring and alerting within a data pipeline
It's time to put it all together! You might have guessed it, but using handling errors using try-except and logging go hand-in-hand. These two practices are essential for a pipeline to be resilient and transparent, and are the building blocks for more advanced monitoring and alerting solutions.
pandas has been imported as pd, and the logging module has been loaded and configured for you. The raw_sales_data DataFrame has been extracted, and is ready to be transformed.
Latihan ini adalah bagian dari kursus
ETL and ELT in Python
Latihan interaktif praktis
Cobalah latihan ini dengan menyelesaikan kode contoh berikut.
def transform(raw_data):
return raw_data.loc[raw_data["Total Price"] > 1000, :]
try:
# Attempt to transform DataFrame, log an info-level message
clean_sales_data = transform(raw_sales_data)
logging.____("Successfully filtered DataFrame by 'Total Price'")
except Exception:
# Log a warning-level message
____.____("Cannot filter DataFrame by 'Total Price'")