Calculating a threshold and moving averages

You'll track forecast accuracy over time to detect model drift. By calculating rolling windows and defining a threshold level for triggering drift alerts, you can identify when the model becomes misaligned with reality and requires retraining.

You'll use the first 14 forecasts from fc_log_test to establish the threshold, then apply it to the remaining forecast logs. The forecast logs fc_log_test and fc_log containing model performance scores have been preloaded, along with pandas as pd.

This exercise is part of the course

Designing Forecasting Pipelines for Production

View Course

Exercise instructions

Define the threshold level from fc_log_test by adding three standard deviations to the RMSE mean, storing as rmse_threshold.
Calculate the RMSE moving average using 7-day and 14-day rolling windows for fc_log.
Print the threshold and rolling averages values.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

# Set threshold: mean + 3 standard deviations
rmse_threshold = fc_log_test["____"].____() + 3 * fc_log_test["____"].____()

# Create rolling window averages for RMSE
fc_log["rmse_ma_7"] = fc_log["____"].rolling(window=____).____()
fc_log["rmse_ma_14"] = fc_log["____"].rolling(window=____).____()

# Print threshold and rolling averages
print(f"RMSE threshold: {round(rmse_threshold, 2)}")
print()
print("Forecast log with rolling averages:")
print(fc_log[["forecast_start", "rmse", "rmse_ma_7", "rmse_ma_14"]].head(20))

Edit and Run Code