1. Learn
    2. /
  2. Courses
    3. /
  3. Designing Forecasting Pipelines for Production
Connected

Exercise

Forecast evaluation & experimentation

In this exercise, you'll evaluate the forecast model's performance to explore the use cases of experimentation.

The merged forecast (fc), combining predictions with actual test results, is preloaded. Evaluation functions (mape, rmse, coverage) and pandas (as pd) are also ready for use. Here's a quick reference for the functions:

def mape(y, yhat):
    mape = mean(abs(y - yhat) / y) 
    return mape

def rmse(y, yhat):
    rmse = (mean((y - yhat) ** 2)) ** 0.5
    return rmse

def coverage(y, lower, upper):
    coverage = sum((y <= upper) & (y >= lower)) / len(y)
    return coverage

First, compute performance metrics for the model. Then, answer a question about the goals of experimentation in forecasting.

Instructions 1/2

undefined XP
    1
    2
  • Calculate mape, rmse, and coverage metrics for each model in the fc DataFrame.
  • Convert the list into a DataFrame named fc_performance, then sort the values by rmse.