Evaluating forecast performance

In this exercise, you'll evaluate and visualize the performance of the forecast model you built in the previous exercise.

The test dataset, ml_forecast results, and plot_series are preloaded, along with evaluation functions (mape, rmse, coverage, shown below) and pandas as pd.

def mape(y, yhat):
    mape = mean(abs(y - yhat)/ y) 
    return mape

def rmse(y, yhat):
    rmse = (mean((y - yhat) ** 2 )) ** 0.5
    return rmse

def coverage(y, lower, upper):
    coverage = sum((y <= upper) & (y >= lower)) / len(y)
    return coverage

Let's first assess the model's performance and then visualize the forecast.

Deze oefening maakt deel uit van de cursus

Designing Forecasting Pipelines for Production

Cursus bekijken

Oefeninstructies

Create fc by merging the ml_forecast and test datasets.
Calculate rmse using the rmse() custom function provided, passing fc["y"] and fc[model] to the two arguments in that order; storing as a variable called r within the for loop.
Complete the evaluation by sorting fc_performance by rmse in ascending order.

Praktische interactieve oefening

Probeer deze oefening eens door deze voorbeeldcode in te vullen.

# Combine the data
fc = ml_forecast.____(____, how="left", on="ds")
fc_performance = None

for model in ["LGBMRegressor", "XGBRegressor", "LinearRegression"]:
    m = mape(y=fc["y"], yhat=fc[model])
    
    # Calculate RMSE
    r = ____(y=fc["____"], yhat=fc[____])
    c = coverage(y=fc["y"], lower=fc[model + "-lo-95"], upper=fc[model + "-hi-95"])

    perf = {"model": model, "mape": m, "rmse": r, "coverage": c}
    if fc_performance is None:
        fc_performance = pd.DataFrame([perf])
    else:
        fc_performance = pd.concat([fc_performance, pd.DataFrame([perf])])

# Sort the performance metrics by rmse
print(fc_performance.____("____"))

Code bewerken en uitvoeren