Evaluating forecast performance

In this exercise, you'll evaluate and visualize the performance of the forecast model you built in the previous exercise.

The test dataset, ml_forecast results, and plot_series are preloaded, along with evaluation functions (mape, rmse, coverage, shown below) and pandas as pd.

def mape(y, yhat):
    mape = mean(abs(y - yhat)/ y) 
    return mape

def rmse(y, yhat):
    rmse = (mean((y - yhat) ** 2 )) ** 0.5
    return rmse

def coverage(y, lower, upper):
    coverage = sum((y <= upper) & (y >= lower)) / len(y)
    return coverage

Let's first assess the model's performance and then visualize the forecast.

Diese Übung ist Teil des Kurses

Designing Forecasting Pipelines for Production

Kurs anzeigen

Anleitung zur Übung

Create fc by merging the ml_forecast and test datasets.
Calculate rmse using the rmse() custom function provided, passing fc["y"] and fc[model] to the two arguments in that order; storing as a variable called r within the for loop.
Complete the evaluation by sorting fc_performance by rmse in ascending order.

Interaktive Übung

Vervollständige den Beispielcode, um diese Übung erfolgreich abzuschließen.

# Combine the data
fc = ml_forecast.____(____, how="left", on="ds")
fc_performance = None

for model in ["LGBMRegressor", "XGBRegressor", "LinearRegression"]:
    m = mape(y=fc["y"], yhat=fc[model])
    
    # Calculate rmse
    r = ____(y=fc["____"], yhat=fc[____])
    c = coverage(y=fc["y"], lower=fc[model + "-lo-95"], upper=fc[model + "-hi-95"])

    perf = {"model": model, "mape": m, "rmse": r, "coverage": c}
    if fc_performance is None:
        fc_performance = pd.DataFrame([perf])
    else:
        fc_performance = pd.concat([fc_performance, pd.DataFrame([perf])])

# Sort the performance metrics by rmse
print(fc_performance.____("____"))

Code bearbeiten und ausführen