Backtesting with MAE, MSE

In this exercise, you will practice how to evaluate model performance by conducting backtesting. The out-of-sample forecast accuracy is assessed by calculating MSE and MAE.

You can easily estimate prediction errors MSE and MAE with pre-defined functions in the sklearn.metrics package. The actual variance and predicted variance have been preloaded in actual_var and forecast_var respectively.

This exercise is part of the course

GARCH Models in Python

View Course

Exercise instructions

In evaluate(), perform the MAE calculation by calling the corresponding function from sklean.metrics.
In evaluate(), perform the MSE calculation by calling the corresponding function from sklean.metrics.
Pass variables to evaluate() in order to perform the backtest.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

def evaluate(observation, forecast): 
    # Call sklearn function to calculate MAE
    mae = ____(observation, forecast)
    print('Mean Absolute Error (MAE): {:.3g}'.format(mae))
    # Call sklearn function to calculate MSE
    mse = ____(observation, forecast)
    print('Mean Squared Error (MSE): {:.3g}'.format(mse))
    return mae, mse

# Backtest model with MAE, MSE
evaluate(____, ____)

Edit and Run Code