Get startedGet started for free

Backtesting with MAE, MSE

In this exercise, you will practice how to evaluate model performance by conducting backtesting. The out-of-sample forecast accuracy is assessed by calculating MSE and MAE.

You can easily estimate prediction errors MSE and MAE with pre-defined functions in the sklearn.metrics package. The actual variance and predicted variance have been preloaded in actual_var and forecast_var respectively.

This exercise is part of the course

GARCH Models in Python

View Course

Exercise instructions

  • In evaluate(), perform the MAE calculation by calling the corresponding function from sklean.metrics.
  • In evaluate(), perform the MSE calculation by calling the corresponding function from sklean.metrics.
  • Pass variables to evaluate() in order to perform the backtest.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

def evaluate(observation, forecast): 
    # Call sklearn function to calculate MAE
    mae = ____(observation, forecast)
    print('Mean Absolute Error (MAE): {:.3g}'.format(mae))
    # Call sklearn function to calculate MSE
    mse = ____(observation, forecast)
    print('Mean Squared Error (MSE): {:.3g}'.format(mse))
    return mae, mse

# Backtest model with MAE, MSE
evaluate(____, ____)
Edit and Run Code