Backtesting with MAE, MSE
In this exercise, you will practice how to evaluate model performance by conducting backtesting. The out-of-sample forecast accuracy is assessed by calculating MSE and MAE.
You can easily estimate prediction errors MSE and MAE with pre-defined functions in the sklearn.metrics
package. The actual variance and predicted variance have been preloaded in actual_var
and forecast_var
respectively.
This exercise is part of the course
GARCH Models in Python
Exercise instructions
- In
evaluate()
, perform the MAE calculation by calling the corresponding function fromsklean.metrics
. - In
evaluate()
, perform the MSE calculation by calling the corresponding function fromsklean.metrics
. - Pass variables to
evaluate()
in order to perform the backtest.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
def evaluate(observation, forecast):
# Call sklearn function to calculate MAE
mae = ____(observation, forecast)
print('Mean Absolute Error (MAE): {:.3g}'.format(mae))
# Call sklearn function to calculate MSE
mse = ____(observation, forecast)
print('Mean Squared Error (MSE): {:.3g}'.format(mse))
return mae, mse
# Backtest model with MAE, MSE
evaluate(____, ____)