1. AIC and BIC
In the last lesson, we mentioned how ACF and PACF can't be used to choose the order of a model, when both of the orders p and q are non-zero. However there are more tools we can use, the AIC and the BIC.
2. AIC - Akaike information criterion
The Akaike information criterion, or AIC, is a metric which tells us how good a model is.
A model which makes better predictions is given a lower AIC score.
The AIC also penalizes models which have lots of parameters. This means if we set the order too high compared to the data, we will get a high AIC value. This stops us overfitting to the training data.
3. BIC - Bayesian information criterion
The Bayesian information criterion, or BIC, is very similar to the AIC.
Models which fit the data better have lower BICs and the BIC penalizes overly complex models.
4. AIC vs BIC
For both of these metrics a lower value suggests a better model.
The difference between these two metrics is how much they penalize model complexity.
The BIC penalizes additional model orders more than AIC and so the BIC will sometimes suggest a simpler model. The AIC and BIC will often choose the same model, but when they don't we will have to make a choice.
If our goal is to identify good predictive models, we should use AIC.
However if our goal is to identify a good explanatory model, we should use BIC.
5. AIC and BIC in statsmodels
After fitting a model in Python, we can find th AIC and BIC by using the summary of the fitted-models-results object. These are on the right of the table.
6. AIC and BIC in statsmodels
You can also access the AIC and BIC directly by using the dot-aic attribute and the dot-bic attribute of the fitted-model-results object.
7. Searching over AIC and BIC
Being able to access the AIC and BIC directly means we can write loops to fit multiple ARIMA models to a dataset, to find the best model order.
Here we loop over AR and MA orders between zero and two, and fit each model.
Then we print the model order along with the AIC and BIC scores
8. Searching over AIC and BIC
If we want to test a large number of model orders, we can append the model order and the AIC and BIC to a list, and later covert it to a DataFrame.
9. Searching over AIC and BIC
This means we can sort by the AIC score and not have to search through the orders by eye. We can do the same with the BIC score.
In this case the AIC and BIC favor different models, but we want a good predictive model so we will choose the model with the lowest AIC. This is an ARMA(2,1) model.
10. Non-stationary model orders
Sometimes when searching over model orders you will attempt to fit an order that leads to an error.
This ValueError tells us that we have tried to fit a model which would result in a non-stationary set of AR coefficients. This is just a bad model for this data, and when we loop over p and q we would like to skip this one.
11. When certain orders don't work
We can skip these orders in our loop by using a try and except block in python. Here's the code we had before.
12. When certain orders don't work
First we try to run the code in the try statement. We try to fit the model and print the scores.
If this fails then the code in the except statement is run, where we print None for the scores
13. Let's practice!
You've been learning a lot about the AIC and BIC. Time for you to put it to practice!