Model choice and residual analysis

1. Model choice and residual analysis

As you have seen in the exercises, it is often the case that two or more models seem reasonable. In general, it is typically a good idea to fit a few models before deciding on the best one.

2. AIC and BIC

The two most popular methods for choosing the best model are Akaike's Information Criterion (AIC), and Bayesian Information Criterion (BIC). As more parameters are included in a model, the error gets smaller whether or not the parameters are needed. Thus, model choice criteria include a penalty for adding parameters. BIC has a bigger penalty (log n vs 2 for AIC) and tends to choose a model with fewer parameters. The two, however, often agree. The goal is to find the model with the smallest AIC or BIC.

3. Model Choice: AR(1) vs. MA(2)

Here's an example using the growth rate of US GNP. Recall that growth rate can be calculated by differencing the log of the data. Two models were fit to the data, an AR(1) and an MA(2). The models were fit using sarima and the AIC and BIC for each model is displayed. Note that the BIC prefers the AR(1) while the AIC perfers the MA(2). Since the AR(1) is the simpler model, it would be appropriate to prefer that to the MA(2).

4. Residual Analysis

The basic goal of residual analysis is the same as in regression. That is, we want to make sure the residuals are white Gaussian noise. If not, then we have not found the best model. sarima provides a residual graphic like the one seen here. This particular graphic is for the AR(1) fit to the growth rate of US GNP. There are 4 plots. 1. The plot of the standardized residuals should be inspected for patterns. It's difficult to tell if the noise is white from this plot, but it's easy to tell if it's not white noise, for example, are there obvious patterns in the residuals. 2. The ACF of the residuals can be used to assess whiteness. 95% of the ACF values should be between the blue dashed lines. 3. The Q-Q plot assesses normality. If the residuals are normal, the points will line up with the line. There are often extreme values on the ends as there are here. As long as there are no huge departures from the line, then the normal assumption is reasonable. 4. Finally there is a Q-statistic that tests for whiteness in the residuals. As long as most points are above the blue dashed line, then you can safely assume the noise is white. If, however, many of the points are below the line, then there is still some correlation left in the residuals and you should try to fit another model or add a parameter. For this model, the residuals look fine.

5. Bad Residuals

Here's an example of a residual plot that is not ok.

6. Bad Residuals

The standardized residuals show

7. Bad Residuals

a periodic pattern

8. Bad Residuals

The ACF has large values

9. Bad Residuals

at lag 1 and 2 (at least)

10. Bad Residuals

Q-statistic has

11. Bad Residuals

all points below the line.