1. Posterior predictive model checks
The R-squared statistic tells us how much variance in the dependent variable can be explained by the predictors. But this isn't the only way to assess model fit. In fact there are several methods that are specific to Bayesian estimation. The most important among these is a class of analyses called posterior predictive model checks, which can only be calculated when a Bayesian estimation process is used.
2. Using posterior distributions
As the name might suggest, posterior predictive model checks make use of the posterior distributions. Recall our model predicting a child's IQ score from their mom's IQ score. By default, this model is estimated with 4 chains, and 2,000 iterations in each chain, with the first 1,000 iterations of each discarded for warmup. This means that we have a total of 4,000 iterations that make up our posterior distribution (1,000 from each chain). Here we can see the values of the intercept and coefficient for the `mom_iq` variable for the first 10 iterations. As we can see, the values change with each iteration, which is expected as we are randomly sampling from the posterior distribution.
3. Posterior predictions
We can use these posterior draws to calculate predicted score for our data. For example, we can calculate predicted scores using the parameter values at iteration 1, another set of predicted scores using the parameter values at iteration 2, and so on. Ultimately, because we have 4,000 iteration, we can calculate 4,000 predicted scores for each observation. We can get all the predicted values for all observations using the posterior_linpred function. This returns a matrix that has a row for each iteration, and a column for each observation. In this example, we can see the predicted IQ scores for the first 10 iterations for children 1 through 5. Because these scores are generated using the model parameters, these predicted scores form a distribution of what our observed scores should look like - if the specified model was correct. Thus, deviations from these predictive distributions are an indication of poor fit.
4. Comparing score distributions
One way we can look for deviations is to compare the distribution of predicted scores in an iteration to the distribution of observed scores. For example we can get the predicted scores for the first and second iterations by pulling the first and second rows of the matrix return by posterior_linpred. We can then look at a summary of those scores, compared the summary of the observed scores. Here we can see that the mean of the iterations is similar to the mean of the observed data, but min and max values are less extreme than in the observed data.
5. Comparing single scores
Similarly, we can compare individual scores to their expected distribution. For example, child 24 had an observed score of 87, which falls in line with the distribution of predicted scores for that child, as shown in column 24 of the matrix returned by posterior_linpred. In contrast, child 185 had an observed score of 111, which is quite different from the distribution of predicted scores for this student. This means that the model does not do a very good job of predicting the IQ score for this student.
6. Let's practice
Now it's your turn to use posterior predictive model checks!