Model fit with posterior predictive model checks
1. Model fit with posterior predictive model checks
In the last lesson, we talked about how we can get predicted scores from our posterior, and compare those predictions to our observed data. However, it would be incredibly tedious to compare the observed data to every replication one at a time, or look at the results for each observation independently. In this lesson we'll learn how we can summarize results across all replications to make global evaluations of how well the model fits the data.2. R squared posterior distribution
The first measure of fit using posterior predictive model checks is one we've looked at before: the R-squared. With our estimated `stan_model`, we can get a posterior distribution of the R-squared using the `bayes_R2` function. We can then look at a summary to get an idea of the distribution, or create a 95% credible interval using the quantile function.3. R squared histogram
We can also make a histogram of the R-squared values. From this, we can see that the true R-squared of our model is mostly likely somewhere between .1 and .3.4. Density overlay
We can also compare the distributions from all replications to the observed data at one time using the `pp_check` function, and specifying a density overlay. This will generate a plot like this. Here, each light blue line represents the distribution of predicted scores from a single replications, and the dark blue line represents the observed data. If the model fits, the dark blue line should align closely with the light blue lines.5. Posterior predictive tests
We can also test certain characteristics of the dependent variable. By specifying "stat" in the `pp_check` function, we can get a distribution of the mean of the dependent variable from all replications. These are the light blue bars: the means from each replication plotted as a histogram. The mean from the observed data is then plotted on top as a dark blue bar. In our example, the observed mean of kids' IQ scores falls within the expected range of means from the posterior predictions. Therefore, we have evidence that our model fits.6. Posterior predictive tests
However, the mean is just one aspect of the observed data. By changing "stat" to "stat_2d", we can look at multiple aspects of our dependent variable: the mean and standard deviation. In this plot, each light blue dot represents the mean and standard deviation of the predicted IQ scores in one replication. The dark blue dot shows our observed data. Because the dark blue dot is inside the mass of light blue dots, we have more evidence that our model fits our data.7. Let's practice!
Now you try using posterior predictive model checks to assess the model fit of our Spotify model.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.