Exercise

Estimating test error

Now that you have your posterior_predictive (available to you in your workspace), you can evaluate model performance on new data. To do this, you will need to loop over the test observations, and for each of them, compute the prediction error as the difference between the predictive distribution for this observation and the actual, true value. This will give you the distribution of your model's error, which you can then visualize.

You will need pymc3 and numpy, which have been imported for you as pm and np, respectively. The test data, bikes_test, is also available in your workspace. Let's get to it!

Instructions

100 XP
  • Initialize errors as an empty list.
  • For each row in bikes_test, calculate prediction error as the predictive draws for this row from posterior_predictive minus the single true value of num_bikes from the row.
  • Reshape errors by converting them to a numpy array and applying the .reshape() method to the outcome, and assign the final result to error_distribution.
  • Plot the test error distribution using pymc3's plot_posterior() function.