Estimating test error
Now that you have your posterior_predictive
(available to you in your workspace), you can evaluate model performance on new data. To do this, you will need to loop over the test observations, and for each of them, compute the prediction error as the difference between the predictive distribution for this observation and the actual, true value. This will give you the distribution of your model's error, which you can then visualize.
You will need pymc3
and numpy
, which have been imported for you as pm
and np
, respectively. The test data, bikes_test
, is also available in your workspace. Let's get to it!
This exercise is part of the course
Bayesian Data Analysis in Python
Exercise instructions
- Initialize
errors
as an empty list. - For each row in
bikes_test
, calculate prediction error as the predictive draws for this row fromposterior_predictive
minus the single true value ofnum_bikes
from the row. - Reshape
errors
by converting them to anumpy
array and applying the.reshape()
method to the outcome, and assign the final result toerror_distribution
. - Plot the test error distribution using
pymc3
'splot_posterior()
function.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Initialize errors
errors = ____
# Iterate over rows of bikes_test to compute error per row
for index, test_example in bikes_test.iterrows():
error = ____[____][:, ____] - ____[____]
errors.append(error)
# Reshape errors
error_distribution = ____(____).____()
# Plot the error distribution
____
plt.show()