1. Accuracy metrics: regression models
Now that we have learned about holdout samples let's discuss accuracy metrics used when validating models — starting with regression models.
2. Regression models
Remember, regression models are built for continuous variables. This could be predicting the number of points a player will score tomorrow, or the number of puppies a dog is about to have!
3. Mean absolute error (MAE)
To assess the performance of a regression model, we can use the mean absolute error. It is the simplest and most intuitive error metric and is the average absolute difference between the predictions (y_i) and the actual values (y_i hat). If your dog had six puppies, but you had predicted only four, the absolute difference would be two.
This metric treats all points equally and is not sensitive to outliers. When dealing with applications where we don't want large errors to have a major impact, the mean absolute error can be used. An example could be predicting your car's monthly gas bill, when an outlier may have been caused by a one-time road trip.
4. Mean squared error (MSE)
Next is the mean squared error (MSE). It is the most widely used regression error metric for regression models. It is calculated similarly to the mean absolute error, but this time we square the difference term.
The MSE allows larger errors to have a larger impact on the model. Using the previous car example, if you knew once a year you might go on a road trip, you might expect to occasionally have a large error and would want your model to pick up on these trips.
5. MAE vs. MSE
Picking between the MAE and the MSE comes down to the application. These results are in different units though and should not be directly compared!
6. Mean absolute error
To practice these metrics, let's use the ultimate Halloween candy data dataset. Here we are predicting the win-percentage of candies in head-to-head match-ups with other candies. Let's assume we have already fit a random forest model and calculated predictions for the test dataset.
For the mean absolute error, we can calculate this two ways. A manual calculation, which takes the sum of the absolute differences and divides by the total number of observations, or we can use scikit-learn's mean_absolute_error() function. We provide an array of the actual values, followed by an array of the predictions. Both methods produce a single value of 9-point-99 as the output. We are covering the manual calculations for these functions to understand the results of these error metrics.
Notice that we are looking at the test data accuracy. This error means that we are about 10 percentage-points off on average when predicting the win-percentage. As win-percentages range from 0 to 1, this is fairly good.
7. Mean squared error
For the mean squared error, we can calculate this manually or with the mean_squared_error() function. Both methods produce a value of 141-point-4.
In this example, the mean squared error is a more appropriate accuracy metric, as we want outliers to have more of an impact on the model's performance. For example, if one chocolate bar really underperforms, there may be attributes of that chocolate bar that truly matter other than it being chocolate.
8. Accuracy for a subset of data
Sometimes we want to know a model's accuracy for a specific subset, such as how this model performs on only chocolate candies.
If column 1 in our test set has 1's for candies containing chocolate, and 0; otherwise, we filter the test array based on these values and run the accuracy metrics. Since the chocolate candies had errors of less than 9 and the non-chocolate candies had errors of 11, the model performed better on chocolate candy.
9. Let's practice
Let's work through a couple of examples on regression accuracy metrics.