Performance metrics for regression trees
1. Performance metrics for regression trees
Well done! Now that you’ve trained a regression tree, it’s time to evaluate the performance of the model.2. How to measure performance?
In classification, a single prediction can be either right or wrong, and nothing else. So it makes sense to evaluate classification models using a metric like accuracy or the confusion matrix. In regression, however, your prediction can be almost right or even totally wrong. There is no binary correctness that calculates accuracy. You need to evaluate the regression trees using a different metric. Since the response is some real-valued number, and our prediction is the same, it makes sense to measure how far our predictions are away from the true values.3. Common metrics for regression
There are several metrics for regression and two popular ones are the Mean Absolute Error, or MAE, and the Root Mean Square Error, also known as RMSE. MAE is the average absolute distance between the actual (or observed) values and the predicted values. In this picture, the predictions are depicted by the blue line, and the true values are the black dots. The red bars are the prediction errors, and the mean absolute error would be the average length of the red bars. The root mean squared error is the square root of the average squared length of the red bars.4. Formulas and intuition
The formula for MAE is very straight-forward. You sum up all the absolute differences between the actual and the predicted values and divide them by the number of predictions made to get the average error. The RMSE is similar, but instead of taking the absolute difference between errors, you square the differences of the errors. Then you’ll average those values5. Formulas and intuition
and take the square root of the whole thing. Taking the square root brings the metric back to the original scale of the response. Both MAE and RMSE express average model prediction errors in units of the variable of interest. The key difference between the two is that RMSE punishes large errors more harshly than MAE. Since the errors are squared before they are averaged, the RMSE gives a relatively high weight to large errors. This means the RMSE should be more useful when large errors are particularly undesirable.6. Coding: predictions
Tidymodels provides the yardstick package, which is a toolbox for evaluating your models. Simply load the parsnip package to make predictions and the yardstick package for evaluations, or just the tidymodels package to load both. First, we use the predict function with our model and chocolate_test data to get predicted values. Then, we use the bind_cols() function to add these predictions to the test data as first column dot-pred.7. Coding: mae() and rmse()
This is then passed to the mae() function which also expects the column that contains the predictions or estimates, dot-pred, and the column that contains the truth values, final_grade. The result is a tibble that contains the mean absolute error in the column dot-estimate. The same goes for the rmse() function. This works exactly the same as with the accuracy() function for classification trees.8. Let's evaluate!
Time to put this into practice.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.