Assessing model fit

1. Assessing model fit

The next step in the modeling process is assessing model performance.

2. Binary classification

In binary classification, we have two possible categories. The positive class is the category that is of interest to predict, such as 'yes' in our purchased variable. The negative class is the remaining category. In tidymodels, the outcome variable must be a factor with the first level as the positive class. To check the ordering of a factor vector, pass it into the levels() function. To change the ordering, assign a character vector with the desired order to this levels() function call.

3. Confusion matrix

A confusion matrix displays the counts of all combinations of actual and predicted outcomes from a binary classification model. The predicted categories appear along the rows while the true outcomes appear along the columns. There are two types of correct predictions, known as true positives and true negatives along the diagonal. A binary classification model can make two types of errors. Predicting a positive class for a row that has an actual negative class, known as a false positive, and predicting a negative class for a row with an actual positive class, known as a false negative.

4. Classification metrics with yardstick

To calculate confusion matrices and other metrics we need a tibble of model results that includes the true outcome, the predicted categories, and estimated probabilities for each category. These correspond to the columns in our leads_results tibble.

5. Confusion matrix with yardstick

To create a confusion matrix, we pass the leads_results tibble to the conf_mat() function and set the truth argument to the purchased column and the estimate argument to the dot_pred_class column. We see that our model correctly classified 252 rows in the test dataset. It made 46 false negative errors, where it predicted that a customer will not purchase when in fact they did, and 34 false positive errors, where it predicted that a customer will purchase when in fact they did not.

6. Classification accuracy

The accuracy() function is used to calculate classification accuracy. It takes the same arguments as conf_mat() and calculates the proportion of correctly classified rows in the results tibble. yardstick functions always return a tibble where the dot-metric column lists the name of the calculated metric and the dot-estimate column contains the value.

7. Sensitivity

Accuracy is generally not the best metric. Classifying all rows as 'no' in our leads_df data would give an accuracy of 64%, for example. Sensitivity is the proportion of all positive cases that were correctly classified and improves with lower false negative rates. For example, of the customers who did purchase a product, what proportion did our model predict correctly?

8. Calculating sensitivity

The sens() function calculates sensitivity and takes the same arguments as the conf_mat() and accuracy() functions.

9. Specificity

Specificity measures the proportion of all negative cases that were correctly classified and improves with lower false positive rates. For example, of the customers who did not purchase a product, what proportion did our model predict correctly? Another common metric is the false positive rate, which is just 1 minus the specificity and measures the proportion of false positives among all negative cases.

10. Calculating specificity

The spec() function calculates specificity and takes the same arguments as the sens() function.

11. Creating a metric set

Instead of calculating metrics one by one, we can create a custom metric function with metric_set(). We pass the desired yardstick functions by name into metric_set() to create a new metric function. This new function can be used to calculate our metrics all at once!

12. Many metrics

The yardstick package offers many more metrics, which can be found in the documentation. Passing the results of conf_mat() to the summary() function will calculate all binary classification metrics at once.

13. Let's practice!

Let's practice calculating performance metrics!

Create Your Free Account

By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.