1. Model Metrics
You've learned how to split your data into training and testing sets, how to fit a classifier, and how to evaluate its performance by computing its accuracy. Your out-of-the-box scikit-learn model has a high accuracy. Is your churn modeling journey then complete? Unfortunately, not quite.
2. Imbalanced classes
In practice, many datasets you'll encounter will tend to have imbalanced classes: That is, more data points that belong to one category than another. In our churn dataset, for example, there are nearly 6 times as many non-churners as there are churners. This can have an impact on the performance of your model as it might learn to always predict the majority class - in this case, that the customer will not churn. There are techniques such as upsampling and downsampling that you can use to balance the classes and solve this problem. Check out DataCamp's more advanced machine learning courses for more insight into those techniques. For our purposes, what this means is that when working with imbalanced classes, accuracy is not a very useful metric. If we were to build a classifier that ALWAYS predicted that a customer would NOT churn, it would be very accurate - about 85% accurate, in this case - yet it would completely fail at identifying customers who will actually churn. This creates the need for metrics that are more nuanced.
3. Confusion Matrix Part 1
To better understand this, we can draw up a 2 by 2 table that summarizes the possible predictions our model can make. In machine learning, this table is known as a confusion matrix.
4. Confusion Matrix Part 2
Across the top are the actual classes,
5. Confusion Matrix Part 3
and down the side are our model's predicted classes.
6. Confusion Matrix Part 4
In the top left box, we have the number of churners correctly classified.
7. Confusion Matrix Part 5
These are the true positives.
8. Confusion Matrix Part 6
In the bottom right box, we have the number of non-churners that are correctly classified.
9. Confusion Matrix Part 7
These are the true negatives. Note that this is the case because we defined our positive class to be the customers who churned.
10. Confusion Matrix Part 8
In the top right box, we have the number of non-churners incorrectly classified as churners.
11. Confusion Matrix Part 9
These are the false positives, or the "false alarms".
12. Confusion Matrix Part 10
Finally, in the bottom left box,
13. Confusion Matrix Part 11
we have the number of churners incorrectly classified as non-churners.
This table of predictions is very powerful, as you can use it to compute a number of different metrics.
14. Confusion Matrix Part 12
For example, you can calculate the accuracy by adding the true positives and true negatives, and dividing by the sum of all the elements in the table.
More interestingly, there are a couple of other metrics you can also calculate called precision and recall.
15. Precision
Precision is True Positives divided by the sum of True Positives and False Positives. In other words, it is the number of correctly classified churners divided by the total number of customers classified as churners. If our model had high precision, it would mean that there were not many false positives - that is, not many non-churners who were classified as churners.
16. Recall
Recall, or sensitivity, is True Positives divided by the sum of True Positive and False Negatives. A high recall in our case means that the model correctly classified most churners.
17. Precision vs. Recall
Which metric you focus on optimizing often depends on the business case. If the offers you make to keep potential churners is high in relation to the value of losing customers you want your model to have high precision or minimize false positives. Conversely, if losing customers you didn't think would churn is more expensive than an offer to keep a customer that would not have churned than you want a high recall or minimize false negatives. Both recall and precision are far more valuable than just looking at accuracy.
18. Confusion Matrix in scikit-learn
To create a confusion matrix in scikit-learn, you can use the confusion_matrix function from the sklearn dot metrics class. It takes in two arguments: the actual labels, and the predicted labels. Sklearn's metrics class has similar functions that allow you to compute the metric of interest, whether that is precision or recall.
19. Let's practice!
It's time now for you to practice calculating these metrics yourself!