1. Targeting both leavers and stayers
As the objective of this course, is to develop a model that will correctly predict churn, recall score seems to be our target. However, recall alone is not enough, as by only targeting one class, we may have dramatically low accuracy for the other. Thus, a general rule is to use a measure that is not concentrated on one class alone.
2. AUC score
If our target are leavers, we would concentrate on recall, if stayers, then on specificity. But if your target is to have good predictions on both, then probably the best choice is to use AUC score. AUC stands for Area Under Curve and is basically a compound measure that is maximized when both recall and specificity are maximized. To calculate AUC score, one needs to place Recall on vertical, and 1- Specificity on horizontal axis and draw the Blue curve in the graph, which is called ROC. The are between ROC that we obtained and the green diagonal line that a random prediction could obtain is the AUC score
Using AUC as a target to maximize, the model will try to correctly classify both 1s and 0s keeping an eye on recall and specificity at the same time.
3. Let's practice!
Excellent, let's now put this into practice.