1. Other model metrics
You've now fit several different classifiers and evaluated their performance using metrics such as precision, sensitivity, and specificity. In this video, you'll add another technique to your churn modeling toolbox: The receiver operating characteristic curve, or ROC curve, that will allow you to visualize the performance of your churn classifier.
2. Probability thresholds
Every prediction your classifier makes on a new data point has an associated probability. By default in scikit-learn, if this probability is above 50%, then your model would predict the data point as belonging to the positive class, and if it is lower than 50%, it would predict the negative class. In other words, the default probability threshold here is 50%.
What happens if we vary this threshold, and, for each threshold, plot the model's true positive rate against the false positive rate? We get what is known as the ROC curve.
3. ROC curve
This curve allows you to diagnose the performance of the model at different thresholds.
4. ROC curve: Part 2
The y-axis is the true positive rate and ranges from 0 to 1,
5. ROC curve: Part 3
and the x-axis is the false positive rate which also ranges from 0 to 1.
6. ROC curve: Part 4
A perfect classifier, then, would have a true positive rate of 1 and false positive rate of 0.
7. ROC curve: Part 5
This information can be used to extract another useful metric: the area under the ROC curve, known as the AUC. Notice that a better performing model will have a larger area under the curve.
8. ROC curve: Part 6
The ROC curve of a model that randomly guesses would be a diagonal line,
9. ROC curve: Part 7
and the area under this line
10. ROC curve: Part 8
would be 0 point 5. Therefore, having an AUC above 0 point 5 would be better than randomly guessing, while an AUC above 0 point 7 or 0 point 8 would indicate a well-performing model.
11. Generating probabilities in sklearn
Together with the ROC curve, the AUC allows you to easily evaluate and compare the performance of different classifiers. In order to plot the ROC, you need to first calculate the probabilities that your model generates for each data point. Scikit-learn classifiers such as the logistic regression classifier shown here have a predict proba method for this purpose.
Each column contains the probabilities for the respective target values. So for the first data point, the probability of the predicted label being "0" is around 80%, and the probability of it being "1" is 20%.
Since our positive class is "1", we're interested in the second column, and we can select that as shown here.
12. ROC curve in sklearn
To plot the ROC curve, import roc curve from sklearn dot metrics. This function takes in two arguments: The actual labels, followed by the predicted probabilities we just computed. Unpack this result into three variables: false positive rate, FPR; true positive rate, TPR; and the thresholds. We can then plot the FPR and TPR using matplotlib. The additional code here provides informative labels and plots the diagonal line which we can use to compare against.
13. Area under the curve
To compute the AUC, sklearn dot metrics has a roc_auc_score function that you can use similarly to the other metrics functions in scikit-learn.
14. Let's practice!
Now it is your turn to evaluate the performance of your classifiers using ROC curves!