1. Introducing the ROC curve
Manually evaluating classification thresholds is hard work!
2. The challenge
In order to do this correctly, we'd have to manually calculate dozens (or hundreds) of confusion matrices, and then visually inspect them until we find one we like.
This seems un-scientific, as it requires a lot of manual work, is heuristic-based, and could easily overlook a particular important threshold. We need a more systematic approach to evaluating classification thresholds.
3. ROC curves
One common approach to this problem is to let the computer iteratively evaluate every possible classification threshold and then calculate the true-positive rate and false-positive rate for each of them. We can then plot the true postive / false positive rate at every possible threshold, and visualize the trade-off between the 2 extreme models (predict all mines vs predict all rocks, or 100% true positive rate vs 0% false positive rate).
The resulting curve is called a ROC curve, or receiver operating characteristic curve. (Don't worry, no one actually remembers that acronym.) The ROC curve was developed during World War 2 as a method of analyzing radar signals. In this historically interesting case, a true positive would be correctly identifying a bomber by it's radar signal, while a false positive would be identifying a flock of birds as a bomber.
4. An example ROC curve
Let's take a look at a ROC curve for one of our models from the previous video. We use our predicted probabilities along with the actual classes as inputs to the colAUC function from the caTools package. If we specify the argument plotROC = TRUE, the function also plots the ROC curve for us.
Here, the X axis is the false positive rate, the y axis is the true positive rate, and we can see each possible prediction threshold as a point on the curve. Each of these points represents a confusion matrix we didn't have to evaluate by hand.
5. Let's practice!
Let's practice creating some ROC curves.