1. The ROC-curve
In the previous exercise,
2. Until now
you saw how banks might prefer one model over another depending on the minimum "bad rate" they are allowed to have, or, turning it around, depending on their acceptance rate. Often, banks just want to know what the best model is overall, without having to make any assumptions regarding acceptance rates or cutoffs. To illustrate this,
3. Confusion matrix
let's go back to our decision matrix metrics. Here, we have learned about accuracy, sensitivity, and specificity. We already discussed that classification accuracy includes all the test set elements in the denominator and, as a result, is tempting to use.
4. Accuracy?
However, classification accuracy is also a problematic measure in the credit scoring context as accuracy is generally maximized when a very high cutoff is chosen, or in other words, when all test set elements are classified as non-defaults. This is why one of the most popular methods to evaluate credit risk models is based on sensitivity and specificity.
This evaluation method is called the receiver operating characteristics curve, often referred to as
5. The ROC-curve
the ROC curve. The curve is constructed by plotting the sensitivity against one minus the specificity for each possible cutoff. The curve always starts in the lower-left corner, where sensitivity is 0 and specificity is 1,
6. The ROC-curve
corresponding to a cutoff of 1, which means classifying all the cases as non-defaults. The other end of the curve in the upper right corner
7. The ROC-curve
corresponds to a cutoff equal to zero, where sensitivity is equal to 1, and specificity is equal to 0.
8. The ROC-curve
Generally speaking, the closer an ROC curve
9. The ROC-curve
is to the top left corner, the better. These curves have higher specificities associated with higher sensitivities. On the other extreme, a 45-degree line,
10. The ROC-curve
as indicated by the red line here, would be the result of assigning cases to the default versus non-default group randomly rather than through a model.
In reality, when comparing models, you will often see that the differences between several ROC curves are not as clear as shown in the previous slides. Additionally, ROC curves may look different but show crossovers, which makes it hard to tell which model is better.
11. Which one is better?
This problem is illustrated here by ROC curves for two models: a black ROC curve for model A, and a blue ROC curve for model B. A measure that banks like to use when comparing ROC curves is the so-called "area under the curve" of these ROC curves, also referred to as the AUC. The AUC of a model is between point-5 (which corresponds to the red diagonal line shown before) and 1 (which would correspond with a perfect model). Computing the AUC here, we see that model B leads to a higher AUC and should be preferred over model A.
12. Let's practice!
Let's have a look at some ROC curves for the models we constructed previously!