1. Quantifying logistic regression fit
In this last lesson, we'll assess the performance of logistic regression models. The diagnostic plots we drew for linear models are less useful in the logistic case. Instead, we'll look at confusion matrices.
2. The four outcomes
A logical response variable leads to four possible outcomes.
If the customer didn't churn and we predicted they wouldn't, or if they did churn and we predicted that, the model did well.
There are two bad cases. Predicting the customer churned when they didn't is called a false positive. Predicting the customer didn't churn when they did is called a false negative.
The counts of each outcome are called a confusion matrix.
3. Confusion matrix: counts of outcomes
Recall the model of churn versus recency. Getting the counts of model outcomes required some data manipulation.
First, we get the actual responses from the has_churned column of the dataset.
Next we get the predicted responses from the model. Calling the predict method on the fitted logistic regression model returns the predicted values of each observation in the dataset. These predicted values are probabilities. To get the most likely outcome, we need to round the values to zero or one.
We then combine actual and predicted responses in a DataFrame, and use the value_counts method to get the counts of each combination of values. This is the confusion matrix mentioned earlier.
We correctly predicted that one hundred and forty one customers didn't churn and eighty nine customers did churn. There were fifty nine false positives and one hundred and eleven false negatives.
4. Visualizing the confusion matrix
The confusion matrix can also be created automatically with the pred_table method. Calling pred_table on the fitted model object will return an array. The true negatives and true positives are on the main diagonal of the matrix, the false negatives and false positives are on the second diagonal of the matrix. These values are the same as what we calculated on the previous slide.
The mosaic function from the statsmodels package lets you easily plot the confusion matrix.
To interpret this, start by looking at the column widths. The width of each column is proportional to the fraction of observations in each category of actual values. Here, there are two hundred actual churns and two hundred actual not churns, so each column has the same width.
Then each column displays the fraction of predicted observations with each value. Here, just over a quarter of the actual not churns were predicted to be churns, so the block in the upper left is just over a quarter of the height of the first column.
5. Accuracy
Now let's look at ways of quantifying model fit using performance metrics.
The first metric is the model accuracy. This is the proportion of correct predictions. That is, the number of true negatives plus the true positives, divided by the total number of observations. Higher accuracy is better.
The total number of correct observations is one hundred and forty one plus eighty nine. We divide this total by the total number of observations, which is the sum of all four numbers.
6. Sensitivity
The second metric is sensitivity. This is the proportion of observations where the actual response was true where the model also predicted that they were true. That is, the number of true positives divided by the sum of the false negatives and true positives. Higher sensitivity is better.
Here, eighty nine of the two hundred customers who churned were correctly predicted to churn.
7. Specificity
The third metric is specificity. This is the proportion of observations where the actual response was false where the model also predicted that they were false. That is, the number of true negatives divided by the sum of the true negatives and false positives. Again, higher specificity is better, though there is often a trade-off where improving specificity will decrease sensitivity, or increasing sensitivity will decrease specificity.
Here, one hundred and forty one of the two hundred customers who didn't churn were correctly predicted to not churn.
8. Let's practice!
Time for the last round of exercises. Try not to get confused by the confusion matrix.