Logistic regression and the ROC curve

1. Logistic regression and the ROC curve

It's time to introduce another model: logistic regression.

2. Logistic regression for binary classification

Despite its name, logistic regression is used for classification. This model calculates the probability, p, that an observation belongs to a binary class. Using our diabetes dataset as an example, if p is more than or equal to zero-point-five, we label the data as one, representing a prediction that an individual is more likely to have diabetes; if p is less than zero-point-five, we label it zero to represent that they are more likely to not have diabetes.

3. Linear decision boundary

Note that logistic regression produces a linear decision boundary, as we can see in this image.

4. Logistic regression in scikit-learn

Using logistic regression in scikit-learn follows the same approach as used for other models. We first import LogisticRegression from sklearn-dot-linear_model. Next we instantiate the classifier, split our data, fit the model on our training data, and predict on our test set. In this video we use the churn dataset.

5. Predicting probabilities

We can predict probabilities of each instance belonging to a class by calling logistic regression's predict_proba method and passing the test features. This returns a 2-dimensional array with probabilities for both classes, in this case, that the individual did not churn, or did churn, respectively. We slice the second column, representing the positive class probabilities, and store the results as y_pred_probs. Here we see the model predicts a probability of point-zero-eight-nine that the first observation has churned.

6. Probability thresholds

The default probability threshold for logistic regression in scikit-learn is zero-point-five. This threshold can also apply to other models such as KNN. So what happens as we vary this threshold?

7. The ROC curve

We can use a receiver operating characteristic, or ROC curve, to visualize how different thresholds affect true positive and false positive rates. Here, the dotted line represents a chance model, which randomly guesses labels.

8. The ROC curve

When the threshold equals zero, the model predicts one for all observations, meaning it will correctly predict all positive values, and incorrectly predict all negative values.

9. The ROC curve

If the threshold equals one, the model predicts zero for all data, which means that both true and false positive rates

10. The ROC curve

are zero. If we

11. The ROC curve

vary the threshold, we get a series of different false positive and true positive rates.

12. The ROC curve

A line plot of the thresholds helps to visualize the trend.

13. Plotting the ROC curve

To plot the ROC curve, we import roc_curve from sklearn-dot-metrics. We then call the function roc_curve; we pass the test labels as the first argument, and the predicted probabilities as the second. We unpack the results into three variables: false positive rate, FPR; true positive rate, TPR; and the thresholds. We can then plot a dotted line from zero to one, along with the FPR and TPR;

14. Plotting the ROC curve

to produce a figure such as this. This looks great, but how do we quantify the model's performance based on this plot?

15. ROC AUC

If we have a model with one for true positive rate and zero for false positive rate, this would be the perfect model. Therefore, we calculate the area under the ROC curve, a metric known as AUC. Scores range from zero to one, with one being ideal. Here, the model scores point-six-seven, which is only 34% better than a model making random guesses.

16. ROC AUC in scikit-learn

We can calculate AUC in scikit-learn by importing roc_auc_score from sklearn-dot-metrics. We call roc_auc_score, passing our test labels and our predicted probabilities, calculated by using the model's predict_proba method on X_test. As expected, we get a score of zero-point-six-seven.

17. Let's practice!

Now let's build a logistic regression model and evaluate its performance!

This exercise is part of the course

Supervised Learning with scikit-learn

IntermediateSkill Level

4.4+

Start Course for Free

In this chapter, you'll be introduced to classification problems and learn how to solve them using supervised learning techniques. You'll learn how to split data into training and test sets, fit a model, make predictions, and evaluate accuracy. You’ll discover the relationship between model complexity and performance, applying what you learn to a churn dataset, where you will classify the churn status of a telecom company's customers.

Exercise 1: Machine learning with scikit-learn Exercise 2: Binary classification Exercise 3: The supervised learning workflow Exercise 4: The classification challenge Exercise 5: k-Nearest Neighbors: Fit Exercise 6: k-Nearest Neighbors: Predict Exercise 7: Measuring model performance Exercise 8: Train/test split + computing accuracy Exercise 9: Overfitting and underfitting Exercise 10: Visualizing model complexity

In this chapter, you will be introduced to regression, and build models to predict sales values using a dataset on advertising expenditure. You will learn about the mechanics of linear regression and common performance metrics such as R-squared and root mean squared error. You will perform k-fold cross-validation, and apply regularization to regression models to reduce the risk of overfitting.

Exercise 1: Introduction to regression Exercise 2: Creating features Exercise 3: Building a linear regression model Exercise 4: Visualizing a linear regression model Exercise 5: The basics of linear regression Exercise 6: Fit and predict for regression Exercise 7: Regression performance Exercise 8: Cross-validation Exercise 9: Cross-validation for R-squared Exercise 10: Analyzing cross-validation metrics Exercise 11: Regularized regression Exercise 12: Regularized regression: Ridge Exercise 13: Lasso regression for feature importance

Having trained models, now you will learn how to evaluate them. In this chapter, you will be introduced to several metrics along with a visualization technique for analyzing classification model performance using scikit-learn. You will also learn how to optimize classification and regression models through the use of hyperparameter tuning.

Exercise 1: How good is your model?Exercise 2: Deciding on a primary metric Exercise 3: Assessing a diabetes prediction classifier Exercise 4: Logistic regression and the ROC curve

Current Exercise

Exercise 5: Building a logistic regression model Exercise 6: The ROC curve Exercise 7: ROC AUC Exercise 8: Hyperparameter tuning Exercise 9: Hyperparameter tuning with GridSearchCV Exercise 10: Hyperparameter tuning with RandomizedSearchCV

Learn how to impute missing values, convert categorical data to numeric values, scale data, evaluate multiple supervised learning models simultaneously, and build pipelines to streamline your workflow!

Exercise 1: Preprocessing data Exercise 2: Creating dummy variables Exercise 3: Regression with categorical features Exercise 4: Handling missing data Exercise 5: Dropping missing data Exercise 6: Pipeline for song genre prediction: I Exercise 7: Pipeline for song genre prediction: II Exercise 8: Centering and scaling Exercise 9: Centering and scaling for regression Exercise 10: Centering and scaling for classification Exercise 11: Evaluating multiple models Exercise 12: Visualizing regression model performance Exercise 13: Predicting on the test set Exercise 14: Visualizing classification model performance Exercise 15: Pipeline for predicting song popularity Exercise 16: Congratulations