Lab introduction (optional)
1. Lab introduction (optional)
SPEAKER: Apply your new skills in an optional hands-on lab designed for practical experience. While encouraged, completing this lab won't affect your course completion status. In this lab, you'll use AutoML, a no-code tool, to build a machine learning model to predict loan risk. The data set used in the lab relates to loans from a financial institution and has 2,050 data points. AutoML requires at least 1,000 data points in a data set. The goal is to practice working through the three phases of the machine learning workflow-- data preparation, model development, and model serving. Before you start the lab, let's explain the details about model evaluation so that you can interpret the training results. Let's start with the confusion matrix. You will get a similar result like this in the lab. Pause for a second and try to interpret this matrix yourself. What does it tell you? 100% true positive rate. The positive class here is 'repay' (0), the desired business outcome. This 100% means that the model is perfect at identifying everyone who will actually repay their loan. It never misses a good, safe customer. That’s a fantastic outcome for maximizing business opportunities. Note that the True Positive Rate equals True Positives divided by the sum of True Positives and False Negatives. If the terms sound unfamiliar, please refer to the previous example, where you learned about the confusion matrix. 87% true negative rate. The negative class is 'not repay' (1), the high-risk outcome. This 87% means that the model correctly identifies 87% of all the people who are actually defaulters. This is a strong result for risk management, as the bank successfully catches and rejects most of the high-risk applications. Note that the True Negative Rate equals True Negatives divided by the sum of False Positives and True Negatives. 13% false positive rate. A 'False Positive' is the most expensive mistake. The model predicts the customer is safe ('repay' or 0) when they would have actually defaulted ('not repay' or 1). This means that 13% of people who will actually default on their loan are mistakenly approved. This directly leads to financial loss for the bank. Note that the False Positive Rate equals False Positives divided by the sum of False Positives and True Negatives. Finally, 0% false negative rate. A 'False Negative' is the error where the model predicts risk ('not repay' or 1) when the customer would have been safe ('repay' or 0). Since this rate is 0%, the model never incorrectly rejects a customer who is actually a safe borrower. The bank avoids all lost business opportunity from turning away good customers. This perfect rate for customer acceptance is a core strength, despite the trade-off of approving some bad loans seen in the False Positive rate. Let's look at the precision recall curve you will encounter in the upcoming AutoML lab. The confidence threshold determines how a machine learning model counts the positive cases. A higher threshold increases the precision but decreases recall. A lower threshold decreases the precision but increases recall. Moving the confidence threshold to 0 produces the highest recall of 100% and the lowest precision of 50%. So what does that mean? That means the model predicts that 100% of loan applicants will be able to repay a loan they take out. However, actually, only 50% of people were able to repay the loan. Using this threshold to identify the default cases in this example can be risky because it means you can only get half of the loan investment back. Now, let's view the other extreme by moving the threshold to 1. This will produce the highest precision of 100% with the lowest recall of 1%. What does this mean? It means that of all the people who were predicted to repay the loan, 100% of them actually did. However, you rejected 99% of loan applicants by only offering loans to 1% of them. That's a pretty big business loss for your company. These are both extreme examples, but it's important that you always try to set an appropriate threshold for your model. Now that we've made a review, let's start the lab.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.