Mitigate Bias - Threshold Calibration
1. Mitigate Bias - Threshold Calibration
Now that we have learned how to mitigate bias from the data, let's see the techniques intervening machine learning models used to mitigate issues in different ways. First, even after training and fixing a machine learning model, there is still room to control classification behavior by calibrating thresholds. Threshold calibration defines the proportion of the prediction results by categorizing them into true positives, false positives, false negatives, and true negatives. After you determine these metrics on the entire data set, ensure that you go one step further by calculating the metrics across the different subgroups within your data. For example, you might find that a false negative rate at 0.1 is acceptable for the problem you're trying to solve with your machine learning system. Given that overall rate, how does that rate look across the subgroups as shown in these plots? After you compute your metrics, you can visualize the distribution of your evaluation metrics across each subgroup. This is depicted by the blue and green colors. Each color represents a different subgroup within the data. As you can see, predictions on each subgroup result in different proportions that could mean that the best and fair threshold can vary among different subgroups. By keeping this in mind, you are one step closer to identifying ways in which you can make your machine learning system more inclusive. To help you think through the thresholding from fairness purpose and to highlight what happens when you change variables. Let's look at an example where a machine learning model decides whether a bank approves or rejects loan applications. Imagine that we have two groups of people identified here by the colors blue and orange. Notice how the prediction score distributions are different between the two groups. Let's say we set the same threshold for the two groups at 55. With the same threshold. The blue and orange groups have different distributions and how they'll be able to pay back loans. However, the differences in group distribution and loan thresholds are a result of historical and systemic biases that have somehow made it into our data. Because of this, it might be unfair to ignore the differences between the two groups. Let's examine the distributions of these loan thresholds a bit more. The light gray color represents those who are denied loans as they would default anyhow. The dark gray color represents those who are denied loans but won't default. The light blue and orange colors represent those who are granted loans but default. The dark blue and orange colors represent those who are granted the loan and pay back. Examining these distributions between groups is important and varied, applicable in real life. For instance, in the United States, wealth inequality may affect the ability of different race and gender groups to repay a loan. This is a result of historical discrimination. Here's another example, women tend to live longer than men. Therefore, would it make sense to consider insurance rates and policies in the same way for both men and women? These two examples highlight the concept of group unaware or a simple threshold to set the same value across all groups. Looking back at the plots on the screen, the blue group is now being given more loans overall. But the orange group has been given fewer loans among people who would pay back the loan. The orange group is at a disadvantage. This is because the real differences that exist between the two groups are being ignored. Now let's consider another fairness definition, demographic parity. Let's have the bank issue the same percentage of loans to both blue and orange groups. What happens is that the ratio of loans given to each group becomes the same. But among the people who would pay back a loan, the blue group is now at a disadvantage. This is because the model only looks at loans given, not rates, at which loans are paid back. This demographic parity strategy is helpful in a use case where having equal final percentages of groups is important. Here's a third fairness definition, equality of opportunity. This means offering the same percentage of loans to those who can pay them back in both groups. In other words, set the same recall. Now you can see that among people who would pay back a loan, blue and orange proofs to be equally well. Throughout this loan example, we looked at different constraints to determine the level of fairness when issuing loans to two different groups. Each of these constraints has different mathematical properties and different outcomes. One is not necessarily better than the other. You also can't satisfy all properties and objectives at the same time. These are not the exhaustive list of the constraints. Which constraint you choose depends a lot on the context of the ML use case. The Aequitas Fairness Tree guideline offers you guidelines about which metrics you should prioritize on a specific use case. The decision tree includes conditions such as if positive predictions assist or punish people. If your system can intervene with most people with need, or only a small fraction.2. Let's practice!
Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.