Multiple comparisons and corrections
1. Multiple comparisons and corrections
Often, when we conduct several inferences or tests simultaneously, the probability of detecting an effect that is not in fact present increases. Let's explore what this means.2. Getting the result we want
Let's start with an example. Suppose we wanted to see if people liked a new product our company was developing. We start by asking one person and they say no. We then ask another person and they also say no. We repeat this until eventually one person says yes. We then go back to your boss and say "People like our product!" Clearly this is silly, as we're taking samples until we get the result we want. But this is precisely the multiple comparisons problem.3. Real-world example
Of course, we wouldn't actually do that experiment. But imagine that we were trying to determine what factors increased sales, and we sent a survey to customers with fifty questions. We might take the response to each question one-by-one and see if it seems to correlate with, or help explain, the choice of customer to buy or not buy our product. We may also look at pairs of variables, or maybe even more. Imagine how many comparisons we're making this way! In essence, we're repeatedly looking at the data and making comparisons until we find the result we want. This is called the multiple comparisons problem.4. The role of alpha in inference
There's nothing wrong with making multiple comparisons. The problem however arises from alpha. Remember that in a hypothesis test our p-value measures how likely a result is to occur at random. If a highly unlikely result occurs, it raises our suspicion, and makes us consider if something in our experiment caused that result. Such as a very high number of people getting better after taking our treatment, which causes us to suspect that our treatment is what did it. Alpha is the cutoff for how strong this evidence needs to be before we reject the null. But when we make many comparisons, we increase the probability of finding results which demonstrate a strong relationship, even if it's only due to random chance!5. Correcting for multiple comparisons
A common fix is to make alpha smaller! This is precisely what the Bonferonni-Holm correct, or Bonferonni correction for short, does. It takes our value of alpha, normally five percent, and divides it by the number of comparisons we are making. For example, suppose we wanted to compare all fifty variables in our data against a target variable of our customer's purchase decision. Since we will be making fifty comparisons, the Bonferonni correction asks us to divide our choice of alpha by fifty. This gives a corrected alpha of zero-point-zero-zero-one, a reduced probability of detecting a significant result by chance.6. Let's practice!
The best way to understand this relationship is to dive into the data. So let's practice!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.