Anova: Multiple comparisons

An Anova checks for an overall difference in groups. However, it does not give any indication of which groups differ. Under most circumstances you would therefore like to follow up an Anova with one or multiple post-hoc analyses. However, one has to be careful to control the family-wise error rate when following up an anova with multiple post hoc analyses.

The family-wise error rate is the probability of making a type 1 error. Normally our type 1 error rate is denoted by \(\alpha\) which we usually keep at 0.05. This means that there is a 5% chance of falsely rejecting the null hypothesis. If we would do multiple analyses, say 10, our family-wise error rate increases. This is because for each individual test we do, there is a 5% chance that we falsely reject the null hypothesis and this adds up. If we don't control for multiple testing, the family-wise error rate \(1 - (1 - \alpha)^m\) where m is the number of tests that we do.

If we do 10 tests all with a significance level \(\alpha\) of 5%, what would become our family-wise error rate? Use the formula displayed above and fill in the gaps. What does this family-wise error rate mean in this context?