The necessity of post-hoc tests

In the previous chapter, the F-test showed a significant effect somewhere among the groups. However, it did not tell you which pairwise comparisons are significant. This is where post-hoc tests come into play. They help you to find out which groups differ significantly from one other and which do not. More formally, post-hoc tests allow for multiple pairwise comparisons without inflating the type I error.

What does it mean to inflate the type I error?

Suppose the post-hoc test involves performing three pairwise comparisons, each with the probability of a type I error set at 5%. The probability of making at least one type I error is then equal to \(1-(no\:type\:I\:error\:\times\:no\:type\:I\:error\:\times\:no\:type\:I\:error)\). If, for simplicity, you assume independence of these three events, the maximum familywise error rate becomes \(1 - (0.95\times0.95\times0.95) = 14.26\%\). In other words, the probability of having at least one false alarm (i.e. type I error) is 14.26%.

What is the maximum familywise error rate for the working memory experiment, assuming that you do all possible pairwise comparisons with a type I error of 5%?