1. Post-hoc testing
So far we've introduced ANOVA as a method for comparing many means to each other concurrently. Finding a statistically significant result at the end of an ANOVA however only tells us that at least one pair of means are different, but not which pair of means are different. In this video we set out to answer this follow up question.
2. Which means differ?
And when doing so we're going to discuss how to control the Type I error rate that would be inflated by doing many pairwise tests in the quest for identifying the groups whose means are significantly differnt from each other.
Remember that to determine whether two means are different from each other we use t-tests, with each test you incur a possibility of a Type 1 error. The probability of committing a Type 1 error is the significance level of the test, which is often set at 5%.
When you do multiple tests to compare each possible pairing of groups to each other, then you inflate your overall Type 1 error rate, which is an undesirable outcome
However there's a simple solution: use a modified significance level, that is, a lower significance level, for each pairwise test, so that the overall Type 1 error rate for the series of tests you have to do can still be held at a low rate.
3. Multiple comparisons
Testing many pairs of groups is called multiple comparisons
and a common modification we use when doing multiple comparisons is the Bonferroni correction which uses a more stringent significance level for each of the pairwise tests
more specifically, we adjust our alpha by the number of comparisons we have to do
the Bonferroni corrected significance level can be calculated as the original significance level divided by the number of pairwise comparisons to be carried out. This number can be calculated as k times k-1 divided by 2, where k is the number of groups in the ANOVA.
4. Pairwise comparisons
There are a couple other considerations for these multiple comparisons following anova.
First is related to the constant variance condition of ANOVA. Since to do the ANOVA in the first place this condition must be satisfied, we need to re-think the standard error and the degrees of freedom to be used in the multiple comparisons tests.
And of course, since we now have a new modified significance level, we compare the resulting p-values to this significance level.
5. Let's practice!
Now it's your turn.