1. Why 0.05?
In the previous exercises, you determined quantiles of the null distribution. The cutoff values allow you to know how big or small the observed statistic should be in order for you to reject the null hypothesis.
The choice of (point) 05 for the quantile cutoff is somewhat arbitrary. However, the reasons to use it are historical, ingrained in much of science, and somewhat intuitive.
2. RA Fisher
In 1929, RA Fisher wrote that a result should be judged significant if it was not likely to happen by chance more than once in twenty trials.
3. RA Fisher
However, Fisher went on to remark that 1 out of 20 is arbitrary, albeit convenient.
4. RA Fisher
And should not be used as a way to determine truth.
Indeed, the test of significance should be used to figure out which research to replicate
5. RA Fisher
and which research should not be pursued.
We should heed Fisher's words and understand that a 5% cutoff level is historical and arbitrary. However, it is also worth considering that any given level of significance is also personal and subjective.
As it turns out, the 5% cutoff value is consistent with what many of us consider to be unusual.
6. Personal level of significance
Consider a situation where you as the researcher are trying to decide whether a coin is fair or not. Would you think that flipping one head is evidence that the coin is unfair? Probably not.
7. Personal level of significance
What about two heads?
8. Personal level of significance
Three heads?
9. Personal level of significance
At what number of heads in a row would you start thinking that maybe this coin isn't actually a fair coin?
10. Personal level of significance
For many of you, four or five heads in a row would seem unusual. Interestingly, four or five heads in a row correspond to probabilities that are very close to 0 (point) 05. Some of you might need six or seven heads in a row before you start to doubt the fairness of the coin.
11. Degree of skepticism
The notion of statistical significance is similar using a cutoff of 0 (point) 01 instead of 0 (point) 05 indicates a higher degree of skepticism that an observed result is not simply due to random noise. Which is to say that the cutoff of (point) 05 is personal and subjective, but not meaningless.
As Fisher says, it is the significant results which should lead to further investigation. And depending on the research setting and an appropriate level of skepticism, you can decide which hypotheses to pursue further.
12. Let's practice!
OK, now it's your turn to practice what you've learned.