1. Statistical errors
Any time you perform a hypothesis test, there is a chance that you've made an error. I'm not talking about an error in your measurements or an error in your code - those happen too. I'm talking about statistical errors, errors that are baked in to the procedure of a hypothesis test.
2. Statistical errors
The first such error can occur in situations where the null hypothesis that you're testing is indeed true.
3. Statistical errors
In this case, you would be right to retain the null hypothesis,
4. Statistical errors
and you'd be committing what's called a type I error if you rejected it.
5. Statistical errors
6. Statistical errors
What is the probability that you will reject a true null hypothesis?
7. Statistical errors
Consider the situation where you're testing a null hypothesis that the difference in two proportions is zero.
8. Statistical errors
You can use either permutation or the normal approximation to find the null distribution...
9. Statistical errors
the distribution of the differences in p-hats that you might observe in a given sample if the null is true.
10. Statistical errors
If the sample statistic calculated from your data falls into the tails of this distribution, it is considered too unlikely to have happened under the null hypothesis, so you would reject the null.
11. Statistical errors
This is formalized by checking to see if the statistic falls within the area of the tails that account for some proportion, alpha, of the total distribution. It's most common to set alpha to point-0-5.
12. Statistical errors
Since you will reject the null hypothesis anytime you observe a statistic in this region, in setting alpha you're actually setting your type I error rate.
13. Statistical errors
If you use alpha equal to point-0-5, you're resigning yourself to incorrectly rejecting the null hypothesis 5% of the time.
14. Statistical errors
There's a second type of error to be aware of when conducting hypothesis tests, one that occurs when in fact H0 is false and some other model we'll call HA is the true model.
15. Statistical errors
One hopes that after looking at your data and computing a p-value you would decide to reject H0.
16. Statistical errors
If you fail to reject H0, you've committed what's called a type II error.
17. Statistical errors
To visualize a type II error, you have to start by realizing that H0 is no longer the model that is generating your data. Now, it's coming from HA.
18. Statistical errors
However, you are using the same thresholds, those dotted gray lines, for the reject and fail-to-reject regions, so you will retain H0 whenever you observe a statistic in this region.
19. Statistical errors
The probability of that event corresponds to the area shaded here, which is usually given the Greek letter beta.
The other potion of the area under this curve, 1 - Beta, is called the power. The statistical power of the test is the probability that it will reject H0 if in fact it is false. Said another way, it's your power to detect an effect if it exists. This is an extremely important property of a statistical test. In general, you can affect and increase in the power in two ways. First is to increase the size of the reject region by increasing alpha. This has a tradeoff though: now you're increased the type I error rate. The second is to increase your sample size. This will have the effect of shrinking the variance of these curves, which would increase the area of the curve that is in the reject region.
20. Let's practice!
Ok, with the definitions of power, type I, and type II errors in hand, let's practice!