1. Assumptions in hypothesis testing
Each hypothesis test we've seen so far makes assumptions about the data. It's only when these assumptions are met that it is appropriate to use that hypothesis test.
2. Randomness
Whether it uses one or multiple samples, every hypothesis test assumes that each sample is randomly sourced from its population.
If we don't have a random sample, then it won't be representative of the population.
To check this assumption, we need to know where our data came from. There are no statistical or coding tests we can perform to check this.
If in doubt, ask the people involved in data collection, or a domain expert that understands the population being sampled.
3. Independence of observations
Tests also assume that each observation is independent. There are some special cases like paired t-tests where dependencies between two samples are allowed, but these change the calculations, so we need to understand where such dependencies occur.
As we saw with the paired t-test, not accounting for dependencies results in an increased chance of false negative and false positive errors.
Not accounting for dependencies is a difficult problem to diagnose during analysis. Ideally, it needs to be discussed before data collection.
4. Large sample size
Hypothesis tests also assume that our sample is large enough that the Central Limit Theorem applies, and the sample distribution can be assumed to be normally distributed. Smaller samples incur greater uncertainty, which may mean that the Central Limit Theorem does not apply and the sampling distribution might not be normally distributed.
The increased uncertainty of a small sample means we get wider confidence intervals on the parameter we are trying to estimate.
If the Central Limit Theorem does not apply, the calculations on the sample, and any conclusions drawn from them, could be nonsense, which increases the chance of false negative and false positive errors.
How big our sample needs to be to be "big enough" depends on the test.
5. Large sample size: t-test
For one sample t-tests, a popular heuristic is that we need at least thirty observations in our sample.
For the two sample case or ANOVA, we need thirty observations from each group. That means we can't compensate for one minority group sample by making the majority group bigger.
In the paired case, we need thirty pairs of observations. Sometimes we can get away with less than 30 in each of these tests; the important thing is that the null distribution appears normal. This is often the case at around 30 and that's the reason for this somewhat arbitrary threshold.
6. Large sample size: proportion tests
For one sample proportion tests, the sample is considered big enough if it contains at least ten successes and ten failures. Notice that if the probability of success is close to zero or close to one, then we need a bigger sample.
In the two sample case, we require ten successes and ten failures from each sample.
7. Large sample size: chi-square tests
The chi-square test is slightly more forgiving and only requires five successes and five failures in each group, rather than ten.
8. Sanity check
One more check we can perform is to calculate a bootstrap distribution and visualize it with a histogram. If we don't see a bell-shaped normal curve, then one of the assumptions hasn't been met. In that case, we should revisit the data collection process, and see if any of the three assumptions of randomness, independence, and sample size do not hold.
9. Let's practice!
Time to check some assumptions!