1. Bootstrapping
Previously we saw non-parametric and parametric tests. In this video we'll introduce a non-parametric analogue of a confidence interval, and explain how it differs for inference from the confidence interval we've already learned.
2. Bootstrapping
Bootstrapping, which is sampling with replacement, is a common statistical technique. We start with a dataset, randomly choose a sample, write it down, put it back in the data, and repeat. This is called a bootstrapped sample.
We can imagine having a group of people and asking them to rate their experience at an event on a scale from zero to ten. We ask someone, record their answer, and go ask again. It's possible we may ask the same person twice, but that's ok.
3. Non-parametric confidence interval
For the purposes of inference, our primary interest in bootstrapping is creating bootstrapped confidence intervals, which is a non-parametric analogue of the confidence intervals that assume normality. A bootstrapped confidence interval is formed by taking a bootstrap sample, computing some test statistic, such as a mean, recording this, and repeating this process many times. In doing so we build an empirical distribution of outcomes.
4. SciPy example
Bootstrapped confidence intervals are commonly used to compute a confidence interval for the mean. However, bootstrapping can be used with any statistic we like! In this case, we'll use the range.
Let's load data on employees from the City of Austin, TX. Suppose we put employees into groups of ten. How much would the experience levels vary within these groups? Would we expect most employees to be around the same experience, or would it be reasonable to expect drastic swings in experience levels?
We'll measure this by randomly selecting ten employees and finding the difference between the most and least experienced employees by looking at their years of employment. In this one example we saw a spread of seven years. We could repeat this many times and form a sampling distribution. By then taking the middle 95 percent of outcomes, we could form a bootstrapped 95% confidence interval.
5. SciPy example
While we could do this whole process by hand, SciPy can do it for us! To do so it needs our data, as well as a function which computes the statistic in question. Here we'll code up that function. It also requires the data to be a tuple, so we'll form that here.
We've also added a few keyword arguments. Setting vectorized to False makes it easier to write our own statistics. Setting n_resamples tells SciPy how many samples to take to form the sampling distribution.
This returns the confidence interval for the experience ranges, as well as the standard error.
6. Normal versus bootstrapped confidence intervals
Bootstrapped confidence intervals are not better or worse than confidence intervals that assume normality, just different. Normal confidence intervals require data to be normally distributed, make their computations based solely on the mean and standard error, make valid inference only for normally distributed data, but are very fast to compute.
Bootstrapped confidence intervals, on the other hand work for any data, are computed directly from the data by resampling, allow for valid inference with any data, but are much slower to compute.
7. Use cases for bootstrapping
Because of these differences, if our data is normally distributed, a normal confidence interval should be our first choice. However, when working with non-normal data, such as ranked data or skewed data, bootstrapped confidence intervals are a great choice. In addition, normal confidence intervals sometimes returns nonsensical values, like negative years of experience. Finally, bootstrapped confidence intervals work with any statistic we like.
8. Let's practice!
Now that we see how normal and bootstrapped confidence intervals compare, let's practice!