1. Permutation tests
Permutation tests are a type of non-parametric test that allows us incredible flexibility. The ability to make inference in a broad range of situations makes permutation tests a great tool to have.
2. Permutation tests
Permutation tests show up in many scenarios, so let's review. Suppose we are testing a change to our product and want to see if customer satisfaction goes up. We give one group of customers the new version and one the old version and measure the difference in satisfaction. We see a small difference, but want to know if this difference is significant.
3. Permutation tests
A permutation test takes the customers, randomly shuffles them, and then computes the difference in the satisfaction ratings. Just due to random choice we'd expect to see some slight differences. By repeating this "shuffle and observe" process many times, we can understand what sorts of outcomes are expected just due to random chance alone. That allows us to take the situation we're interested in, namely the new versus old version, and see if the outcome we observed looks like those from random shuffling, or if it's a significantly larger difference.
4. Permutation tests in SciPy
We can perform permutation tests in SciPy using the permutation_test function. We first gather our data and package it together into a tuple. Note that even if we only have a single dataset, we still need to write it in a tuple.
Next we define a function which computes our test statistic. In this case it's the difference in mean satisfaction scores between the two groups.
Finally, we feed all of this information into the permutation_test function. The n_resamples argument tells permutation_test how many times the data should be shuffled to build the null distribution, meaning the distribution of outcomes that occur purely by chance. Setting vectorized to False makes the code slightly slower, but also simplifies the statistic function. Finally, we choose whether our alternative hypothesis is that the observed value is greater than the null distribution, less than the null distribution, or a two-sided test, which looks for a difference.
5. Permutation tests for correlation
Note that permutation tests go well beyond comparing means! The power of a permutation test is that, as long as we can write a function defining the statistic we're interested in, we can test it!
Let's write a permutation test which compares the correlation between two data sets. Here we see two data sets that have a correlation coefficient of about zero-point-zero-eight. Should we conclude these data sets are correlated? Let's conduct a permutation test to find out!
6. Permutation tests for correlation
We group together our data and write a function for our statistic, just like before. Note that the statistic function must return just the test statistic, and not other information like a p-value. That's why we access the zeroth entry of the return value. We then conduct a two-sided test using this data. The p-value returned from this test is not significant at the five percent level, meaning the correlation could just be due to random chance.
7. Permutation tests vs. bootstrapping
Permutation tests and the bootstrapping technique we saw earlier are similar, but slightly different. Permutation tests build a null distribution by randomly shuffling the data, while bootstrapping builds a probability distribution by randomly sampling the data. Permutation tests test for the significance of an outcome, while bootstrapping creates a confidence interval showing the most likely outcomes.
8. Let's practice!
Now that you see the power of permutation tests, let's practice!