Simulation for Power Analysis
1. Simulation for power analysis
Now we turn our attention to yet another powerful application of simulation in the analysis of statistical power. I have personally used simulation multiple times for this purpose and have been able to spot underpowered tests before they are run, saving a lot of time and resources.2. What is power?
So, what is statistical power? I'd encourage you to read the Wikipedia article on statistical power or the article linked in the slide to refresh your concepts. Formally, it is defined as the probability of correctly rejecting the null hypothesis given a true alternative hypotheses. More intuitively, you can think of it as the ability of the test or experiment to detect an effect or difference between treatment and control if it really exists. Power depends on the sample size, significance level alpha and the effect size. Typically, 80% power is recommended as good statistical hygiene.3. News media website
A power analysis is performed before running an experiment and is typically used to determine the sample size needed to detect our hypothesized effect size. Once we know the data generating process, this task can be easily achieved using simulation. Suppose you are in charge of a news media website and want to increase the amount of time users spend on your site. You are testing a new feature that loads pages much faster and want to determine the sample size required to detect a 10% increase in time spent on your site. Let's see how simulation can help.4. Simulation for power analysis
Suppose you find that the time users spend on your website is normally distributed and you simulate time spent by control users. You can then simulate time spent by treatment users by increasing the mean by the effect size, say 10%. The number of samples in both cases is initially fixed at a small number, say 50. Now just run a t-test or a permutation test from our earlier lesson, and get a p-value. Repeat this process a number of times and calculate the fraction of times this p-value was less than 0.05. This is your power. If power is less than 80%, increase the sample size and rerun the process. If it is greater than or equal to 80%, stop. You've found a sample size that gives you 80% power to detect a 10% change at a significance level of 0.05.5. Let's practice!
In my work, the data generating process can sometimes get quite involved with multiple levels of complexity. In such cases, instead of relying on distributional assumptions, I'll just analyze the power using simulation and it's proved very helpful time and again. Let's code up this algorithm in the next series of exercises.Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.