Power Analysis - Part II

Previously, we simulated one instance of the experiment & generated a p-value. We will now use this framework to calculate statistical power. Power of an experiment is the experiment's ability to detect a difference between treatment & control if the difference really exists. It's good statistical hygiene to strive for 80% power.

For our website, suppose we want to know how many people need to visit each variant, such that we can detect a 10% increase in time spent with 80% power. For this, we start with a small sample (50), simulate multiple instances of this experiment & check power. If 80% power is reached, we stop. If not, we increase the sample size & try again.

For the time_spent random variables, set size as tuples such that shape is sample_size \(\times\) sims.
Calculate power as a fraction of p-values less than 0.05 (statistically significant).
If power is greater than or equal to 80%, break out of the while loop. Else, keep incrementing sample_size by 10.

Exercise

Power Analysis - Part II

Instructions