Power Analysis - Part II
Previously, we simulated one instance of the experiment & generated a p-value. We will now use this framework to calculate statistical power. Power of an experiment is the experiment's ability to detect a difference between treatment & control if the difference really exists. It's good statistical hygiene to strive for 80% power.
For our website, suppose we want to know how many people need to visit each variant, such that we can detect a 10% increase in time spent with 80% power. For this, we start with a small sample (50), simulate multiple instances of this experiment & check power. If 80% power is reached, we stop. If not, we increase the sample size & try again.
This exercise is part of the course
Statistical Simulation in Python
Exercise instructions
- For the
time_spent
random variables, setsize
as tuples such that shape issample_size
\(\times\)sims
. - Calculate
power
as a fraction of p-values less than 0.05 (statistically significant). - If
power
is greater than or equal to 80%,break
out of the while loop. Else, keep incrementingsample_size
by 10.
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
sample_size = 50
# Keep incrementing sample size by 10 till we reach required power
while 1:
control_time_spent = np.random.normal(loc=control_mean, scale=control_sd, size=(____,____)))
treatment_time_spent = np.random.normal(loc=control_mean*(1+effect_size), scale=control_sd, size=(____,____))
t, p = st.ttest_ind(treatment_time_spent, control_time_spent)
# Power is the fraction of times in the simulation when the p-value was less than 0.05
power = (p < 0.05).sum()/____
if ____:
____
else:
____ += ____
print("For 80% power, sample size required = {}".format(sample_size))