Get startedGet started for free

Power Analysis - Part II

Previously, we simulated one instance of the experiment & generated a p-value. We will now use this framework to calculate statistical power. Power of an experiment is the experiment's ability to detect a difference between treatment & control if the difference really exists. It's good statistical hygiene to strive for 80% power.

For our website, suppose we want to know how many people need to visit each variant, such that we can detect a 10% increase in time spent with 80% power. For this, we start with a small sample (50), simulate multiple instances of this experiment & check power. If 80% power is reached, we stop. If not, we increase the sample size & try again.

This exercise is part of the course

Statistical Simulation in Python

View Course

Exercise instructions

  • For the time_spent random variables, set size as tuples such that shape is sample_size \(\times\) sims.
  • Calculate power as a fraction of p-values less than 0.05 (statistically significant).
  • If power is greater than or equal to 80%, break out of the while loop. Else, keep incrementing sample_size by 10.

Hands-on interactive exercise

Have a go at this exercise by completing this sample code.

sample_size = 50

# Keep incrementing sample size by 10 till we reach required power
while 1:
    control_time_spent = np.random.normal(loc=control_mean, scale=control_sd, size=(____,____)))
    treatment_time_spent = np.random.normal(loc=control_mean*(1+effect_size), scale=control_sd, size=(____,____))
    t, p = st.ttest_ind(treatment_time_spent, control_time_spent)
    
    # Power is the fraction of times in the simulation when the p-value was less than 0.05
    power = (p < 0.05).sum()/____
    if ____: 
        ____
    else: 
        ____ += ____
print("For 80% power, sample size required = {}".format(sample_size))
Edit and Run Code