1. Understanding statistical significance
Awesome work! Picking up where we left off, we have our A/B test results as well as a conceptual understanding of how to unpack whether or not our observed difference is statistically significant.
2. Revisiting statistical significance
Returning to statistical significance for one last time it is often helpful to look at a picture of what we are describing. Here is the distribution of our data under the Null hypothesis. The line is the conversion rate difference we observed and the inverse of it. The p-value gives the probability of being at or past those lines in the tails, thus a value as or more extreme than our observation.
3. p-value Function
Here is a function that finds the p-value. It takes in our test and control conversion rates and the size of each group. It returns the p-value.
4. Calculating our p-value
Calculating it for our data we see that we have an extremely small p-value. Referring back to the table in the previous lesson, we see that this result would be considered statistically significant. The reason for this is our lift achieved is much larger than what we anticipate in our preparations.
We are off to a good start! Now let’s work to provide some context for our result.
5. Finding the power of our test
First, we should find our power. This will help make sure that our estimate is not off and ensure we are not under-weighting the risk of a false negative.
We can find this by using the get_power() function from earlier.
Calculating the result, we have an extremely high power, which is also influenced by our high percentage lift.
6. What is a confidence interval
We should also provide confidence intervals for our estimate.The confidence interval contextualizes the confidence we have in our estimation process. Specifically, if we provide a 95% confidence interval we are saying that if we had a series of independent experiments and for each estimated an unrelated parameter and a 95% confidence interval, then 95% of the intervals will contain the true parameter. The important thing to note is that the parameters, such as the true conversion rate, are not random, it is the interval that is random.
7. Confidence interval calculation
More details are out of the scope of the course, but calculating the interval for a test such as this is straightforward. We know from previous discussions that our estimated parameter follows a Normal distribution, and from our estimated parameter we can calculate the variance of that distribution. An X% confidence interval then is just the upper and lower bounds that contain X% of the probability density around the mean between them, which we can easily find by using similar to those we have used before.
8. Confidence interval function
Here is a function for calculating the confidence interval, `get_ci()` which takes in the estimated mean of the test and control, as well as the sample sizes of each, and finally the desired confidence interval. It returns a list with the first entry being our lower bound and the second our upper bound.
9. Calculating confidence intervals
Applying this to our own data, we can see that we have a lower bound of 0 point 005 and an upper bound of 0 point 010. Reporting these values along with our estimated difference is much better than just reporting the point estimate. This interval provides context to your audience as to how certain you are about your result and allows them to make more informed decisions based on the data.
10. Next steps
In the next chapter we will build on this theme of contextualizing the results and explore how to visualize the data to even better present it to your audience.
11. Let's practice!
Good luck on the exercises!