Analyzing the A/B test results
1. Analyzing the A/B test results
Great Work! Now that our A/B test parameters have been determined, all that is left is for the test to run and for us to analyze the results.2. Analyzing A/B test results
In this chapter we will explore the techniques used to analyze a test as well as some expansions on this topic that will further your analysis.3. Evaluating our paywall test
We have now run our discussed A/B test for the required amount of time. We will evaluate the two paywalls by measuring the impact on the general purchase rate from users encountering this paywall.4. Test results data
Here we have our demographics data and a dataset ab_test_results dot csv. This contains data similar to that from the previous chapter only now limited to the set of results from our A/B test. Additionally, there is a new `group` column that contains either `V` or `C` representing if the user was in the variant or control group.5. Confirming our test results
Before proceeding it is worth taking a moment to confirm that our test was administered correctly. While you may not always be in a position to determine this directly, you can ensure that the data is sufficiently random.6. Are our groups the same size?
Here we do this by segmenting our data by the `group` column and comparing the size of each group. They are roughly comparable.7. Do our groups have similar demographics?
Further we can break out by the relevant demographics: country, device, and gender, and compare the numbers of those two groups to make sure there is no obvious bias in our assignment procedure. This all looks right, so let’s move on.8. Test & control group conversion rates
Now let’s find the mean conversion for each of our two groups. This is similar to our previous calculations, only now we want to split by the `group`. We first group by `group` and then we can find the total number of purchasers, and paywall viewers in each group. Finally, we can divide these two groups and see our final results of 3 point 4% percent for the control and 4 point 19 percent for the test, which appears substantially higher.9. Is the result statistically significant?
The next step is to determine whether this difference is statistically significant. That is, measuring if the values differ more than would be expected due to randomness. If this is the case, we can reject the null hypothesis.10. p-values
To do this we must calculate the p-value. The p-value is the probability of observing a value as or more extreme than the observed value under the Null hypothesis. If this value is low, then it means either our power is low or there is a low probability of observing this value if the Null hypothesis is true. The formula is not important to know at this point. Sufficed to say that is connected in the way described above to the concepts of power and confidence level discussed in the previous chapter, and understanding it conceptually is what is important.11. Interpreting p-values
Standard practice is to use the following guide to accept or reject based on the p-value. It is worth noting that the p-value and hypothesis testing in general are controversial topics and used with varying degrees of success. This debate is out of the scope of this course but worth following up on if you are interested.12. Next steps
Now that we have examined our data and confirmed it looks correct, and laid this foundational groundwork, we will confirm our results and explore how to further contextualize them in the next chapter.13. Let's practice!
Great work, and good luck!Create Your Free Account
or
By continuing, you accept our Terms of Use, our Privacy Policy and that your data is stored in the USA.