1. What is A/B testing?
Hi, my name is Moe and I have been working in the fields of data science and analytics over the past 10 plus years in various settings including research, academia, and industry. Over my career I've had the pleasure to consult and build analytics and experimentation functions for several fortune 500 companies, and I hope that this course in AB testing can offer you the guidance you need to start your experimentation journey.
2. Intro to A/B testing
Let's start with what an AB test is. An AB test is an experiment used to determine which version of an online experience performs better based on some metric(s) such as signup rate. We do so by presenting each version to users at random and analyzing the results.
3. To A/B test or not to test?
But before we jump into testing, it's important to recognize that AB tests may not always be a good idea.
They can be used to optimize conversion rates, detect the impact of releasing a new feature, evaluate the value of advertising, and many more use-cases. But we may need to consider other qualitative methods when faced with scenarios such as
if we have limited traffic that limits our confidence in the results,
if there is no clear hypothesis to be tested,
if there are ethical consequences to our tests,
or if the opportunity cost of blocking a certain feature from a large segment of users is high.
4. A/B testing fundamental steps
AB tests generally consist of
specifying the goal of the test and having the designs or user experience ready,
selecting a random sample of users to enroll in the experiment,
randomly assigning those users to a control group, which is usually the current state that we are looking to improve upon, and a treatment group, which receives the new feature, landing page, ad, or whatever the change may be,
then, logging user actions for each group and computing metrics of interest,
and finally, testing for statistically significant differences to make our decision.
5. Value of randomization
All of these steps are important but the concept of randomization is specifically critical to the success of AB tests. It allows us to generalize our results from a small number of users to the rest of the population, minimizes the chances of having inherent bias in the groups, and allows us to isolate the effects of our changes.
Thanks to randomization, the only difference between the groups is the change that we've introduced. This allows us to conclude that if there is a statistically significant difference in the metrics between the groups, then this difference is attributed only to that change and not other variables.
6. Python example of random assignment
Let's illustrate this with a Python example on the checkout dataset which is an experiment ran on different checkout page designs to evaluate their impact on users' purchases. Examining the checkout DataFrame using the pandas method dot info, we can see that it contains 9000 rows and six columns.
7. Python example of random assignment
We can analyze the distribution of the 'gender' column, using the value_counts method and passing 'True' to the 'normalize' argument. Assuming the DataFrame represents our entire population, the gender split seems to be 50/50 male to female. If we take a random sample of 3000 observations using the pandas method dot sample and analyze the gender distribution, we also see that it's 50/50. Not surprisingly, this will be the case for other attributes as well which allows us to generalize our inferences to the rest of the population.
8. Python example of random assignment
Furthermore, since the users were randomly assigned to each of the variants in the experiment, passing the checkout page column to the pandas groupby method, we can see that the ratio remains the same between the three checkout page variants A, B, and C. This demonstrates the power of random assignment in reducing inherent biases between groups in the data.
9. Let's practice!
Now that we've learned some of the basics of AB testing, let's practice these concepts in the following exercises.