1. A/B testing for marketing
In this lesson, we discuss how A/B testing is used within marketing departments.
2. What is A/B testing?
A/B testing refers to a randomized experiment which evaluates which variant performs better.
In order for our tests to have meaning, we must have a clear control. The control should be something that currently exists and is running in production.
Each variant in the test should have only one major change from the control; otherwise, it will be impossible to parse what led to the change in your key metrics.
Prior to beginning a test, you must develop a hypothesis and determine which metric you are trying to impact. Always set key metrics ahead of running the test. It's easy to redefine success in retrospect, especially if you are under pressure to find a positive result. If you document success metrics ahead of time, you can maintain clarity around the success of the test.
3. Testing allows us to understand marketing impact
A big benefit of running A/B tests is we can be confident that the increase in the relevant metrics was due to the action we took in the test.
4. How long does a test need to run?
Your stakeholders will likely ask you how long a test needs to run.
If a decision is needed by the end of the week, but a test would require two weeks to reach statistical significance, you should use a different methodology for this decision.
Testing is popular because it can provide a definitive answer to controversial business questions, but it is only effective if tests are run properly.
5. Personalized email test
The focus of this chapter will be the A/B test that was run where half the emails were generic upsells to our product while the other half contained personalized messaging around individual usage of our site.
Note that we will take a high-level look into how A/B tests are conducted in marketing departments. If you are interested in developing a more robust understanding of A/B testing and statistics, there are other excellent DataCamp courses where you can learn about these topics in depth.
6. Test allocation
Before we can begin assessing the impact of the test, we must ensure the test was executed correctly. The variant column contains the group each user was allocated to.
We can do so by looking at how many people were allocated to the control and personalization variants.
The code should look similar to how we sliced DataFrames and used groupby in previous lessons, except this time, we are counting the unique number of users who received each variant of email.
Next, let's plot the results. Since we are comparing two groups, we'll use a bar chart.
7. Allocation plot
Allocation is relatively even, but not exactly the same. This will typically be the case. If you're worried allocation has gone wrong, there are statistical tests to determine the likelihood that the difference in allocation is due to random chance, but we will not explore that in this lesson.
In this case, we can proceed with the assumption that there were no issues in the randomization process.
8. Setting up our data to evaluate the test
First, we ensure each user and variant has only one subscription outcome by using the groupby() and max() methods. We use the max() method on the 'converted' column since it's a boolean, and if there are multiple rows with False and True values, we want to consider the row where the user was converted, that is, True. Next, we unstack the DataFrame.
9. Setting up our data to evaluate the test
Finally, we create a Series of outcomes for both the control and the personalization variants. In this case, the series has a row for each user in the test which equals "True" if the user subscribed and "False" otherwise. We can use dropna() on each Series to only include conversion outcomes for all users in each variant.
10. Conversion rates
We can then calculate the conversion rate by taking the mean of each Series. Is this difference significant?
11. Let's get testing!
We will find out in the next lesson. For now, its time for you to analyze the personalization test!