1. A/B testing & segmentation
One of the most common pitfalls in A/B testing is assuming that a treatment equally affects everyone in a population.
2. Don't forget about segmentation!
Just like with any other kind of marketing, some treatments are particularly effective on users of a specific engagement level, age, race, or any other of a number of factors. It is important to break down results by various demographics in order to obtain a holistic understanding of the impact of the test. Not all customers are alike!
3. Personalization test segmented by language
Much like in the previous chapters, the primary challenge of segmentation is figuring out the best way to avoid repetitive work. In this lesson, we will use for loops to calculate lift and statistical significance across multiple segments of users.
We begin by looping through all languages in the language_displayed column using numpy's unique() function and printing the language we are evaluating in this loop for reference.
4. Isolate the relevant data
First, we need to remake our dataset each time to isolate the data for only the selected language in each loop.
5. Isolate subscribers
Then, we ensure each user and variant has only one subscription outcome by using the groupby() and max() methods as we did in the first lesson of this chapter.
6. Isolate control and personalization
Next, we unstack the DataFrame and create a Series of outcomes for both the control and the personalization variants by dropping all missing values.
7. Full for loop
Finally, we can use the lift() and ttest() functions from the previous lesson to print the conversion rate and statistical significance for each language.
8. Results
Our for loop will run all the calculations for each unique language in the language_displayed column, and print the results.
As you can see, the test performed very well among English and Spanish speakers, while the other language's results are not statistically significant. These are the kinds of differences across groups that are crucial to keep an eye out for.
9. Let's practice!
Go ahead and practice the same!