1. Introduction to A/B testing
Great job on those exercises! Now we will transition our discussion to the topic of A/B testing. In the past two chapters we looked at indirect ways of observing user behavior. Now we will explore a set of tools that allows you to truly uncover causal relationships in user actions.
2. Overview
We will start off with an introduction to A/B testing, and then move to an in depth overview of the knowledge and skills you will use to design, understand, and analyze an A/B test.
3. A/B test: an experiment where you...
An A/B test is an experiment in which you test two different values of the same variable against one another to determine which one is “better” by evaluating their performance among two randomly assigned groups of real users.
4. Control and treatment groups
Often, we will be testing the current state, against a variant, in which case our two variables are called the control and treatment respectively.
5. A/B Test - improving our app paywall
Consider a scenario where the initial subscription paywall in our meditation app says:
“I hope you enjoyed your free-trial, please consider subscribing”
and we want to see whether, the phrase
“Your free-trial has ended, don’t miss out, subscribe today!”
will do a better job converting users.
6. A/B testing process
To test this with an A/B test, we randomly selecting a subset of users and show them the new phrase, and monitor the conversion behavior of this group compared to the other users. We can then observe whether one group converts at a higher rate than the other.
7. The importance of randomness
The key to the above is randomly assigning the users. By randomly assigning the users, we ensure that no other confounding variable is likely to explain the change in performance we observe. If instead we use some specific assignment criteria, then some aspect of that may explain the difference in our result and we cannot say with much certainty that it was our treatment.
8. A/B testing flexibility
A/B testing is a powerful and flexible tool. You could apply it to improve sales or interactions within an app or website, or in a variety of other contexts form medicine (called randomized controlled trials in this case) to optimizing efficiency on an assembly line.
9. Good problems for A/B testing
With that said it is important to be aware of the cases that are and are not conducive to improvement through A/B testing. While no general rule exists, there are some things worth considering.
Good problems for A/B testing are those where users are being impacted in an individual way. Questions like which color will make the user more likely to press a button or which price is better are ripe for A/B testing.
10. Bad problems for A/B testing
Bad problems for A/B testing are those in which there are network effects of users, that is the change you want to test impacts or is directly related to an area where users are interacting with one another. An example of this is the matching algorithm for a dating app. It is hard to untangle the impact of the change if users are matching with people outside of the test group.
There are many nuances to this, but it is worth thinking through how directly you can connect the change made in the A/B test to the change in behaviors. If you can't do this well, perhaps consider other methods.
11. Let's practice!
This should start you thinking variety of cases where A/B testing can be applied. We will begin diving into the details of this and the techniques used to execute one in the next lesson. Good luck on the exercises!