1. A/B testing
A/B tests are an incredibly common type of experiment that a data scientist or data analyst might design. Let's talk about what an A/B test is.
2. A/B testing
A/B tests are often discussed in a marketing context and are very common in marketing-related companies, although they're definitely used in other industries as well. One common example of their use is to test customer engagement with different features of a company's website.
3. Power and sample size in A/B tests
One important concept that will come back here is power and sample size calculations, as they're pretty crucial in A/B testing. Usually, you'll be calculating sample size, given some power and significance level. Then, you let your A/B test run until you get the sample size you need. You also need to decide an effect size, just as before! You may have guessed by now that an A/B test is simply an application of your basic experimental design knowledge, though they can and do get complicated, so stick with me so you can get good at them.
An A/B test changes one thing and one thing only and measures the differences in outcome between these two alternatives.
4. Lending Club A/B test
In this section, we'll design and conduct an A/B test with the Lending Club data we've been using. Say Lending Club was interested to see how the color of the website header affected the loan amount, which is how much an applicant asks to borrow. They have a general hypothesis that softer, gentle colors may influence applicants to ask for lower amount of money, perhaps a more reasonable amount that they can more feasibly pay back. They already use a light blue website header, but they've decided to test a second softer, gentler color on the website.
Applicants designated Group A were funneled to an application with the light blue existing header while applicants designated group B were funneled to an application with a new mint green header. We'll examine the amount of money the applicant asked for when applying for the loan based on which colored header they were shown -- not a traditional A/B test, which often focuses on a metric such as click-through rate, which in this case would be number of people who applied, but still an interesting test.
5. Let's practice!
Let's take everything we've learned in the last two chapters and analyze the Lending Club A/B test.