1. Hypothesis tests and z-scores
Hi, I'm Richie. Welcome to the course!
2. A/B testing
Electronic Arts, or EA, make video games. In 2013, they launched a game called SimCity 5. Leading up to its release, they wanted to increase pre-orders.
They used an experimental design technique called A/B testing, which has roots in hypothesis testing, to test different advertising scenarios and see which improved sales the most.
Website visitors were split into a control group and a treatment group. Each group saw a different version of the game's pre-order sales page.
3. Retail webpage A/B test
Here's each version of the SimCity 5 pre-order page. The control group saw the version with a banner advertising money off their next purchase. The treatment group saw the version without the banner.
EA compared the percentage of checkouts for the two groups to see which performed best.
My naive guess would be that the advertisement increases pre-order sales.
4. A/B test results
The results of the A/B test were surprising. The treatment page without the advertisement resulted in forty three percent higher sales than the control page with the advert.
The experiment proved that the intuition that showing more adverts results in more sales was completely wrong.
You might ask yourself, was the forty three percent difference related to a meaningful difference between the control and treatment groups, or was it just chance?
To get this answer, you'd need the original dataset from EA, which isn't publicly available.
However, the method to answer this question of significance would involve techniques from the Sampling in R course, and from this course.
5. Stack Overflow Developer Survey 2020
Each year, Stack Overflow surveys its users, who are primarily software developers, about themselves, how they use Stack Overflow, their work, and the development tools they use.
In this course we'll look at a subset of the survey responses, from users who identified as Data Scientists.
6. Hypothesizing about the mean
Let's hypothesize that the mean annual compensation of the population of data scientists is one hundred and ten thousand dollars.
The first thing we can do to check this is to examine the sample data in the survey. Annual compensation, converted to dollars, is included in the converted_comp column.
The sample mean is a type of point estimate, which is another name for a summary statistic.
You can calculate it with base-R, or more verbosely with dplyr. Either way, the answer is around $120,000.
That's different from our hypothesis, but is it meaningfully different?
7. Generating a bootstrap distribution
To answer this, we need to generate a bootstrap distribution of sample means. This is done by resampling the dataset, calculating the sample mean for that resample, then repeating those steps.
8. Visualizing the bootstrap distribution
Here's a histogram of the bootstrap distribution. Its bell shape means it's roughly normally distributed. Notice that one hundred and ten thousand is on the left of the distribution.
9. Standard error
Recall that the standard deviation of the sample statistics in the bootstrap distribution estimates the standard error of the statistic.
10. z-scores
Since variables have arbitrary units and ranges, before we test our hypothesis, we need to standardize the values. A common way of standardizing values is to subtract the mean, and divide by the standard deviation.
For hypothesis testing, we use a variation where we take the sample statistic, subtract the hypothesized parameter value, and divide by the standard deviation of the statistic, which is the standard error. The result is called a z-score.
The sample mean annual compensation for data scientists of around $120,000, minus the hypothesized compensation for data scientists of $110,000, divided by the bootstrap standard error of $5500 gives a z-score of 1.7.
11. Testing the hypothesis
Is that a big or small number? Determining that is the goal of this course.
In particular, we can now state one of the uses of hypothesis testing. Determining whether a sample statistic is close to or far away from an expected value.
12. Standard normal (z) distribution
Here's a density plot of the probability density function for the standard normal distribution. That is, a normal distribution with mean zero and standard deviation one. It's often called the z-distribution. As you might guess, z-scores are related to the z-distribution. You'll encounter this distribution throughout the course.
13. Let's practice!
Time to begin.