1. Why run experiments?
There are many reasons for running experiments.
2. The value of A/B testing
AB tests allow us to: reduce uncertainty around the impact of new designs and features, formalize decision-making into a scientific, data-driven process rather than relying on intuition, offer a generous value for the investment as many of the successful AB tests involve incredibly simple changes that result in major financial wins. They also allow us to continue to optimize metrics even at the mature stages of the business. And finally, they enable us to make conclusions from carefully designed experiments based on causality rather than correlation.
3. Hierarchy of evidence
The hierarchy of evidence was introduced as a way to grade medical recommendations, an adaptation of which is shown. Randomized controlled experiments, which include AB tests, are considered the gold standard for establishing cause and effect. Observational methods, which are outside the scope of this course, have lower accuracy due to lower experimental control. And at the bottom of the pyramid are mere opinions or correlations without clear causal evidence. Correlations without establishing causality can lead to dangerous outcomes.
4. Do error messages reduce churn?
A good example demonstrating this comes from Microsoft Office when they noticed that users who see more error messages and crashes have lower churn or service cancellation rates. Does this mean that Office should show more error messages or cause more crashes? Certainly not. The explanation for this misleading correlation comes from examining a third variable that impacts both churn and error messages: usage. High-usage customers naturally see more errors, are more likely to experience more crashes, and have lower churn rates.
There might be a strong correlation between two variables, but it doesn't allow us to conclude that one causes the other. These strong non-causal correlations are called spurious correlations.
5. Pearson's correlation coefficient
That said, correlations still offer great value for understanding relationships between variables, such as in exploratory data analysis, and machine learning techniques. This can be useful in situations where AB tests are not feasible due to lack of resources or limited data.
Visual inspection is a great way to explore relationships in the data, but equally important is having a summary statistic. Recall that the Pearson's correlation coefficient (r) is a score between negative one and positive one that measures the strength of a linear relationship between two variables. It is calculated by dividing the covariance of X and Y by the product of the standard deviation of each as shown in the formula. It assumes both normality and linearity of the data.
6. Correlations visual inspection
A great way to visually inspect the relationship between the variables is using `seaborn` library's pairplot. Running this method on some of the admissions dataset's columns shows no relationship between the serial number and the chances of admission, but shows a strong positive linear relationship between the GRE score and the likelihood of admission.
7. Pearson correlation heatmap
To quantify the strength of the relationship, we can use the pandas dot corr method on the two columns, which gives us a result of zero-point-eight confirming the strength and direction of the relationship. Visualizing the values in a heatmap is an excellent way to get a view of how correlated they are. Besides being able to carefully distinguish between correlations versus causal relationships, correlation analyses enable us to identify variables that can serve as a proxy for other metrics that are not easy to measure. We will examine those in more detail in the next video.
8. Let's practice!
Let's see if we can practice what we've learned in the coming exercises.