1. Correlation caveats
While correlation is a useful way to quantify relationships, there are some caveats.
2. Non-linear relationships
Consider this data. There is clearly a relationship between x and y, but when we calculate the correlation, we get 0-point-18.
3. Non-linear relationships
This is because the relationship between the two variables is a quadratic relationship, not a linear relationship.
The correlation coefficient measures the strength of linear relationships, and linear relationships only.
4. Correlation only accounts for linear relationships
Just like any summary statistic, correlation shouldn't be used blindly, and you should always visualize your data when possible.
5. Mammal sleep data
Let's return to the mammal sleep data.
6. Body weight vs. awake time
Here's a scatterplot of each mammal's body weight versus the time they spend awake each day. The relationship between these variables is definitely not a linear one.
The correlation between body weight and awake time is only about 0-point-3, which is a weak linear relationship.
7. Distribution of body weight
If we take a closer look at the distribution for bodywt, it's highly skewed. There are lots of lower weights and a few weights that are much higher than the rest.
8. Log transformation
When data is highly skewed like this, we can apply a log transformation. We'll create a new column called log_bodywt which holds the log of each body weight. We can do this using np-dot-log.
If we plot the log of bodyweight versus awake time, the relationship looks much more linear than the one between regular bodyweight and awake time.
The correlation between the log of bodyweight and awake time is about 0-point-57, which is much higher than the 0-point-3 we had before.
9. Other transformations
In addition to the log transformation, there are lots of other transformations that can be used to make a relationship more linear, like taking the square root or reciprocal of a variable. The choice of transformation will depend on the data and how skewed it is. These can be applied in different combinations to x and y, for example, you could apply a log transformation to both x and y, or a square root transformation to x and a reciprocal transformation to y.
10. Why use a transformation?
So why use a transformation?
Certain statistical methods rely on variables having a linear relationship, like calculating a correlation coefficient.
Linear regression is another statistical technique that requires variables to be related in a linear manner, which you can learn all about in this course.
11. Correlation does not imply causation
Let's talk about one more important caveat of correlation that you may have heard about before: correlation does not imply causation.
This means that if x and y are correlated, x doesn't necessarily cause y.
For example, here's a scatterplot of the per capita margarine consumption in the US each year and the divorce rate in the state of Maine. The correlation between these two variables is 0-point-99, which is nearly perfect. However, this doesn't mean that consuming more margarine will cause more divorces. This kind of correlation is often called a spurious correlation.
12. Confounding
A phenomenon called confounding can lead to spurious correlations.
Let's say we want to know if drinking coffee causes lung cancer. Looking at the data, we find that coffee drinking and lung cancer are correlated, which may lead us to think that drinking more coffee will give you lung cancer.
13. Confounding
However, there is a third, hidden variable at play, which is smoking.
14. Confounding
Smoking is known to be associated with coffee consumption.
15. Confounding
It is also known that smoking causes lung cancer.
16. Confounding
In reality, it turns out that coffee does not cause lung cancer and is only associated with it, but it appeared causal due to the third variable, smoking.
This third variable is called a confounder, or lurking variable. This means that the relationship of interest between coffee and lung cancer is a spurious correlation.
Another example of this is the relationship between holidays and retail sales. While it might be that people buy more around holidays as a way of celebrating, it's hard to tell how much of the increased sales is due to holidays, and how much is due to the special deals and promotions that often run around holidays. Here, special deals confound the relationship between holidays and sales.
17. Let's practice!
Now that you've learned how to use correlation responsibly, time to practice.