1. Statistical significance
So far we have seen differences in our predictions, but how do we know if these differences are meaningful? One of the key concepts in error and uncertainty is that of statistical significance. Tests of statistical significance help us to determine whether the differences we see between groups (for example, two sets of predictions) appear different or are actually statistically different.
2. Statistical significance
We consider differences between groups statistically significant when the probability that they would occur due to chance is 5% or less.
Note that this probability is not 0. That is, we allow for a certain amount of error when determining significant differences. So even differences that we consider to be statistically significant have a 5% probability that they are the results of random variation rather than meaningful differences between groups.
One way to compare groups is to use a t-test.
T-tests test for statistically significant differences between groups using the means, standard deviations, and numbers of observations from two different groups. T-tests compute the likelihood that the two groups are statistically similar.
3. T-tests
The T.TEST() function in Spreadsheets requires four arguments.
The first two arguments specify the two groups of data to compare.
The third and fourth arguments are the number of the tails to test,
and the type of test, both which we'll explain shortly.
4. T-tests
A one-tailed t-test indicates that you have a hypothesis that one average will specifically be higher or lower than the other, whereas
a two-tailed test allows for the data in one group to be either higher or lower than one another. In this course, we'll always choose this option because we want to test whether significant differences are either higher or lower than the comparison group.
The type argument indicates whether you are comparing a pre- and post-test, two samples with even variance, which means the data vary about as much in one sample as the other, or two samples with uneven variance, in which case the data vary much more in one sample than the other. This might happen, for example, if you compared two groups of very different size. Again, we'll always choose the second option for this argument because we are comparing two distinct samples that have roughly even variance.
5. Confidence levels
We determine statistical significance using
confidence levels.
Confidence levels represent the probability that a difference between groups might be due to chance. As we mentioned earlier, we typically consider differences statistically significant at a confidence level of of 0.05, or 5 percent.
In other words, when the result of our T.TEST() function is less than 0.05, we consider the differences significant because there is a 95% probability that the differences are not due to chance. To extend this example, T.TEST() results less than 0.001 have a 0.1% probability that they occurred due to chance.
6. Let's practice!
Now that you have some tools for evaluating significant differences, let's t-test them out!