1. Hypothesis testing
Now that we've gone over confidence intervals, let's speed things up a bit with hypothesis testing. We'll go over the logistics of running a test for both means and proportions.
2. Quick review
Before we dive in deeper, let's quickly review what exactly a hypothesis test is and why it's important in interviews.
Hypothesis testing is really just a means of coming to some statistical inference. This is to say that we want to look at the distribution of our data and come to some conclusion about something that we think may or may not be true.
3. Assumptions
Before we run a hypothesis test, there are a couple of assumptions that we need to check; these are fair game for interviewers.
Our assumptions include that the sample must be taken randomly, each observation must be independent, and the sample data must be normally distributed around the sample mean, which will naturally occur in sufficiently large samples due to the central limit theorem. Lastly, the variance between the sample and population must be constant.
4. Generating hypotheses
After checking the assumptions, we need to generate both our null and alternate hypotheses before we can run our test. The null hypothesis represents the treatment not effecting the outcome in any way. The alternate hypothesis, on the other hand, represents the outcome that the treatment does have a conclusive effect.
As we can see, the null hypothesis, denoted by H sub 0, and the alternate, by H sub 1, change depending on the type of test. However, the consistent theme is that we are taking the sample estimate and comparing it to the expected value from our control.
5. Which test to use
You saw a little about one-tailed and two-tailed tests before, let's focus on the two most common hypothesis tests: z-tests and t-tests. The test that you use depends on the situation. If you know the population standard deviation and you have a sufficient sample size, you'll probably want a z-test, otherwise break out the t-test. In python, we'll use the proportions-underscore-ztest and ttest-underscore-ind functions to run these.
6. Evaluating results
When you run your test, your result will be generated in the form of a test statistic, either a z score or t statistic. Using this, you can compute the p-value, which represents the probability of obtaining the sample results you got, given that the null hypothesis is true.
It's intuitive that if your p-value is small enough, falling in the yellow here, then you can reject the null. We use the significance level to determine how large of an effect you need to reject the null hypothesis, or how certain you need to be. A common alpha value is 0 point 05, which represents 95 percent confidence in your test.
7. Types of errors
When you get your outcome, there will always be a probability of obtaining false results; this is what your significance level and power are for. There are two types of errors that you can get. Let's look at a confusion matrix for more on this, with our predictions on the y-axis.
Type I errors or false positives, shown in the top right, occur when you incorrectly reject a true null hypothesis. Type II errors or false negatives, shown in the bottom left, occur when you accept a null hypothesis when an effect really exists. This means that we predicted no effect when there really was an effect.
8. Summary
To summarize, we reviewed hypothesis tests, assumptions, the process of a test, and the types of errors that can occur.
9. Let's prepare for the interview!
Let's prepare with some exercises!