1. Experiments
Now we know the general workflow for hypothesis testing, we'll discuss one arm of hypothesis-testing - experiments.
2. Experiments, treatment, and control
Experiments are a subset of hypothesis testing that involves performing statistical tests on sample data to draw conclusions about a population.
This doesn't just apply to academia and research; experiments occur in industry too, particularly to draw product insights and drive improvements to commercial performance.
Experiments generally aim to answer a question in the form, "What is the effect of the treatment on the response?", where the treatment refers to the independent variable, and the response to the dependent variable.
3. Advertising as a treatment
As an example of an experiment, we may wish to know what effect an advertisement has on the number of products purchased.
In this case, the treatment is an advertisement, and the response is the number of products purchased.
Visualizing the results using a bar plot suggests the treatment may have been effective in increasing the number of products purchased.
4. Controlled experiments
A common type of experiment is a controlled experiment, where participants are randomly assigned to either the treatment group or the control group.
In our example, the treatment group will see an advertisement and the control group will not.
Other than this difference, the groups should be comparable so that we can determine whether seeing an advertisement causes people to buy more.
If the groups aren't comparable, we may draw incorrect conclusions based on the results.
If the average age of participants in the treatment group is 25 and the average age of participants in the control group is 50, age could potentially influence the results, where younger people are more likely to purchase more, making the experiment biased towards the treatment.
5. The gold standard of experiments
The gold standard, or ideal experiment, will eliminate as much bias as possible.
The first method to help eliminate bias in controlled experiments is to use randomization.
In this experiment protocol, participants are assigned to the treatment or control group randomly, not due to some characteristics. Randomization helps ensure that the groups are comparable. This is called a randomized controlled trial.
The second method is to use blinding. In a blind trial, the participants don't know if they're in the treatment or control group. This ensures that the effect of the treatment is due to the treatment itself, not the idea of getting the treatment.
This can involve using a placebo, which is something that resembles the treatment but has no effect.
This is common in clinical trials that test the effectiveness of a drug, where the control group will be given a sugar pill that has minimal effects on the response.
6. The gold standard of experiments
In a double-blind randomized controlled trial, the person administering the treatment or running the experiment also doesn't know whether they're administering the actual treatment or a placebo.
This protects against bias in the response as well as the analysis of the results.
These different tools all boil down to the same principle: the fewer the opportunities for bias to creep into our experiment, the more reliably we can conclude whether the treatment affects the response.
7. Randomized Controlled Trials vs. A/B testing
Randomized controlled trials can have multiple treatment groups if the aim is to test the difference between multiple treatments, such as different dosage of a medication. They are popular in academia, particularly within scientific and clinical research.
However, randomized controlled trials are also referred to as A/B testing, often when used in industries such as marketing and engineering. The difference is that A/B testing only splits participants into two groups - treatment and control.
8. Let's practice!
Let's experiment with some exercises!