1. Independent-sample t-test
A common analysis in AB design is independent-sample t-tests.
2. Independent t-test in A/B design
Recall this test assesses whether the difference of two group means, each containing different subjects, is significant.
For example, in our AB design, the null hypothesis is that there is no difference in time spent eating Cheese and Pepperoni pizza.
3. Assumptions
There are some assumptions to consider with independent-sample t-tests.
The dependent variable, time, must be an interval or ratio, having equal intervals, samples must be random, data derives from a normally distributed population, and has similar group variances.
We can ensure a parametric t-test is appropriate with a histogram. Provided it is relatively symmetrical, the data is assumed to derive from a normally distributed population, which is the case here.
4. Sample size
Recall pwr-dot-t-dot-test determines the required t-test sample size.
Let's run the test with the expected effect size as d, point-seven-three, expected power, point-eight, alpha, point-zero-five, and specify two samples and two tails.
Each group, Cheese and Pepperoni, is a sample. Recall two-tailed indicates assumptions are not made about which group is larger.
31 subjects must contribute time to eat Cheese pizza and 31 subjects to Pepperoni pizza.
5. Assessing variances
T-tests assume equal group variances.
If untrue, the Type I error rate can increase.
We can test the variances with Levene’s test.
In leveneTest of the car package, input the data by groups formula as Time tilde Topping with the data frame data.
If the test is not significant, as indicated with the point-seven p-value above our point-zero-five alpha, the Cheese and Pepperoni variances are treated as equal.
If significant, the group variances are unequal.
6. Test
Recall t-dot-test, with the data by group formula, runs the t-test.
We can set paired to FALSE to indicate independent-samples. Indicate two-tails and no direction assumption with two-dot-sided, and whether group variances are equal with var-dot-equal. Set var-dot-equal to TRUE to use pooled-variance if the leveneTest was not significant and if significant use FALSE to use a variance approximation. In our case, we use TRUE.
Our p-value is below point-zero-five, therefore the null hypothesis is rejected indicating that the groups are significantly different.
The output also gives the 95% confidence interval. Running the two-sample independent t-test on the mean time to eat Cheese versus Pepperoni pizza indicates the Cheese and Pepperoni pizzas were eaten at different rates.
7. Cohen's d
For t-tests, Cohen’s d is the effect size used. It is a standardized measure of the difference between two means.
To find Cohen’s d, use cohens-underscore-d from the effectsize package, giving the formula of data by grouping.
A Cohen’s d of point-two or less is small, point-five is medium, and point-eight or more is large. Here, we have a large effect size indicating a large effect and difference in the time to eat Cheese and Pepperoni pizza.
8. Power
The power analysis is also dependent on the test run.
Recall that pwr-dot-t-dot-test also computes the t-test power analysis.
To derive the power of the test rather than the sample size required, give the number of subjects in each group with n, leaving out the power. Remember to give the calculated effect size as d, and p-value as sig-dot-level, rather than the expected values or alpha.
For results to be accepted, a statistical power of 80%, or point-eight, is needed, indicating a 20% probability of an error, calculated by subtracting the power percentage from 100.
The output re-specifies the values we put in, as well as gives the power of our time to eat Cheese versus Pepperoni topping t-test.
9. Let's practice!
Let's practice independent-samples t-tests!