1. Power analysis: sample and effect size
We now dive into the intricacies of power analysis, focusing on understanding effect size and how it influences sample size.
2. A primer on effect size
Effect size quantifies the magnitude of the difference between groups, beyond just noting if the difference is statistically significant.
Cohen's d is a commonly used measure, calculated as the difference in means divided by a pooled standard deviation.
3. The dataset: video game engagement
A video game company conducted an experiment with sixty participants to understand player engagement across two game genres: Action and Puzzle.
They recorded the average number of hours players spent engaged to assess which type tends to captivate players more effectively.
4. Calculating power overview
Power analysis revolves around the probability that our test will correctly reject a false null hypothesis. This corresponds to identifying a true effect, avoiding a Type II error. A type II error is denoted as beta, so power is one minus beta and it ranges from zero to one, where one is certainty in our ability to detect a true effect.
To calculate power, we first assume an effect size. Here we choose a value of 1, derived from historical data comparing the engagement scores of video game genres. We can also use our sample data to make an estimate of the effect size, but traditionally power analysis is done prior to the data collection. This can also help us determine how big of a sample size we should use in our study.
We initialize the power object and call the .solve_power() method, using a sample size for how many video game players were assessed in either group (30), our assumed effect size, and our alpha of 0.05.
This high power tells us the likelihood that our test will detect a significant result, given our effect size and sample size.
5. Cohen's d formulation
To calculate Cohen's d as an effect size, we define a function. Its two inputs are numeric data corresponding to the two groups from our sample data.
We calculate the difference in the means of the two groups, their sample sizes, and their variances.
Next, we determine a pooled standard deviation using this information.
Lastly, Cohen's d is the difference in means divided by the pooled standard deviation.
6. Cohen's d for video game data
To apply this to the video game data, we first split the data into two groups based on the genre.
Then we apply our function to get the effect size.
The result here is near the theoretical result of 1 we assumed earlier.
7. Understanding sample size and power
Balancing the need for sufficient power with practical constraints on sample size is a fundamental aspect of planning a study, such as comparing engagement times across different video game genres.
A larger sample size can enhance an experiment's power, improving the likelihood of detecting a true effect.
8. Sample size calculation in context
Let's contextualize this within our video game study.
Assuming our calculated value for Cohen's d engagement time between game genres, we calculate the sample size needed for each group to achieve 99% power with an alpha of 0.05 and equally-sized groups with a ratio of 1. This calculation is pivotal in ensuring our study is adequately powered to detect meaningful differences in player engagement across genres.
Assuming we have an effect size of around 1.2, we would need at least 28 participants in each group to achieve a power of 99%. Recall we collected 30 participants, so we can feel confident about our experiment's power.
9. Visualizing sample size requirements
We next build a visualization illustrating the relationship between effect size measured as Cohen's d and required sample size for our video game study, by plotting varying effect sizes against required sample sizes.
10. Visualizing sample size requirements
As effect size increases, the required sample size for each group decreases, highlighting the importance of understanding the expected magnitude of differences when planning a study.
11. Let's practice!
Test your newfound knowledge with some exercises.