Delta or not?
When you analyze ration metrics, sometimes the analyzed unit in the metric of interest is more granular than the randomization unit. For instance, we can randomize on user_id and try to examine the impact of our changes on per session or page view metrics. This violates some of the assumptions around the calculation of the variance which we use in significance tests. The Delta method can help account for this discrepancy.
This exercise is part of the course
A/B Testing in Python
Hands-on interactive exercise
Turn theory into action with one of our interactive exercises
