Day-of-the-week effect
Although A/B testing enables us to control for external variables such weather and holidays, some treatments may have stronger or weaker effects than others that need to be included to fully capture the user behavior around fluctuating periods. Depending on the tracked metric, you may need to consider running your A/B tests for full weekly increments. Some metrics exhibit seasonality and are affected by holidays, weekends, etc. while others don't fluctuate as much. It is up to you as the data scientist to advise the team on whether they can stop the test upon reaching the required sample size, or extend by a few days to cover seasonal effects such as day-of-the-week.
Examine the novelty dataset that is loaded for you to select the metrics that need to run for weekly increments. pandas and matplotlib are also loaded for you.
This exercise is part of the course
A/B Testing in Python
Hands-on interactive exercise
Have a go at this exercise by completing this sample code.
# Create a line plot
novelty.____('____', '____')
plt.title('Purchase Rate Over Test Duration')
plt.ylabel('Purchase Rate [%]')
plt.ylim([0, 0.04])
plt.show()