Effecten van randomisatie

Willekeurige toewijzing speelt een cruciale rol in A/B-testen. Stel je voor dat je een Data Scientist bent die een experiment opzet om de effecten van verschillende checkout-paginalay-outs op bedrijfsstatistieken te testen.

Je gaat verkennen hoe je een percentage gebruikers kunt samplen om willekeurige instroom van verkeer in het experiment te simuleren en je controleert de verdelingen van bepaalde kenmerken over willekeurig toegewezen gebruikers in elke groep. Zo kun je nagaan of je kunt generaliseren naar de totale verkeerspopulatie en isoleer je de impact van de ene variabele die we tussen groepen veranderen: het ontwerp van de checkoutpagina.

De checkout-DataFrame is voor je geladen. Ga ervan uit dat elke rij in de DataFrame overeenkomt met een unieke gebruiker die de betreffende checkout_page bezoekt, inclusief hun acties en aanvullende kenmerken.

Deze oefening maakt deel uit van de cursus

A/B-testen in Python

Cursus bekijken

Praktische interactieve oefening

Probeer deze oefening eens door deze voorbeeldcode in te vullen.

# Determine the normalized distribution of browser counts
checkout['browser'].____(normalize = ____)

Code bewerken en uitvoeren

Deze oefening maakt deel uit van de cursus

A/B-testen in Python

SkillTag.level.intermediateSkillTag.label

4.8+

Begin de cursus gratis

In this chapter, you’ll learn the foundations of A/B testing. You’ll explore clear steps and use cases, learn the reasons and value of designing and running A/B tests, and discover the most commonly used metrics design and estimation frameworks.

Exercise 1: Wat is A/B-testen?Exercise 2: Wanneer een A/B-test niet de beste keuze is Exercise 3: Stappen van A/B-testen Exercise 4: Effecten van randomisatie

Huidige oefening

Exercise 5: Waarom experimenten uitvoeren?Exercise 6: Correlatievisualisatie Exercise 7: Correlatie of causaliteit?Exercise 8: Ontwerp van metrics en schatting Exercise 9: Gemiddelden en proporties Exercise 10: Advertentie-impressie-metrieken

In Chapter 2, you’ll cover the experiment design process. Starting with learning how to formulate strong A/B testing hypotheses, you’ll also cover statistical concepts such as power, error rates, and minimum detectable effects. You’ll finish the chapter by learning to estimate the appropriate sample size needed to yield conclusive results and tackle scenarios with multiple comparisons.

Exercise 1: Hypothesis formulation and distributions Exercise 2: Strong hypothesis formulation Exercise 3: Plotting distributions Exercise 4: Central limit theorem for means Exercise 5: Experimental design: setting up testing parameters Exercise 6: Interpreting p-values Exercise 7: Error rates in the wild Exercise 8: Experimental design: power analysis Exercise 9: Plotting power curves Exercise 10: Sample size for means Exercise 11: Sample size for proportions Exercise 12: Multiple comparisons tests Exercise 13: Is a multiple comparisons correction needed?Exercise 14: Corrected p-values

Here, you’ll discover a concrete workflow for cleaning, preprocessing, and exploring AB testing data, as well as learn the necessary sanity checks we need to follow to ensure valid results. You’ll explore a detailed explanation and example of analyzing difference in proportions A/B tests.

Exercise 1: Data cleaning and exploratory analysis Exercise 2: Proportions EDA Exercise 3: A/B test data cleaning Exercise 4: Sanity checks: Internal validity Exercise 5: SRM Exercise 6: Distributions balance Exercise 7: Sanity checks: external validity Exercise 8: Novelty effects detection Exercise 9: Simpson's paradox in action Exercise 10: Analyzing difference in proportions A/B tests Exercise 11: Difference in proportions A/B test Exercise 12: Interpretation of confidence intervals Exercise 13: Confidence intervals for proportions

In the final chapter, you’ll develop frameworks for analyzing differences in means and leveraging non-parametric tests when several assumptions aren't met. You’ll also learn how to apply the Delta method when analyzing ratio metrics and discover the best practices and some advanced topics to continue the A/B testing mastery journey.

Exercise 1: Analyzing difference in means A/B tests Exercise 2: T-test for difference in means Exercise 3: Pairwise t-tests Exercise 4: Non-parametric statistical tests Exercise 5: Parametric or non-parametric?Exercise 6: Mann-Whitney U test Exercise 7: Chi-square test for independence Exercise 8: Ratio metrics and the delta method Exercise 9: Delta or not?Exercise 10: Delta method Exercise 11: A/B Testing best practices and advanced topics intro Exercise 12: Best practices Exercise 13: Day-of-the-week effect Exercise 14: Wrap-up: A/B testing in python