1. Interpreting your test results
Once you have successfully completed your analysis it is important to communicate the results effectively. This is often your responsibility as an analyst.
2. Factors to communicate
The first step is deciding on what data to report. Typically, you will want to share the following, when presenting results.
1 - The sample size for each group.
2 - How long the test ran.
3 - The mean and variance of your two groups, as well as the estimated lift and confidence intervals. These can be presented nicely in a table as the above example from another A/B test illustrates
4 - Finally, you want to provide some indication of whether or not the result was statistically significant.
3. Visualizing your results
Visualizations of the results can also be very useful.
While our metric does not lend itself to all types of visualizations, there are other metrics that it might be worth visualizing to provide context.
One type of visualization is a histogram of our data. This provides bucketed counts of observations for each value.
In this case we can provide a histogram of the conversion rates for each user.
4. Generating a histogram
We will create a distinct histogram for the test and control groups.
Here is the purchase rate data already rolled up to the group and user level.
5. Generating a histogram
Now we can call the plt-dot-hist() function on each group. The first argument is the set of data to plot, in this case conversion_rate.
Next, we can add labels and add a call to legend function to help us identify the graphs.
Additionally, since our plots may overlap, we can specify the alpha-parameter, which gives the graphs some transparency.
6. Annotating our plot
We can add vertical lines, as above, to show where our mean values are.
7. Plotting a distribution
An alternative to plotting histograms is to plot the distribution we are approximating. This is useful because now we can provide this plot for our overall conversion rate.
Using our estimated means and variances from before we can easily plot this.
To plot the normal distribution, we first use the linespace() function to generate a range of x values to plot over.
8. Plotting a distribution
Next, we can get the probability values associated with those x's using the normpdf function. Then we must plot a line connecting the y values over this range to give us a normal distribution. Now we have our two distributions.
9. Plotting the difference of conversion rates
The final step is to plot our final metric, the difference between these values. This shows our estimated lift. As discussed, this also has a Normal distribution, the parameters of which we calculated in the previous lesson.
10. Plotting the difference of conversion rates
Using these parameters, we can plot our estimated lift by repeating the previous set of steps.
11. Plotting the confidence interval
One final addition we can make to this is to plot our confidence interval.
In the last lesson we found the bounds of this.
To visualize this, we can color in this region. First, we want to generate a set of points in this region, which can be done with the np-dot-arange() method. We simply pass in a starting point and a step size.
Next we call the fill_between() method as a part of our graph generation. We specify our x and y values as before.
As we can see in our final plot this is a great way to express a large amount of information simply and intuitively.
12. Let's practice!
Great Job! Now let’s practice communicating our results.