Academy / A/B Testing Fundamentals

The statistics you need to read a result

Significance, confidence intervals, power and the four families of statistical models your testing tool may use, and how to match the model to the way your team works.

• Interpret a p-value and a confidence interval correctly, and explain what neither tells you
• Match fixed-horizon, sequential, Bayesian and bandit models to common testing situations
• Estimate how multiple metrics and repeated looks inflate the chance of a false win