Academy / A/B Testing Fundamentals

Read results without fooling yourself

The analysis and reporting errors that make a result look real, larger or more certain than it is, and the checks that catch each one.

• Report a result with its confidence interval and the false positive risk for your programme
• Recognise peeking, underpowered wins, segment fishing, the wrong unit of analysis and Simpson's paradox in a report
• Apply Twyman's law and a results review checklist before a win is shared