A randomised experiment in which visitors are split at random between the current version (control) and a changed version (variation), so that a difference in results can be attributed to the change.
A/A test
A test in which both groups see the same version. It checks that the tool, tracking and randomisation work; about 5% of A/A comparisons will look significant by chance at a 5% level.
A/B/n test
A test with more than one variation against a single control. Each extra variation needs its own share of traffic, so the test takes longer.
Multivariate test
A test that changes several page elements at once and shows every combination of their versions. The number of cells multiplies quickly, so it needs a lot of traffic.
Hypothesis
A testable statement that names the evidence, the change, the audience, the expected outcome and the metric that will measure it.
Evidence score
A 0–5 rating of how strongly research supports an insight, rising as independent sources such as analytics, replays and Voice of Customer agree.
Primary metric
The single metric, agreed before launch, that decides whether the change worked. The test is sized for it.
Secondary metric
A metric that helps explain the result, such as add-to-basket rate, but does not decide it on its own.
Guardrail metric
A metric that must not get worse beyond an agreed harm threshold, such as revenue per visitor, page speed or returns rate. A breached guardrail can block a winning change.
Minimum detectable effect (MDE)
The smallest true effect the test is designed to detect reliably. It is usually stated as a relative lift and must be converted to an absolute difference for sizing.
Significance level (alpha)
The false positive rate you accept when there is no real effect, usually 5%. It is the threshold the p-value is compared with.
Statistical power
The probability that the test detects an effect of at least the MDE if it really exists, usually set at 80%.
p-value
The probability of seeing a difference at least as large as the one observed if the change had no effect. It is not the probability that the variation is better.
Confidence interval
The range of effects consistent with the data at a given level, usually 95%. Report it alongside the lift, and plan with its lower end.
Fixed-horizon test
A frequentist test whose result is read once, at a sample size and end date set in advance.
Peeking
Checking results repeatedly and stopping as soon as they look significant. With a fixed-horizon model this greatly inflates the chance of a false win.
Sequential testing
A statistical method designed for repeated looks, which adjusts the thresholds so that a test can be stopped early without inflating the false positive rate.
Bayesian testing
An approach that reports results as probabilities, such as a 'chance to beat control'. It is useful, but not immune to peeking or to an unagreed stopping rule.
Multi-armed bandit
A design that shifts traffic towards the best-performing variation as data arrives. It suits earning during a short campaign more than learning a reliable effect.
CUPED
A variance reduction technique that uses visitors' behaviour before the test to remove predictable noise, so the test needs fewer visitors. It helps most with returning or logged-in customers.
Sample ratio mismatch (SRM)
A split of visitors that differs from the configured split by more than chance allows, checked with a chi-square test (commonly at p < 0.001). It signals a technical problem, so the result cannot be trusted.
Multiple comparisons
Testing many metrics, segments or variations at once. Each extra comparison raises the chance that at least one looks significant by chance.
JavaScript injection
A delivery method in which the testing tool's script changes the page in the visitor's browser after it loads. It is fast to launch without a release but can cause flicker and slow the page.
Flicker
A brief display of the original page before the variation appears, caused by client-side changes applied too late. It can bias results and harm the experience.
Client-side SDK
A delivery method in which the site's own front-end code asks the testing tool which variant to show and renders it itself. It suits single-page applications and components.
Server-side testing
A delivery method in which the variant is decided and rendered on the server before the page reaches the browser. It is needed for business logic such as pricing, delivery thresholds or search ranking.
Edge testing
A delivery method in which the variant is decided at the content delivery network, close to the visitor, so cached pages can be tested without flicker.
Stable selector
A reliable way for a test to find a page element, ideally a dedicated data attribute such as data-test, which survives site updates better than generated class names or position.
Layout shift
Movement of page content after it has drawn, measured as Cumulative Layout Shift. Variations should reserve space for inserted elements to avoid it.
False positive risk
The share of statistically significant wins that are not real. It depends on the significance level, power and the programme's historical success rate, and is often far higher than 5%.
Winner's curse
The tendency of significant results from low-powered tests to overstate the true effect, because only unusually large estimates cross the threshold.
Twyman's law
The rule that any figure that looks interesting or unusual is probably wrong. Check a surprising lift against past tests and a second data source before sharing it.
Simpson's paradox
A reversal in which a variation wins within every period or segment but loses when they are pooled, typically because the traffic split changed during the test.
Pre-registration
Writing down the hypothesis, primary metric, guardrails, segments, sample size and decision rule before launch, so the analysis cannot be bent to fit the result.
Test repository
A searchable record of every test, with its hypothesis, result, decision and consistent tags for page, lever and principle, so that learnings are reused rather than rediscovered.