A/B Test Significance Calculator
Version B got more conversions than version A. Before you roll it out, check whether that gap is a real effect or just random noise in your sample. Enter your numbers below and get a straight answer, backed by the math.
Result
What this means
To improve this test
What “statistical significance” actually means
A coin can land heads six times in a row and still be a fair coin. Small samples produce odd streaks by chance alone. An A/B test asks one question: is the gap between A and B bigger than the kind of streak random luck can produce on its own?
The p-value is your answer, written as a probability. A p-value of 0.03 means there is only a 3% chance you would see a gap this large if A and B were truly identical. Most teams treat anything under 5% (a p-value below 0.05) as a real, repeatable difference rather than noise.
The formula, step by step
The standard error measures how much conversion rates naturally wobble from one batch of visitors to the next, purely due to sample size. The z-score compares your actual gap against that wobble. A bigger z-score means the gap is unusual, and unusual gaps are the ones worth trusting.
A worked example
An online store tests two checkout button colors on 5,000 visitors per group.
| Group | Visitors | Conversions | Rate |
|---|---|---|---|
| A, teal button | 5,000 | 600 | 12.00% |
| B, gold button | 5,000 | 660 | 13.20% |
Pooled rate p = 1,260 ÷ 10,000 = 0.126. Standard error works out to roughly 0.0066. That gives a z-score near 1.81 and a p-value near 0.071.
At 95% confidence, 0.071 sits above the 0.05 line. The gold button looks promising, but the store should wait for additional visitors before switching for good.
Common questions about A/B testing
It depends on your current conversion rate and the size of the change you expect. Small baseline rates and small expected gaps both call for larger samples, often several thousand visitors per group. Use a sample size calculator before you start, not after.
It is the probability you would see a gap this size, or larger, purely by chance, if A and B had no real difference. A low p-value means chance is an unlikely explanation, so the gap is probably genuine.
Pick two tailed when you are open to either version winning. Pick one tailed only when you decided, before the test began, that you only care whether B beats A specifically, not the reverse.
A flat result usually means one of two things: the true difference is small, or your sample is not yet large enough to detect it. Let the test run longer, or plan a bigger test around a bolder change.
Checking results daily and stopping the moment they look good raises your odds of a false positive by a wide margin. Set a sample size or a test duration in advance, and hold to it.
Keep testing with confidence
Plan your next test properly, or double check a result from a different angle.