A/B Test Significance Calculator
Test whether a difference in conversion rate is distinguishable from chance, with the assumptions checked rather than assumed.
About this tool
Runs a two-proportion z-test on a split test: visitors and conversions for each variant in, p-value and confidence interval out.
Two things it does that most significance calculators do not.
**It checks whether the test applies.** The z-test rests on a normal approximation that breaks down below roughly five expected conversions per arm. Below that the p-value is not merely uncertain, it is meaningless — and "97% confident" computed from four conversions is the most common way a split test misleads the person who ran it. Here that case is refused, not rounded.
**It does not tell you the variant is probably better.** A p-value is the probability of seeing a difference this large if the two variants were identical. That is not the same statement, and reading it as the second one is the standard mistake.
The confidence interval is usually the more useful number: it says how large the difference might plausibly be, which is what a decision actually needs.
Common questions
What does a p-value of 0.03 mean?
That if the two variants converted identically, you would see a difference at least this large about 3% of the time by chance alone. It is not the probability that the variant is better, and it is not the probability that the result is real.
Why does it refuse to give a result on small numbers?
Because the z-test assumes a normal approximation that does not hold when expected conversions per arm drop below about five. The formula still produces a number; that number does not mean what it appears to. Refusing is more honest than rounding.
Can I stop the test as soon as it hits 95%?
No. Checking repeatedly and stopping at the first significant reading inflates the false-positive rate badly — enough that a test with no real difference will cross 95% most of the time if you watch it long enough. Fix the sample size before you start.
What does the confidence interval tell me that the p-value does not?
How big the difference plausibly is. A significant result whose interval runs from +0.1% to +4% is significant and still might not be worth shipping. That is a question the p-value cannot answer.