A/B Test Significance Calculator
Check whether variant B really beats A: conversion rates, relative uplift with its confidence interval, z-test p-value, chance to beat A and sample size needed.
Decide the sample size before the test and stop only when it is reached. Checking the p-value every day and stopping at the first "significant" reading makes false winners far more likely than α suggests.
Statistical significance is not business significance: look at the interval for the uplift and ask whether its lower end would still be worth shipping.
An A/B test splits visitors between the current version (A) and a change (B) and compares conversion rates. This calculator tells you whether the gap is larger than chance would usually produce, using the standard two-proportion z-test, and gives the relative uplift with a confidence interval, a Bayesian "chance to beat A" and the sample size you would need to detect a difference of this size reliably.
How to use this tool
- Enter visitors and conversions for the control (A) and the variant (B).
- Choose the confidence level; 95% is the common default.
- Choose two-sided unless you only care whether B is better.
- Read the result, the p-value and the interval for the uplift before deciding.
Formula
p̂ = (c_A + c_B) ÷ (n_A + n_B) · z = (p_B − p_A) ÷ √(p̂(1 − p̂)(1/n_A + 1/n_B)) · p = 2 × P(Z ≥ |z|)
The interval for the absolute difference uses the unpooled standard error √(p_A(1−p_A)/n_A + p_B(1−p_B)/n_B). The uplift interval is built on the log of the rate ratio. The sample size uses the usual two-proportion formula with 80% power.
Example
With 500 conversions from 10,000 visitors in A (5.0%) and 580 from 10,000 in B (5.8%), the uplift is 16%, z = 2.503 and the two-sided p-value is 0.01232. That is significant at 95% but not at 99%. The 95% interval for the uplift runs from about 3.3% to 30.3%, so B is likely better, but by an uncertain amount.
Frequently asked questions
Why not show "observed power"?
Power computed from the observed difference is just a re-expression of the p-value and says nothing new. The sample-size line answers the useful question instead: how many visitors per variant you would need to detect a difference of this size 80% of the time.
Can I stop the test as soon as it turns significant?
No. Repeatedly checking and stopping at the first significant result inflates false positives. Fix the sample size or the test duration in advance.
What does "chance that B beats A" mean?
It is the Bayesian posterior probability that B's true conversion rate is higher than A's, starting from a flat prior. It answers a different question from the p-value and is easier to explain, but it depends on the prior.
Sources
Last updated:
Page address with these inputs: https://kakao.mobile/tools/a-b-test-significance-calculator/
Cite or embed this tool
kakaoCalc. (2026, November 9). A/B Test Significance Calculator. https://kakao.mobile/tools/a-b-test-significance-calculator/
“A/B Test Significance Calculator.” kakaoCalc, 9 Nov. 2026, kakao.mobile/tools/a-b-test-significance-calculator/.
Paste this code into your page. The widget runs in your visitors' browsers, shows no ads and keeps a link back to this page. Embedding terms