Statistics

A/B Test Calculator

Determine statistical significance of an A/B test experiment

Last reviewed: How we build & verify
Control Group
Variant Group
Control Conversion Rate
N/A
Variant Conversion Rate
N/A
Relative Uplift
N/A
Z-Score
N/A
P-Value
N/A
Confidence
N/A
Statistically Significant?
N/A

How it works

Tests whether the difference between two conversion rates is larger than random variation would plausibly produce. It runs a two-proportion z-test and returns a p-value, which is what separates a real improvement from noise.

Formula
p̂ = (x₁ + x₂) / (n₁ + n₂)
SE = √( p̂(1 − p̂) × (1/n₁ + 1/n₂) )
z = (p₂ − p₁) / SE
Variables
  • x₁, x₂conversions in control and variant
  • n₁, n₂visitors in control and variant
  • p̂pooled conversion rate under the null hypothesis
  • p-valueprobability of seeing a gap this large if the variants were identical
Worked example
Inputs: Control 60 conversions from 1,200; variant 84 conversions from 1,200
  1. p₁ = 60/1200 = 5.0%, p₂ = 84/1200 = 7.0%
  2. Pooled p̂ = 144 / 2,400 = 0.06
  3. SE = √(0.06 × 0.94 × (1/1200 + 1/1200)) = 0.009695
  4. z = (0.07 − 0.05) / 0.009695 = 2.06
  5. Two-tailed p-value ≈ 0.039
Result: z = 2.06, p = 0.039: significant at the 5% level, a 40% relative lift
Notes
  • Fix the sample size before you start. Checking daily and stopping the moment p drops below 0.05 (peeking) inflates the false-positive rate well past 5%.
  • A p-value of 0.039 does not mean a 96% chance the variant is better. It means data this extreme would appear 3.9% of the time if the two were truly identical.
  • Run for whole weeks. Traffic and intent differ sharply between weekdays and weekends, and a partial week bakes that pattern into the result.
  • Statistical significance is not business significance. A reliably detected 0.1% lift may not pay for the engineering work behind it.

Related Tools

Anova CalculatorStatistics CalculatorConfidence IntervalCorrelation CalculatorMargin Of ErrorPercentile Calculator

Most Searched Today

BMI CalculatorPercentage CalculatorCompound InterestMortgage CalculatorLoan EMI CalculatorCurrency ConverterSIP CalculatorAge CalculatorTip CalculatorLength ConverterCalorie CalculatorScientific Calculator

Frequently Asked Questions

Ab Test Calculator runs right in the browser tab you already have open. No install, no account, no cost.

Ab Test Calculator follows the standard method used in textbooks and technical references for this calculation, and we show the formula or approach on the page itself so you can check the working rather than trust a black box. Found an edge case that breaks it? Use the feedback button in the bottom-right corner.

A/B Test Calculator | ExaCalc