Your test ran for two weeks. The new donation page got a 5.1% conversion rate. The old one got 4.2%. That looks like a win. But is it? With 800 visitors per group, random chance alone could easily produce a gap that size. This calculator tells you whether your result is real or whether you're reading tea leaves.
Enter the number of successes (conversions, signups, donations) and the total number of people in each group. The calculator does the rest.
Your Test Groups
Start with your control group. That's whatever you were running before the test. Then add one or more variations. If you tested multiple versions at once, add a row for each.
Confidence Level
How confident do you want to be before calling a winner? 95% is the standard. It means there's only a 5% chance the difference you're seeing is just random noise. If you're making a big, hard-to-reverse decision, bump it to 99%.
Reading the Results
The conversion rate for each group is shown with a confidence interval. That interval is the range where the true rate probably falls. If the intervals for two groups overlap a lot, be cautious about declaring a winner.
The p-value tells you the probability of seeing a difference this large if there were no real difference at all. Smaller is better. Below 0.05 (at 95% confidence) is the conventional threshold for "statistically significant." Below 0.01 is strong evidence.
The relative improvement tells you how much better (or worse) the variation performed compared to the control, as a percentage. A 20% relative improvement on a 5% baseline means the variation converted at about 6%.