Beta Distribution
Your team ran two versions of a campaign email last week. Subject line A went to 2,000 supporters and got 480 opens, a 24% open rate. Subject line B went to a smaller test batch of 50 supporters and got 15 opens, a 30% open rate. Your campaigner calls it: B wins by six points, roll it out to the full list. The math looks obvious. 30 is bigger than 24. But those two percentages are hiding something. One is built on 2,000 observations. The other is built on 50. And that difference changes everything.
The beta distribution is a way to see what's hiding behind those percentages. It represents what you believe about an unknown rate or proportion, given the evidence you've collected so far. Instead of reducing your email results to a single number ("the open rate is 24%"), it gives you a curve showing the full range of plausible true rates and how likely each one is. The curve is tallest where the true rate is most likely and tapers off toward values that are less plausible. With very little data, the curve is wide and flat, meaning almost any rate between 0% and 100% is still on the table. With a lot of data, it becomes a narrow spike centered on the observed rate.
The shape of a beta distribution is determined by two inputs: the number of successes and the number of failures you've observed. For subject line A, that's 480 opens and 1,520 non-opens. For subject line B, it's 15 opens and 35 non-opens. More total observations produce a narrower curve. A higher observed rate shifts the peak to the right. This connects directly to Bayes' theorem. Before sending any emails, your belief about the true open rate is essentially flat. Every email that gets opened or ignored updates that belief, pulling the curve toward the observed rate and narrowing it. The beta distribution is what that updated belief looks like after each batch of evidence. And as the law of large numbers promised, the curve keeps tightening as you collect more data, converging on the truth.
When you're comparing two campaigns, the question isn't which percentage is higher. It's how much the two curves overlap. If A's curve and B's curve overlap heavily, you don't yet have enough evidence to be confident that one is truly better. If they barely overlap, the difference is probably real. You can even calculate the probability that B's true rate is higher than A's by measuring how often B's curve sits to the right. In the example above, A's curve is a tight spike at 24% because 2,000 emails leave almost no room for doubt. But B's curve is a wide, flat hill. With only 50 sends, B's true open rate could plausibly be anywhere from about 18% to 43%. That range extends well below A's 24%. When you calculate the probability that B is genuinely better than A, it comes out to roughly 85%. Your campaigner saw a six-point lead and called it a clear winner, but the beta distribution reveals a roughly 1-in-7 chance that A is actually the better subject line. Would you roll B out to your entire list on those odds?
This is the engine behind Bayesian A/B testing tools. Rather than asking "is the difference statistically significant?" and getting a yes-or-no answer, these tools use the beta distribution to ask "what's the probability that B beats A?" and give you a percentage you can actually reason about. In digital fundraising, when you're comparing conversion rates across two donation page designs that received unequal traffic, the beta distribution makes it obvious that the page with fewer visitors has a wider range of uncertainty, even if its raw conversion rate looks better. In petition campaigning, if you're testing two share-prompt messages and one has a 15% share rate from 80 signers while the other has 12% from 300 signers, the beta distribution will show you whether that lead is real or an illusion built on thin evidence. In supporter journey optimization, when you're evaluating which onboarding sequence better converts one-time petition signers into repeat activists, the beta distribution lets you make a decision calibrated to the actual strength of the evidence rather than to gut instinct or arbitrary thresholds.
A percentage from 30 observations and a percentage from 3,000 observations look identical on a dashboard. The beta distribution pulls back the curtain, showing you the uncertainty behind every rate so you can tell which numbers to trust and which are still guesswork.
See It
Adjust the number of emails sent and opened for two campaigns. Watch how the distributions narrow with more data and how the overlap determines which campaign you can trust.
Reflect
Think about a recent A/B test or campaign comparison at your organization. Did you compare raw percentages, or did you account for the different sample sizes behind each number? If one variant had half the data, how confident should you really have been in the "winner"?
When your team reports a conversion rate or open rate, is there any indication of how uncertain that number is? What would change if every rate on your dashboard came with a range showing the plausible true values?