Law of Large Numbers

Your organization just launched a new SMS campaign channel. After the first 20 messages go out, 7 recipients click the action link. That's a 35% click rate, which would be extraordinary. Your campaigner is thrilled. By the end of the week, after 200 messages, the click rate has dropped to 18%. By the end of the month, with 2,000 messages sent, it sits at 11.4%. Nobody changed the message. Nobody changed the audience. The rate found its level.

This is the law of large numbers at work, and it explains something that trips up nonprofit teams constantly. When you're working with small amounts of data, your observed results can be wildly different from the true underlying rate. As you gather more data, the observed average moves closer and closer to the true value. Not approximately. Not sometimes. Always, given enough observations.

The logic is intuitive once you see it. If a coin is fair, you won't be surprised to get 3 heads out of 5 flips. That's 60%, not 50%, but it doesn't feel wrong. Flip it 10,000 times and you'll be very surprised if the result isn't between 49% and 51%. Each additional flip has a tiny effect on the running average, nudging it ever closer to the truth. The wild swings that are possible with small samples become mathematically impossible with large ones.

If you read yesterday's entry on the central limit theorem, you might wonder how these two ideas differ. They're close relatives that answer different questions. The central limit theorem describes what happens when you take many separate samples and look at the shape of their averages. It says the shape is a bell curve. The law of large numbers describes what happens within a single sample as it grows. It says the average converges to the truth. One is about the distribution across many samples. The other is about convergence within one.

This distinction shows up in your daily work more than you might think. Every time your team launches a new email template, campaign page, or fundraising appeal, you're watching a single sample grow over time. The law of large numbers is the reason those early numbers bounce around and the later numbers settle down. When your email open rate is 30% after 50 sends and 21% after 2,000 sends, the second number isn't "worse" than the first. It's more honest. Early results rest on too few observations to mean much. The law of large numbers guarantees that patience will reveal the true rate, whatever it turns out to be.

In A/B testing, this is why you need to wait before calling a winner. Your testing platform might show variant B with a 25% conversion rate and variant A at 18% after 40 visitors each. With so little data, those rates haven't settled. Check again after 2,000 visitors per variant and you'll get a far clearer picture. In petition analytics, comparing share rates across petition topics is unreliable when any of the petitions have fewer than a few hundred signers. The standard deviation of a rate shrinks as your sample grows, which is the law of large numbers written into the math. In digital fundraising, if your Monday email to 300 supporters shows a 4.5% conversion rate while Tuesday's email to a different 300 shows 2.1%, that gap could easily be noise. Before concluding that Monday supporters convert better, ask whether 300 observations per group is enough for those rates to have stabilized.

The law of large numbers is a guarantee: collect enough data and the truth emerges. The practical question is always whether you've collected enough.

See It

Click "Send 10" or "Send 100" to simulate emails and watch the running open rate converge. Click "New Run" to start another trace and see how different runs all find the same truth.

Reflect

Think about the last time your team reacted to campaign results within the first day or two of a launch. How many observations did you have? If you had waited for ten times as many, would the conclusion have changed?

When you compare performance across campaigns or channels, are you comparing metrics that have had enough data to stabilize, or are some of those numbers still bouncing around on small samples?