A data scientist is running an online A/B test that records whether each visit results in a click (1) or no click (0). Every 5 minutes she streams the cumulative average click-through-rate (CTR) difference between variant B and the control. During the first hour (≈10 000 observations) the cumulative difference swings between −2 % and +3 %, but after ten hours (≈100 000 observations) it fluctuates within ±0.3 % of a stable value. Which statistical result best explains why the running average becomes increasingly stable as more independent observations arrive, assuming each user's CTR has finite variance?
The law of large numbers states that for independent, identically distributed observations with a finite expected value, the sample mean converges in probability (and, under the strong form, almost surely) to the true population mean as the sample size grows. Consequently, as the A/B test accumulates additional visitors, the running average CTR difference drifts less and eventually stabilizes around the underlying expected difference.
The central limit theorem tells us that the distribution of the scaled deviation of the sample mean approaches normality, but it does not itself guarantee that the mean will settle at the population value. Chebyshev's inequality provides only a broad bound on the probability of large deviations at any fixed sample size, without ensuring eventual convergence. Simpson's paradox refers to misleading aggregated comparisons and is unrelated to the convergence of sample averages.
Ask Bash
Bash is our AI bot, trained to help you pass your exam. AI Generated Content may display inaccurate information, always double-check anything important.
What is the law of large numbers?
Open an interactive chat with Bash
How is the law of large numbers different from the central limit theorem?
Open an interactive chat with Bash
Why is Chebyshev's inequality not sufficient in this scenario?