Asymptotic Properties: Law of Large Numbers and Central Limit Theorem

When we use data, we’re almost always trying to understand a huge group (a “population”) by looking at a tiny piece of it (a “sample”). We take a poll of 1,000 people to guess what 100 million will do. We test the fuel efficiency of 50 cars to make a claim about the entire model line. We are, in essence, estimators. But how do we know our estimates are any good? What if our sample of 1,000 people was just… weird? What if we just happened to pick the only 50 cars that had a problem? Our estimate would be wrong. This is where the magic of “asymptotic properties” comes in. It’s a set of ideas that gives us confidence in our work, all built on one simple question: “What happens if our sample just keeps getting bigger?”

What are asymptotic properties anyway? 🔗

In econometrics and statistics, “asymptotic” is a fancy term for what happens as our sample size (which we call ‘$n$’) approaches infinity. We can’t ever *actually* collect an infinite amount of data, but understanding the behavior of our estimator “in the long run” is the next best thing. Asymptotic properties are the promises an estimator makes to us. They tell us if our estimation method is fundamentally sound and whether getting more data will *actually* lead us closer to the truth.

Imagine you’re lost in a forest with a compass that’s slightly off. With a small sample (walking 100 steps), you might end up far from the trail. But what if the compass was “asymptotically” good? It might mean that the more steps you take, the smaller its error becomes, guiding you closer and closer to the true path. We want our statistical tools to be just like that.

The gold standard: A ‘consistent’ estimator 🔗

The most important asymptotic property we look for is consistency. A consistent estimator is one that converges to the *true* population value as the sample size increases. In simpler terms: the more data you feed it, the more accurate it gets. If an estimator is consistent, it’s virtually guaranteed that a sufficiently large sample will give you an estimate that is arbitrarily close to the real answer.

To achieve consistency, two things generally need to happen as the sample size ‘$n$’ grows:

1. The bias must disappear. An estimator is “asymptotically unbiased” if its expected value (its average guess) converges to the true population parameter. Even if it’s a little biased with a small sample, we need that bias to melt away as we get more data.
2. The variance must shrink to zero. The variance tells us how “spread out” our estimates would be if we took many different samples. If the variance didn’t shrink, our estimator would still be wobbly and unreliable, even if it was unbiased on average. A consistent estimator becomes more and more precise.

Think of it like an archer. An “unbiased” archer’s arrows are centered around the bullseye, but they might be spread all over the target. A “consistent” archer, as they shoot more and more arrows (increase ‘$n$’), not only stays centered on the bullseye (asymptotic unbiasedness) but also gets all their arrows into a tiny, tight cluster right at the center (variance approaches zero).

The Law of Large Numbers (LLN): The foundation of consistency 🔗

How do we know that consistency is even possible? How can we be sure that the average of our sample isn’t just a random number? The first pillar holding up this entire idea is the Law of Large Numbers (LLN). This is one of the most intuitive, and powerful, laws in all of statistics. The LLN states that as a sample size grows, its sample mean gets closer and closer to the population mean (the “expected value”).

This law is the reason casinos are profitable. On any single hand of blackjack (a small sample), you might win or lose. But over millions of hands (a very large ‘$n$’), the casino *knows* the average amount they win per hand (the sample mean) will converge to their tiny mathematical house edge (the population mean). The LLN guarantees that in the long run, chance is smoothed out and the underlying average is revealed.

For an econometrician, the LLN is our guarantee that the sample mean ($\bar{X}$) is a consistent estimator for the population mean ($\mu$).

Weak LLN vs. Strong LLN 🔗

Statisticians, being precise, have two “flavors” of this law. While they are mathematically different, they both lead to the same practical conclusion for applied work.

• The Weak Law of Large Numbers (LLN): This states that the *probability* of the sample mean being “far away” from the population mean gets smaller and smaller as the sample size grows. It’s also known as “convergence in probability.”
• The Strong Law of Large Numbers (LLN): This is a more powerful guarantee. It states that the sample mean will (with probability 1, or “almost surely”) converge to the population mean. It doesn’t just say it’s *likely* to be close, it says it *will* get there and stay there.

For our purposes, both versions of the LLN give us faith in our most basic tool: the sample average. It tells us *where* our estimator is heading (to the truth).

The Central Limit Theorem (CLT): Giving our estimate a shape 🔗

The LLN is great, but it’s incomplete. It tells us our estimate will be *close* to the truth, but it doesn’t tell us *how* it’s distributed around that truth. If our sample size is large, how “close” is “close”? 5%? 0.1%? How confident can we be in our estimate?

This is where the second, and arguably most celebrated, pillar of statistics comes in: the Central Limit Theorem (CLT). The CLT is nothing short of magical. It says that if you take a population-*regardless of its shape*-and you draw many large, random samples, the distribution of the *sample means* from those samples will be approximately normal (a bell curve).

This is a huge deal. Your original data could be skewed, uniform, or a bizarre two-humped “bimodal” distribution. But the *distribution of the sample averages* will always be a nice, predictable bell curve. This allows us to use all the known properties of the normal distribution to do our work.

The three magic ingredients of the CLT 🔗

The CLT works its magic as long as our samples are independent and identically distributed (i.i.d.) and the population has a finite variance. When these conditions are met, the CLT tells us three very specific things about the distribution of sample means ($\bar{X}$):

1. It’s Normal: The distribution is approximately normal, especially when the sample size ‘$n$’ is large (often cited as $n > 30$).
2. Its Mean is $\mu$: The center of this bell curve (its mean) is the *same* as the true population mean ($\mu$) that we’re trying to estimate.
3. Its Variance is $\sigma^2/n$: The variance of the sampling distribution is the original population variance ($\sigma^2$) divided by the sample size ($n$).

This third point is critical. The standard deviation of this new bell curve, known as the standard error, is $\sigma/\sqrt{n}$. Notice that the sample size ‘$n$’ is in the denominator. This is the mathematical proof of our intuition: as our sample size ‘$n$’ gets bigger, the standard error gets *smaller*, and our bell curve of estimates gets *tighter and skinnier*. This means our estimate is much more precise, which is exactly what we want.

Using the CLT in the real world: The coin toss test 🔗

The CLT isn’t just a textbook theory; it’s the engine behind almost all hypothesis testing, from A/B tests on websites to clinical trials for new medicines. Let’s use the classic example: testing a coin.

You suspect a coin might be biased. Your “null hypothesis” is that the coin is fair, meaning the true population proportion of heads ($p$) is 0.5. To test this, you flip the coin 100 times ($n = 100$) and get 60 heads. Your *sample proportion* ($\hat{p}$) is $60/100 = 0.60$.

The question is: Is 0.60 “far enough” from 0.50 for us to reject the “fair coin” hypothesis? Or could this difference just be random luck?

Applying the theorem 🔗

A proportion is just a special kind of mean (where heads=1 and tails=0). Therefore, the CLT applies. It tells us that the sampling distribution of proportions will be normal, centered at the true proportion $p$, with a standard deviation of $\sqrt{p(1-p)/n}$.

We can “standardize” our result by calculating a Z-score. This score tells us exactly how many standard deviations away from the expected mean (0.50) our sample result (0.60) is.

The formula is: $Z = \frac{\hat{p} – p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}}$

Let’s plug in the numbers assuming our null hypothesis is true ($p_0 = 0.50$):

• $\hat{p}$ (our sample proportion) = 0.60
• $p_0$ (our hypothesized proportion) = 0.50
• $n$ (our sample size) = 100

$$Z = \frac{0.60 – 0.50}{\sqrt{\frac{0.50(1-0.50)}{100}}}$$

$$Z = \frac{0.10}{\sqrt{\frac{0.25}{100}}}$$

$$Z = \frac{0.10}{\sqrt{0.0025}}$$

$$Z = \frac{0.10}{0.05} = 2.0$$

What does a Z-score of 2.0 mean? 🔗

Our result of 60 heads is 2.0 standard deviations above the mean we would expect from a fair coin. Now we can ask: what is the probability of getting a result *at least* this extreme (2 or more standard deviations away) just by random chance?

We know from the properties of the normal distribution that about 95% of all results fall within 2 standard deviations of the mean. This means the probability of getting a result *outside* this range (either high or low) is only about 5%. The probability of getting a result this high or higher (a “one-tailed” test) is even smaller, at about 2.28%.

Because this probability is so low (typically, we use 5% as a cutoff), we would be justified in rejecting the null hypothesis. We have strong statistical evidence to conclude that the coin is *not* fair and is likely biased towards heads.

Without the Law of Large Numbers, we wouldn’t know if our sample mean was meaningful. And without the Central Limit Theorem, we’d have no idea how to test if it was “far enough” from a hypothesis to be significant. These asymptotic properties are the twin pillars that allow us to move from just *describing* data to *making inferences* about the world.

What do you think? The Central Limit Theorem relies on having a “large enough” sample, but “large enough” can depend on the shape of the original data. In your field or in examples you’ve seen, how do people practically decide what “large enough” actually means for a specific problem?

References

1. https://www.investopedia.com/terms/c/consistent.asp
2. https://www.statisticshowto.com/law-of-large-numbers/
3. https://www.stat.cmu.edu/~cshalizi/mreg/15/lectures/06/lecture-06.pdf
4. https://sphweb.bumc.bu.edu/otlt/mph-modules/bs/bs704_probability/BS704_Probability12.html
5. https://amsi.org.au/ESA_Senior_Years/Senior_Years_HTML/sampling-from-a-population/sampling-distribution-of-a-proportion.html