Exploring the Properties of OLS Estimators in Multiple Regression

Imagine you’ve just bought a brand-new, complex toolkit for a big project, like building a custom piece of furniture. Before you trust it to make critical cuts, you’d want to know how it works, right? You’d want to know: Is this saw accurate? Does this drill run true? Will these tools get the job done reliably, or will they make a mess of your expensive materials?

In the world of economics and data analysis, the Ordinary Least Squares (OLS) estimator is one of the most fundamental and widely used “tools” we have. We use it to build models that help us understand the world, from how education affects income to how interest rates influence inflation. But just like that new toolkit, we can’t just *assume* it works. We need to check its “specs”-its properties. These properties tell us what we can expect from our OLS estimates and under what conditions we can trust them. They are the quality assurance that separates rigorous analysis from blind guesswork.

These properties are typically broken into two fascinating categories: the “algebraic” properties, which are mechanical truths about how the OLS math works, and the “statistical” properties, which tell us how our estimator behaves when we try to learn about the real world. Let’s dive in and explore this essential toolkit.

The ‘house rules’: Algebraic properties 🔗

The first set of properties are called algebraic or numeric properties. Think of these as the “house rules” of OLS. They are not assumptions about the world; they are mathematical facts that are *always* true, simply as a result of using the OLS method to draw a line through a set of data points. They are the direct consequences of the OLS calculation, which, as you’ll recall, is designed to minimize the sum of the squared residuals (the errors).

So, what are these rules? They all revolve around the residuals, which are the differences between the actual observed values (Y) and the values predicted by our model ($\hat{Y}$). We call these residuals $\hat{U}$.

The good, the bad, and the zero sum 🔗

The most basic algebraic property is that the sum of the OLS residuals is always zero (assuming the model includes an intercept, which it almost always does). In simple terms, this means that the model’s over-predictions (positive residuals) and under-predictions (negative residuals) perfectly cancel each other out. The regression line is “pulled” into the data until it finds this perfect point of balance. It’s a bit like a seesaw with the data points on either end; the OLS line is the pivot point, and it adjusts itself until the seesaw is perfectly level.

Residuals and regressors: A strict ‘no-contact’ rule 🔗

This is where things get really interesting. A core algebraic property states that the residuals ($\hat{U}$) are uncorrelated with each of the explanatory variables (the ‘X’ variables). In matrix notation, this is famously written as $X’\hat{U} = 0$.

What does this mean in plain English? It means that, within our sample, there is no lingering information in the “leftovers” (the residuals) that can be explained by our “ingredients” (the X variables). Imagine you’re trying to explain a person’s salary (Y) using their years of education (X). After you fit your OLS model, you get your residuals. If you found that people with *more* education tended to have *higher* positive residuals (meaning your model consistently under-predicted their salaries), it would imply your education variable wasn’t doing its job properly. It would mean there’s still a relationship between education and the part of the salary you *failed* to explain.

The OLS procedure mathematically guarantees this doesn’t happen. By minimizing the sum of squared residuals, the OLS formula “forces” the residuals to be uncorrelated with the explanatory variables. It squeezes every last drop of explanatory power out of the X variables until the remaining residuals are, by construction, “indifferent” to them.

Residuals and predictions: Also not on speaking terms 🔗

In a very similar vein, the OLS residuals are also uncorrelated with the predicted values ($\hat{Y}$). This is written as $\hat{Y}’\hat{U} = 0$. This makes perfect sense, because the predicted values ($\hat{Y}$) are just a linear combination of the X variables. If the residuals are already uncorrelated with all the $X$ variables, they must also be uncorrelated with any combination of them.

This property reinforces the idea that your model’s predictions have “used up” all the available information from the explanatory variables. The residuals represent the part of Y that is completely, 100% “other” than the prediction. The two components, $\hat{Y}$ (the explained part) and $\hat{U}$ (the unexplained part), are mathematically separated and have no relationship with each other in the sample data.

These algebraic properties are neat, but they are just mechanical outcomes. They don’t tell us if our model is “good” or “true.” They just tell us what OLS *does*. To know if our model is “good,” we need to move on to statistical properties.

The ‘real world’ tests: Statistical properties 🔗

Statistical properties are not about what happens in our one, specific sample. They are about how the OLS estimator would behave if we could repeat our experiment-draw new samples from the population-over and over again. These properties are what allow us to make inferences about the *true* underlying parameters (the $\beta$ values) that govern the relationship in the real world.

This is a critical distinction: To talk about statistical properties, we must make a set of assumptions about the *data-generating process* (the “true” model). These are the Classical Linear Regression Model (CLRM) assumptions, which include things like the model being linear in parameters and the error term having a zero conditional mean. When these assumptions hold, OLS has some truly wonderful statistical properties.

Unbiasedness: Hitting the target on average 🔗

The first, and perhaps most celebrated, statistical property is unbiasedness. An estimator is unbiased if its expected value (the average of its estimates across infinite repeated samples) is equal to the true population parameter it’s trying to estimate. Mathematically, we say $E(\hat{\beta}) = \beta$.

Think of it like target practice. An unbiased estimator is like a rifle that, on average, hits the bullseye. Any single shot (any single sample) might be slightly high, low, left, or right. But if you took a thousand shots and averaged them all out, the center of that cluster would be right on the bullseye. A *biased* estimator, by contrast, would be like a rifle with a bent sight-even if your shots are tightly clustered, the center of that cluster is always, say, two inches to the right of the bullseye. It’s systematically wrong.

Unbiasedness in OLS means that our formula for calculating the coefficients doesn’t have any built-in tendency to overestimate or underestimate the true effect. It’s a “small-sample” property, meaning it holds true for any sample size, as long as the CLRM assumptions (especially the zero conditional mean assumption) are met. This is a very comforting property. It tells us that our method is “aimed” in the right direction.

Consistency: Getting closer with more data 🔗

Unbiasedness is great, but it’s not the only thing we care about. What if our “unbiased” rifle scatters shots all over the target? We’re not very confident in any *single* shot. This is where consistency comes in. Consistency is a “large-sample” property.

An estimator is consistent if, as the sample size ($n$) gets larger and larger (approaching infinity), the estimates ($\hat{\beta}$) get closer and closer to the true population parameter ($\beta$). In fact, they converge *to* the true value. The distribution of the estimator “collapses” around the true parameter.

Let’s go back to our archer analogy. A consistent estimator is like an archer who may be a bit shaky at first, but with every piece of new information (more data), their aim gets steadier and their arrows land in an ever-tighter cluster around the bullseye. Given enough data, a consistent estimator will get you arbitrarily close to the true population value.

This is an incredibly important property in economics, where we often rely on large datasets. Even if an estimator is slightly biased in small samples (which OLS is not, but other estimators can be), if it’s consistent, we can be confident in it as long as our sample is “large enough.” It gives us faith that gathering more data is a worthwhile pursuit that will, in fact, improve our estimates.

Why OLS is the ‘best’ (with an asterisk) 🔗

So, we’ve established that OLS is aimed at the right target (unbiased) and gets better and better with more data (consistent). But is it the *best* way to get there? This is where the most famous theorem in introductory econometrics comes in.

Meet BLUE: Best linear unbiased estimator 🔗

Under the CLRM assumptions, the OLS estimator is BLUE. This is an acronym that packs a huge amount of information and is worth breaking down letter by letter.

• Best: This is the key. “Best” here means minimum variance. Among all the estimators in its class, the OLS estimator produces estimates that are the most “tightly clustered” around the true value. It has the smallest variance, which means it is the most efficient and precise.
• Linear: This means the estimator is a linear function of the dependent variable, $Y$. Our $\hat{\beta}$ is calculated as a weighted sum of the $Y$ values.
• Unbiasised: We’ve already covered this. Its average estimate across many samples is the true population parameter.
• Estimator: It’s a formula, or “estimator,” used to estimate a parameter.

So, OLS is the “Best Linear Unbiased Estimator.” In our target analogy, this means that if you compared OLS to *any other* “linear unbiased” rifle, OLS would be the one that has the tightest shot grouping around the bullseye. It’s not just aimed correctly; it’s also the most stable and precise. This means any single estimate we get from our sample is more likely to be close to the true value than an estimate from a different, less efficient (higher variance) estimator.

The engine behind it all: The Gauss-Markov theorem 🔗

How do we know OLS is BLUE? We know because of the Gauss-Markov theorem. This is the formal “certificate of quality” for OLS. The theorem, first formulated by Carl Friedrich Gauss and later extended by Andrey Markov, provides the mathematical proof.

It states that: Given the set of classical linear model assumptions (specifically, the first five, often called the Gauss-Markov assumptions), the Ordinary Least Squares (OLS) estimator is the Best Linear Unbiased Estimator (BLUE) of the regression coefficients.

The assumptions it relies on are crucial:

1. The model is linear in its parameters.
2. We have a random sample of data.
3. There is no perfect multicollinearity (our X variables aren’t redundant).
4. The error term has a zero conditional mean ($E(u|X) = 0$). This is the key assumption for unbiasedness.
5. The error term has a constant variance, or homoscedasticity. This is the key assumption for the “Best” part. If this assumption fails (a condition called heteroscedasticity), OLS is still linear and unbiased, but it is no longer the “best.” Its variance can be improved upon.

The Gauss-Markov theorem is the reason OLS is the starting point for almost all econometric analysis. It assures us that, as long as our assumptions hold, we are using the most efficient, unbiased, linear tool available. It’s the theoretical bedrock that gives us confidence in our $\hat{\beta}$ estimates. Understanding these properties-from the simple mechanics of the algebraic rules to the profound implications of unbiasedness, consistency, and the BLUE property-is what allows us to move from just *running* a regression to truly *understanding* what it tells us about the world.

What do you think? In your experience, or in the types of economic data you can imagine, which of the Gauss-Markov assumptions (like constant variance or the error being uncorrelated with the X variables) do you think is the most likely to be violated in the real world, and why?

References

1. https://en.wikipedia.org/wiki/Ordinary_least_squares#Classical_linear_model_assumptions
2. https://www.statlect.com/fundamentals-of-statistics/Gauss-Markov-theorem
3. https://stats.stackexchange.com/questions/61657/understanding-and-interpreting-consistency-of-ols
4. https://statisticsbyjim.com/regression/gauss-markov-theorem-ols-blue/