When you look at a cloud of data points on a chart, how do you draw the “best” line through them? When you flip a coin 10 times and get 7 heads, whatโs your best guess for its true probability of landing on heads? These questions are at the heart of economics and data science. We almost never know the true, underlying “parameters” of a population-the real relationship between interest rates and inflation, or the true average spending of a consumer. All we have is a sample of data. The entire field of econometrics is built on using that sample to make an educated, defensible guess about the real world. This process of “guessing” is called estimation.
But not all guesses are created equal. We need a consistent, logical “recipe” for getting from our data to our estimate. In econometrics, there are three primary recipes, or philosophies, for doing this: the Method of Least Squares, the Method of Maximum Likelihood, and the Method of Moments. Each has a different way of defining what makes an estimate “good,” and each is a powerful tool in an economist’s toolkit.
Table of Contents
The big three of estimation methods
Imagine you have a simple model. You theorize that a student’s exam score depends on how many hours they study. Your model might be a simple line: $Score = \beta_0 + \beta_1 \times Hours + \epsilon$. In this model, the “parameters” we need to estimate are $\beta_0$ (the score a student would get with zero hours of study) and $\beta_1$ (the boost in score for each additional hour). We also have the error term $\epsilon$, which captures everything else (like talent, sleep, or luck).
We go out and collect data from 50 students (our sample). Now what? How do we use this sample to find the specific numbers for $\beta_0$ and $\beta_1$ that best represent the *true* relationship? This is where our three methods come in, each offering a different strategy.
The Method of Least Squares: Minimizing the ‘miss’
This is perhaps the most intuitive and famous of all estimation methods, forming the bedrock of linear regression. The Method of Least Squares has a very simple goal: find the line that is “closest” to all the data points at the same time.
How do we define “closest”?
Imagine you’ve drawn a potential line through your scatter plot of study hours versus exam scores. For any single student, the vertical distance between their *actual* score (the data point) and the score *predicted* by your line is called a residual. You can think of it as the model’s “error” or “miss” for that specific person. Some points will be above the line (positive residuals), and some will be below (negative residuals).
If we just added up all these residuals, the positive and negative ones would cancel each other out, giving us a useless measure of fit. To fix this, the Method of Least Squares squares every single residual first (making them all positive) and then adds them all up. This total is called the Sum of Squared Residuals (SSR).
The principle of Least Squares is simple: the “best” line is the one that makes this total SSR as small as humanly possible. It finds the parameters (the $\beta_0$ and $\beta_1$) that minimize the sum of the squared errors. This minimization problem is solved using calculus. By taking the derivative of the SSR with respect to each parameter and setting it to zero, we get a system of equations called the normal equations. Solving these equations gives us our Ordinary Least Squares (OLS) estimators, which are the values for our parameters that provide the “best fit” under this definition.
The Method of Maximum Likelihood (MLE): Finding the most probable cause
The Method of Maximum Likelihood (MLE) takes a completely different, and arguably more profound, philosophical approach. Instead of minimizing errors, it asks a backward-sounding question: “Given the data I actually collected, what set of parameters would have made this specific outcome the most probable?”
Letโs use a simple analogy. Imagine you have a bag of coins. You don’t know if they are fair (50% heads) or biased. You pull one coin out, flip it 10 times, and get 7 heads. Now, you have to guess the *true* probability of heads (let’s call it $p$) for *that specific coin*.
- What if the *true* $p$ was 0.5 (a fair coin)? The probability of getting 7 heads out of 10 is possible, but not incredibly high (it’s about 11.7%).
- What if the *true* $p$ was 0.1? The probability of getting 7 heads would be astronomically low.
- What if the *true* $p$ was 0.7? The probability of getting 7 heads is highest under this assumption (it’s about 26.7%).
The MLE approach says your best guess for $p$ is the one that makes your observed data (7 heads) most likely. In this case, your MLE for $p$ would be 0.7.
To do this formally, we construct a likelihood function. This function, written as $L(parameters | data)$, is the joint probability of observing your *entire* sample, treated as a function of the unknown parameters. We then use calculus to find the specific parameter values that maximize this likelihood function. Because multiplying many small probabilities is computationally difficult, we almost always maximize the log-likelihood function instead (since the logarithm turns products into sums, and the $log$ function is maximized at the same point as the original function). The equations we solve to find this maximum are called the likelihood equations.
Deriving some common MLEs
This method gives us intuitive answers for many standard distributions:
- For a Binomial distribution: The problem is like our coin flip. If we have $m$ experiments of $n$ trials each and observe a total of $\sum x_i$ successes, the MLE for the probability of success $p$ is $\hat{p} = \frac{\sum x_i}{mn}$. This is just the total number of successes divided by the total number of trials-the sample proportion.
- For a Poisson distribution: This models count data (e.g., number of emails arriving per hour). If we observe a sample of counts $x_1, x_2, …, x_n$, the MLE for the rate parameter $m$ is simply $\hat{m} = \bar{x}$, the sample mean. If you observe 5, 3, and 7 customers in three different hours, your best guess for the true average rate is (5+3+7)/3 = 5.
- For a Normal distribution: For the familiar bell curve, we need to estimate two parameters: the mean $\mu$ and the variance $\sigma^2$. The MLE for $\mu$ is, unsurprisingly, the sample mean $\bar{x}$. The MLE for $\sigma^2$ is $\frac{\sum(x_i – \bar{x})^2}{n}$, which is the sample variance (though with $n$ in the denominator, not $n-1$).
Why econometricians love MLE: The properties
MLE is a cornerstone of modern econometrics, far beyond simple linear regression (it’s how models like Logit, Probit, and countless others are estimated). This is because MLEs have a set of beautiful properties, especially as the sample size (n) gets large. We call these asymptotic properties.
- Consistency: This is a must-have. A consistent estimator is one that gets closer and closer to the *true* parameter value as the sample size increases. MLEs are consistent.
- Asymptotic Efficiency: Among all consistent estimators, the MLE has the *smallest possible variance* (in large samples). This means it’s the most precise guess you can make. It squeezes the most information out of your data.
- Asymptotic Normality: As the sample size grows, the distribution of the MLE estimator itself becomes a normal (bell) curve, centered around the true parameter. This is the magic property that allows us to construct confidence intervals and conduct hypothesis tests (e.g., calculate p-values).
- Invariance: This is a fantastic “math shortcut.” If $\hat{\theta}$ is the MLE for a parameter $\theta$, then the MLE for some *function* of that parameter, say $g(\theta)$, is simply $g(\hat{\theta})$. For example, if the MLE for a proportion is $\hat{p}$, the MLE for the *odds* ($\frac{p}{1-p}$) is simply $\frac{\hat{p}}{1-\hat{p}}$.
One small note: MLEs are not always unbiased in small samples. The MLE for the normal variance (dividing by $n$) is slightly biased. The unbiased estimator divides by $n-1$. However, this bias disappears as $n$ gets large, which is why we say MLEs are *asymptotically* unbiased.
The Method of Moments: A simple matching game
Finally, we have the Method of Moments (MoM). This is often the simplest and most straightforward of the three, and itโs built on a very simple idea: the sample’s characteristics should match the population’s characteristics.
These “characteristics” are called moments.
- The first moment is the mean (the expected value, $E[X]$).
- The second moment is related to the variance (the expected value of $X^2$, $E[X^2]$).
In any statistical distribution, these theoretical moments are functions of the distribution’s parameters. For example, in a Bernoulli distribution (one coin flip), the theoretical mean (first moment) is simply $p$. In a Poisson distribution, the theoretical mean is $m$.
The Method of Moments works in three steps:
- Write down the theoretical moments (e.g., $E[X] = p$).
- Calculate the corresponding sample moments from your data (e.g., the sample mean, $\bar{x}$).
- Equate the moments and solve for the parameters.
For our Bernoulli (coin flip) example, we set the theoretical mean equal to the sample mean: $p = \bar{x}$. The MoM estimator for $p$ is just the sample proportion of successes. For the Poisson example, we set the theoretical mean equal to the sample mean: $m = \bar{x}$. In many simple cases, the MoM estimator is exactly the same as the MLE, but this is not always true.
If you have two parameters to estimate (like the mean $\mu$ and variance $\sigma^2$ of a normal distribution), you simply set the first *two* sample moments equal to the first two theoretical moments and solve the resulting system of two equations.
While often simpler to calculate, MoM estimators are not always as *efficient* (precise) as MLEs. However, this method is the conceptual foundation for a much more advanced and powerful technique called the Generalized Method of Moments (GMM), which is a workhorse in advanced econometrics for situations where MLE is too difficult or the model’s assumptions are uncertain.
Which method is ‘best’?
There is no single “best” method. The “best” one depends on the job at hand.
- Least Squares (OLS) is the undisputed champion of linear regression. It’s computationally fast, easy to interpret, and has excellent properties when its assumptions are met.
- Maximum Likelihood (MLE) is more flexible and powerful. It’s the engine behind most non-linear models (like those predicting choices or counts) and is loved for its outstanding asymptotic properties.
- Method of Moments (MoM) is a simple, intuitive starting point and the conceptual basis for the highly robust GMM, which can handle very complex economic models.
Understanding these three philosophies isn’t just academic. It’s about knowing *how* an econometric model is coming up with its answers, allowing you to trust, critique, and properly interpret the numbers that shape our understanding of the world.
What do you think? When you see a “best-fit line” on a news report, do you find it more intuitive to think of it as the line that “minimizes errors” (Least Squares) or the line that makes the data “most probable” (Maximum Likelihood)? Does this distinction change how you think about the line’s meaning?
Leave a Reply