Understanding the Specification of Multiple Regression Models

Imagine you’re trying to predict something complex, like the price of an apartment in a big city. Your first thought might be to use its size (square footage). A simple regression, Price = β₀ + β₁(Size) + ε, would probably show that bigger apartments cost more. But what about the neighborhood? The number of bedrooms? How new the building is? Suddenly, your simple model feels incomplete. You’re not just dealing with one factor, but a whole collection of them. This is the world of multiple regression, and it’s the workhorse of modern economics and data analysis. But how do you handle all that information-say, 10, 15, or even 50 variables-without writing an equation that stretches across an entire whiteboard?

The answer is elegance itself: we use the language of matrices. And once we’ve built this powerful model, how do we know if the answers it gives us are any good? That’s where the “rules of the game” come in-a set of critical checks known as the Classical Assumptions. Today, we’re going to build this model from the ground up, starting with its elegant blueprint and then inspecting the “rules” that make it a reliable tool for finding the truth.

Taming complexity: Why we use matrix notation 🔗

Let’s go back to our apartment example. A more realistic model would look something like this:

Price = β₀ + β₁(Size) + β₂(Bedrooms) + β₃(Neighborhood_Score) + β₄(Building_Age) + ε

This is already getting long, and we’ve only got four variables (predictors). Now imagine you’re a policy economist at NITI Aayog trying to model a state’s GDP using 20 different variables (investment, literacy, infrastructure spending, etc.). Writing this out for each of your ‘n’ observations (e.g., data for 100 different districts) would be a nightmare. It would look like:

GDP₁ = β₀ + β₁(Invest₁) + … + β₂₀(Infra₁) + ε₁

GDP₂ = β₀ + β₁(Invest₂) + … + β₂₀(Infra₂) + ε₂

…all the way to…

GDP₁₀₀ = β₀ + β₁(Invest₁₀₀) + … + β₂₀(Infra₁₀₀) + ε₁₀₀

This is not just tedious; it’s mathematically clumsy. We can’t easily solve for all the ‘β’ (beta) coefficients-the numbers that tell us the *effect* of each variable. Matrix algebra takes this massive, messy system of equations and condenses it into one clean, beautiful line.

The building blocks: Meet the vectors and matrices 🔗

To perform this magic trick, we just need to organize our data into four specific components. Think of them as the “cast of characters” in our story.

1. The ‘Y’ Vector (The Dependent Variable)
This is what we are trying to predict. It’s a simple column vector (a matrix with only one column) that stacks all ‘n’ of our observations on top of each other. For our 100-district GDP example, it would be a [100 x 1] vector.

Y = [GDP₁; GDP₂; …; GDP₁₀₀]

2. The ‘X’ Matrix (The Design Matrix)
This is the big one. It’s our master spreadsheet of all our explanatory variables (our ‘X’s). It will have ‘n’ rows (one for each observation) and ‘k+1’ columns (one for each of our ‘k’ variables, *plus* one extra column). This extra column, which we always put first, is just a column of ones. This simple trick is how we handle the intercept term (β₀). That column of ones acts as a “dummy variable” that is “on” for every single observation, so its coefficient, β₀, represents the baseline value of Y when all other X’s are zero.

For our GDP example with 20 variables, the X matrix would be [100 x 21] and look like this:

X = [1, Invest₁, …, Infra₁; 1, Invest₂, …, Infra₂; …; 1, Invest₁₀₀, …, Infra₁₀₀]

3. The ‘β’ (Beta) Vector (The Parameters)
This is the “treasure” we’re hunting for. It’s a [(k+1) x 1] column vector that holds all the unknown coefficients we want to estimate. It includes the intercept (β₀) and the slope coefficient for each of our ‘k’ variables.

β = [β₀; β₁; …; β₂₀]

4. The ‘ε’ (Epsilon) Vector (The Errors)
This is the “mystery” vector. It’s a [n x 1] column vector that contains all the unobserved error terms. The error term (εᵢ) for each observation is the difference between the *actual* observed value (GDPᵢ) and the value our model *would* have predicted. It represents everything else that affects GDP that we didn’t (or couldn’t) include in our model-random chance, measurement error, unobserved factors like local political climate, etc.

ε = [ε₁; ε₂; …; ε₁₀₀]

Putting it all together: The model in matrix form 🔗

Now, we assemble our cast. The entire system of 100 equations boils down to this:

Y = Xβ + ε

That’s it. It’s beautiful. But how does it work? Let’s look at the multiplication. When we multiply the X matrix [100 x 21] by the β vector [21 x 1], the rules of matrix multiplication give us a new [100 x 1] vector. Let’s look at the very first row:

(Row 1 of X) * (β vector) = (1 * β₀) + (Invest₁ * β₁) + … + (Infra₁ * β₂₀)

So, our matrix equation `Y = Xβ + ε` really says:

[GDP₁; GDP₂; …; GDP₁₀₀] = [(β₀ + β₁Invest₁ + …); (β₀ + β₁Invest₂ + …); …; (β₀ + β₁Invest₁₀₀ + …)] + [ε₁; ε₂; …; ε₁₀₀]

If you look at just the first row, it reads: GDP₁ = β₀ + β₁Invest₁ + … + β₂₀Infra₁ + ε₁. It perfectly recreates our original equation! This compact matrix formulation is the foundation for all modern econometrics. It allows us to use matrix algebra to find the Ordinary Least Squares (OLS) solution for *all* our coefficients at once with another famous formula: β̂ = (X’X)⁻¹X’Y. A computer can solve this in an instant.

The rules of the game: Classical assumptions for reliable results 🔗

So, our computer has used that formula and given us our “treasure”-a list of estimated beta coefficients (we call them ‘β̂’ or “beta-hat” to show they are estimates). For example, it might tell us that β₁ (the coefficient for investment) is 0.2, suggesting a 1-crore rupee increase in investment is associated with a 0.2-crore (or 20 lakh) rupee increase in GDP.

But… can we trust this number? Is it the *best* possible estimate? Is it even *correct* on average?

The answer is “yes,” *if* our model and data follow a set of rules. These rules are the Classical Linear Regression Model (CLRM) Assumptions. If they hold true, the OLS estimator we just used is BLUE: the Best Linear Unbiased Estimator. This means our estimates are, on average, correct (unbiased) and have the smallest possible variance among all linear estimators (best/efficient). Let’s look at the most important assumptions.

Assumption 1: The error term has a zero mean 🔗

This assumption, E(εᵢ | X) = 0, is quite simple. It means that, on average, the “mistakes” our model makes (the errors) are zero. For any given set of X variables (e.g., for all 3-bedroom, 1500 sq. ft. apartments), our predictions are sometimes a little high, sometimes a little low, but they don’t have a *systematic* bias in one direction. As this summary based on Gujarati’s classic textbook notes, if the mean of the error *wasn’t* zero (say, it was +5), this uncaptured value would simply get absorbed by the intercept term (β₀), leading to a biased estimate of the true intercept.

Assumption 2: Homoscedasticity (constant error variance) 🔗

This is a big word for a simple idea: Var(εᵢ | X) = σ². It means the variance (or “spread”) of the errors is the same, no matter what the values of our X variables are. Think of it as our model having a consistent level of (un)certainty.

Relatable example: Imagine modeling household spending (Y) based on household income (X).

• Homoscedasticity (Good): The model’s predictions for low-income households have about the same “margin of error” as its predictions for high-income households. The scatter plot of errors looks like a random cloud.
• Heteroscedasticity (Bad): This is the opposite and a common problem. At low-income levels, spending is very predictable (mostly on necessities), so the model’s errors are small. At high-income levels, spending choices are vast (savings, luxury, travel), so the model’s errors are *all over the place*-some high-income people save a lot, others spend a lot. The plot of errors would look like a “fanning out” or cone shape.

Why it matters: If we have heteroscedasticity, our OLS estimates are still unbiased, but they are no longer “best” (efficient). More critically, the formulas our software uses to calculate standard errors (which determine p-values and t-stats) become wrong. We might conclude a variable is statistically significant when it isn’t, or vice-versa. We’d be making policy decisions on faulty evidence.

Assumption 3: No autocorrelation 🔗

This assumption, Cov(εᵢ, εⱼ) = 0 for i ≠ j, means the errors are independent of each other. A mistake in one observation gives us no information about the mistake in another. This is almost always a safe assumption for cross-sectional data (like our apartment or GDP-district data). The error for apartment #5 has no reason to be related to the error for apartment #82.

When is this a problem? This is the single biggest-selling “gotcha” for time-series data. Imagine modeling monthly ice cream sales (Y) based on price (X). If an unobserved factor (like a heatwave, which is in the error term) causes you to *under-predict* sales in June (a large positive error), that heatwave is probably still there in July. This will cause another positive error. The errors are correlated over time. This is autocorrelation.

Why it matters: Just like heteroscedasticity, autocorrelation makes our OLS estimators inefficient and, more importantly, it invalidates our standard errors, p-values, and F-statistics. We can’t trust our hypothesis tests.

Assumption 4: Explanatory variables and errors are uncorrelated 🔗

This assumption, Cov(Xⱼᵢ, εᵢ) = 0, is perhaps the most important and most difficult. It says that our explanatory variables are not correlated with the “unexplained part” (the error term). If this assumption is violated, we have a fatal problem called endogeneity.

Relatable example: Imagine we model a student’s exam score (Y) based on their hours spent in tutoring (X). We run the regression and find tutoring has a huge positive effect. But what if a student’s *underlying motivation* (which we can’t measure, so it’s in the error term, ε) is what *really* drives their success? It’s very likely that more motivated students (high ε) *also* seek out more tutoring (high X). In this case, X and ε are positively correlated. Our model will mistakenly give “tutoring” credit for the effect that “motivation” is actually having. We are getting a biased and inconsistent estimate. This is a huge problem in economics, often called “omitted variable bias.”

Assumption 5: Full rank of the X matrix (no perfect multicollinearity) 🔗

This is a more technical assumption, but it’s easy to understand. It means none of our explanatory variables is a perfect linear combination of any of the others. Each variable must bring some *unique* information to the table.

Relatable example: You try to model a person’s weight (Y) using two variables: `height_in_feet` (X₁) and `height_in_inches` (X₂). This is perfect multicollinearity. You can perfectly predict X₂ from X₁ (since `X₂ = 12 * X₁`). When the computer tries to solve `β̂ = (X’X)⁻¹X’Y`, it finds that the `(X’X)` matrix is “singular” and its inverse doesn’t exist. The math simply breaks. The model cannot figure out whether to assign the “effect of height” to the feet variable or the inches variable.

A more subtle version is high (but not perfect) multicollinearity, for example, including `household_income` and `luxury_car_value` in the same model. They aren’t *perfectly* related, but they are very highly correlated. The model will run, but the standard errors on *both* coefficients will become huge, making it hard to separate their individual effects. The model might tell you “neither of these variables is significant,” even when together they are very powerful predictors.

In short, the matrix form `Y = Xβ + ε` gives us an incredibly powerful tool. But that power is only as good as its weakest link. A good economist or data scientist doesn’t just *run* regressions; they *diagnose* them, carefully testing these classical assumptions to ensure that the answers they get are not just numbers, but numbers that we can actually trust.

What do you think? Which of these assumptions seems the most likely to be violated in real-world economic data, and why? Can you think of another example of perfect multicollinearity that might accidentally sneak into a model?

References

1. https://online.stat.psu.edu/stat501/lesson/5/5.4
2. https://www.scribd.com/document/716565106/Handouts-Ch-3-Gujarati