Cautions in Econometric Model Selection: Beyond Statistical Criteria

Imagine you’re a detective trying to solve a complex case. You have a room full of clues: fingerprints, witness statements, timelines, and security footage. If you just grab the clue that looks the shiniest-say, a single, muddy footprint-and build your entire case around it, you might end-up accusing the mailman while the real culprit slips away. Why? Because you ignored the bigger picture, the underlying story, and the simple logic of the situation. Econometric modeling, the art and science of using data to understand economic life, is surprisingly similar. We have powerful statistical tools, our “clues,” but if we use them blindly, we risk telling the wrong story.

In econometrics, we often build models to explain things like “What drives GDP growth?” or “How does education affect income?” To do this, we’re tempted to let software pick the “best” model by looking at statistical criteria like a high R-squared or a low AIC. But this is where the detective’s mistake comes in. Relying only on these numbers is like grabbing the shiniest clue and ignoring the case file. The most important tool in your kit isn’t a statistical test; it’s economic theory. This post is a word of caution: in the dance between data and theory, theory must always lead.

The primacy of economic theory 🔗

Before you ever run a regression, your most important job is to think. What does economic theory tell you about the relationship you’re studying? If you’re trying to model consumer demand for, say, mangoes in India, your first stop isn’t a dataset. It’s a textbook. Basic microeconomics tells you that the demand for a good should depend on its own price, the prices of substitutes (like bananas or apples), the prices of complements (perhaps cream or sugar), and consumer income. This gives you a theoretically-sound starting point.

This is what we call the primacy of economic theory. A model that is statistically impressive but makes no economic sense is, quite frankly, useless. It might even be dangerous, as it could lead to poor business or policy decisions. Your model must be built on a foundation of sound economic reasoning. This means economic theory should guide your choice of variables, the functional form of your equations, and the expected signs of your coefficients (e.g., you’d expect a negative relationship between price and quantity demanded).

Why ‘data mining’ is a dirty word 🔗

What happens when we ignore theory? We fall into the trap of “data mining.” This is the practice of throwing hundreds of potential variables into a statistical program and letting it “mine” for any combination that produces a high R-squared. The result? You might find that rainfall in Chennai has a statistically significant relationship with stock market returns in Mumbai. Does this mean you should base your investment strategy on the weather report? Absolutely not.

This is a classic case of correlation without causation. The statistical “fit” is a mirage, a random pattern in the data that has no real-world, logical connection. A theory-first approach protects you from this. By forcing yourself to justify every variable’s inclusion based on economic logic, you build a model that is robust and interpretable, not just a black box that happens to fit the numbers well. Reviewing existing literature-seeing what other economists have found when studying the same topic-is a crucial part of this process.

Beware of spurious and confounded relationships 🔗

One of the first things you learn in any statistics class is a simple, powerful mantra: correlation does not imply causation. Just because two variables move together does not mean one is causing the other. When we build models without a theoretical anchor, we are extremely vulnerable to two specific types of correlation traps: spurious relationships and confounded relationships.

The trap of the spurious relationship 🔗

A spurious relationship is an association between two variables that appears to be causal but is not. Often, it’s just a result of pure chance or, more commonly, both variables are being driven by a third, unseen factor. The classic, funny example is the strong positive correlation between ice cream sales and shark attacks. As ice cream sales rise, so do shark attacks. Does this mean buying a cone puts you at risk?

Of course not. The hidden variable, or “lurking” variable, is hot weather. When it’s hot, more people buy ice cream. When it’s hot, more people also go swimming in the ocean, which unfortunately increases the chances of a shark encounter. Ice cream and shark attacks aren’t related to each other; they are both related to the weather. In econometrics, this could happen if you find a correlation between, for example, the import of luxury cars and the number of new engineering colleges. There is likely no direct link. Both are probably being driven by a third factor: overall economic growth (GDP).

The problem of the confounder 🔗

Confounding is a specific and more troublesome type of spurious relationship. A confounding variable is a variable that is related to *both* your independent variable (the one you think is the cause) and your dependent variable (the one you think is the effect). Because of this, it “confounds” your ability to see the true relationship.

Let’s say you’re trying to build a model to see if attending coaching classes (Variable X) leads to higher exam scores (Variable Y). You collect data and find a strong positive correlation. Case closed? Not so fast. What about the variable student motivation (Variable Z)?

1. Highly motivated students are more likely to sign up for extra coaching classes (Z causes X).
2. Highly motivated students are also more likely to study hard on their own, leading to better scores regardless of coaching (Z causes Y).

In this scenario, student motivation is a confounder. It’s mixing up your results. You’re seeing the effect of *motivation*, but you’re incorrectly attributing all of it to the *coaching classes*. A good econometric model, guided by theory (in this case, educational psychology), would know that it must find a way to “control for” motivation. If you fail to include the confounder (motivation) in your model, your model will suffer from omitted variable bias, and your conclusions about the effectiveness of coaching classes will be wrong.

Statistical criteria like R-squared won’t warn you about this. They will happily report a “good fit.” Only your theoretical understanding and logical reasoning can help you identify and solve the problem of confounders.

When selection criteria disagree 🔗

So, you’ve started with a strong theoretical foundation. You’ve identified your key variables and potential confounders. Now you have a few different “candidate” models. For example:

• Model 1: Demand = f(Price, Income)
• Model 2: Demand = f(Price, Income, Price of Substitute)
• Model 3: Demand = f(Price, Income, Price of Substitute, Advertising Spend)

This is where statistical criteria are supposed to help. You’re no longer data mining; you’re making reasoned choices between theoretically-plausible models. The most common tools for this are:

• Adjusted R-squared: This is a version of R-squared that penalizes you for adding variables that don’t actually improve the model’s explanatory power. You’d look for the model with the highest Adjusted R-squared.
• Akaike Information Criterion (AIC): This is a very popular measure that balances model fit with model simplicity. It penalizes models for having more parameters (more complexity). You’d look for the model with the lowest AIC.
• Bayesian Information Criterion (BIC or SIC): This is similar to AIC but applies a much heavier penalty for complexity, especially in large datasets. It strongly favors “parsimonious” models (simple ones). You’d look for the model with the lowest BIC.

The inevitable conflict 🔗

Here’s the problem: these criteria will often disagree. You might find that Model 3 has the highest Adjusted R-squared, Model 2 has the lowest AIC, and Model 1 (the simplest) has the lowest BIC. Now what? You have three different “best” models, all according to valid statistical measures.

This is the moment of truth for an econometrician. This is where you earn your paycheck. You cannot just “pick the lowest AIC.” You have to step back and think, using all the information at your disposal. AIC, BIC, and Adjusted R-squared are not answering the same question. As one stats expert put it, R-squared is about how well your model explains your *current* data, while AIC is an estimate of how well your model will predict on *new* data.

BIC, with its heavy penalty, is often preferred when you are searching for the “true” underlying model, assuming one exists. AIC is often preferred when your goal is purely predictive. Adjusted R-squared is a good, intuitive guide but can be less reliable than the information criteria.

How to make the final choice 🔗

When the statistics are ambiguous, your final choice must be a “holistic” one, based on a combination of factors. This is more of an art, guided by science.

1. Go back to theory: Does the “Advertising Spend” variable in Model 3 really make strong economic sense? Is its coefficient sign what you expected? Is it statistically significant? If adding the “Price of Substitute” (Model 2) is fundamental to the theory of demand, then Model 1 is probably too simple, even if it has the best BIC.

2. Look at the data quality: Is your “Advertising Spend” variable a reliable measurement, or is it a rough guess? If the data quality is poor, you might be better off with a simpler model (like Model 2) that relies on more robust data points.

3. Use practical sense (Parsimony): This is the principle of parsimony, or Occam’s Razor. If two models perform almost identically, always choose the simpler one. A simple model is easier to interpret, easier to explain to a non-expert, and less likely to be “overfitted” to your specific dataset. The BIC often helps enforce this principle.

4. Check the diagnostics: Don’t forget to check the model’s “health.” Are the residuals (the errors) well-behaved? Is there evidence of multicollinearity (your independent variables being too related to each other)? A model with a great AIC score might be violating key assumptions of the regression, making it invalid.

Ultimately, statistical criteria are just advisors. You are the judge. They provide evidence, but they don’t make the final ruling. That ruling must come from your deep understanding of the economic principles at play, your practical judgment, and a healthy skepticism of any number that looks “too good to be true.”

What do you think?

Have you ever seen a statistic in the news that made you suspicious? Perhaps a strange correlation that didn’t seem to have a logical explanation? How does this idea of “theory over statistics” change how you might look at data-driven claims in the future?

References

1. https://www.researchgate.net/publication/278828137_The_monetary_policy_in_econometric_models_primacy_of_the_theory_over_the_empirics
2. https://en.wikipedia.org/wiki/Spurious_relationship
3. https://en.wikipedia.org/wiki/Confounding
4. https://www.sthda.com/english/articles/38-regression-model-validation/158-regression-model-accuracy-metrics-r-square-aic-bic-cp-and-more/