A Guide to Model Selection Criteria: R-squared, AIC, SIC, and Mallows’ Cp

Imagine you’re trying to build the “perfect” economic model. You have a pile of data-unemployment rates, inflation, GDP, consumer spending-and you want to understand how they all fit together. It’s tempting to throw every single piece of data into your model, thinking “more is better,” right? But in econometrics, more is often just… more. A model that’s too complex, stuffed with irrelevant variables, is just as useless as a model that’s too simple. This is the art of model selection: finding the “Goldilocks” model that is just right. We need a model that explains the data well without being unnecessarily complicated. Luckily, we have a toolkit of statistical criteria designed to do just that, helping us compare models and pick the winner.

The classic: R-squared and its smarter cousin, adjusted R-squared 🔗

When you first learn about regression, R-squared is often the first “grade” you’re taught to look at. It’s the most common measure of “goodness-of-fit.”

What is R-squared (R²)? 🔗

At its simplest, R-squared (or the Coefficient of Determination) tells you the percentage of the variation in your dependent variable (the thing you’re trying to predict) that is “explained” by the independent variables (the things you’re using to predict). It’s a score from 0 to 1 (or 0% to 100%).

Let’s say you’re modeling housing prices. An R-squared of 0.75 would mean that 75% of the movement in housing prices can be explained by your model’s inputs (like square footage, number of bedrooms, and location). That sounds pretty good! It seems intuitive: just pick the model with the highest R-squared.

But there’s a huge, hidden trap. R-squared has a critical flaw: it always increases (or at best stays the same) every time you add a new variable to the model. It doesn’t matter if the variable is completely useless. If you add “average number of rainy days” or “local high school’s mascot” to your housing price model, your R-squared will go up a tiny bit, purely by chance. If you followed this metric alone, you’d end up with a monstrous model full of “junk” variables. This is called overfitting.

Why adjusted R-squared is a fairer judge 🔗

This is where the Adjusted R-squared comes in to save the day. It’s a modified version of R-squared that understands that adding more variables isn’t free. Adjusted R-squared introduces a penalty for every new variable you add.

Think of it this way:

• R-squared is like a boss who gives you a raise just for showing up, even if you don’t do any work.
• Adjusted R-squared is a smart boss who only gives you a raise if your new contribution *significantly* improves the company’s performance, more than what would be expected by just having another person in the room.

The adjusted R-squared will only increase if the new variable you add is a powerful predictor, powerful enough to overcome the “penalty” for its inclusion. If you add a useless variable, the penalty will outweigh the tiny, by-chance increase in R-squared, and your *adjusted* R-squared will actually go down. This makes it a much more honest judge. When comparing models that have the same dependent variable, the model with the highest adjusted R-squared is generally the better and more parsimonious (simpler) choice.

The information criteria: AIC and SIC 🔗

Adjusted R-squared is a great tool, but its main job is to assess in-sample “goodness-of-fit.” What if our goal is different? What if we want the model that is most likely to be “true” or the one that will give us the best *forecasts* out-of-sample? For this, we turn to “information criteria.”

These criteria are based on a concept from information theory and maximum likelihood estimation. They don’t give you a simple percentage of “fit.” Instead, they provide a single score that balances two things: the model’s error (how badly it fits the data) and its complexity (how many variables it uses). For both of these, the rule is simple: lower is better.

The Akaike information criterion (AIC) 🔗

Developed by Japanese statistician Hirotugu Akaike, the Akaike Information Criterion (AIC) is one of the most popular model selection tools. The AIC’s score is calculated from the model’s maximum likelihood (a measure of fit) and a penalty term based on the number of parameters (k).

The penalty for AIC is 2k. AIC estimates the quality of each model relative to the other models you’re testing. It doesn’t tell you if a model is “good” in an absolute sense, only which one is the *best* in the group. By finding the model with the minimum AIC, you are finding the one that best balances fit against complexity. It’s particularly valued for selecting models in time-series analysis and forecasting.

The Schwarz information criterion (SIC) 🔗

The Schwarz Information Criterion (SIC), also known as the Bayesian Information Criterion (BIC), is a very close relative of the AIC. It was derived by Gideon Schwarz and, like AIC, it aims to find the model with the lowest score.

It also uses a penalty for model complexity, but the penalty is different. Instead of 2k, the SIC’s penalty is k * ln(n), where ‘n’ is the number of observations in your data. This is a crucial difference. Because the natural log of ‘n’ (ln(n)) is almost always larger than 2 (as long as you have more than 8 observations), the SIC imposes a much stronger penalty for adding new variables than the AIC does, especially with large datasets.

This means that SIC tends to favor simpler models (fewer variables) than AIC.

• If your goal is finding the best model for forecasting (predictive accuracy), AIC is often preferred.
• If your goal is finding the model that best represents the “true” underlying data-generating process (explanatory power), SIC is often preferred because it is “consistent” and does a better job of identifying the true, simple model.

A major advantage of both AIC and SIC is that they can be used to compare non-nested models-models that use completely different sets of independent variables-which adjusted R-squared cannot do.

Mallows’ Cp: The balancing act of precision and bias 🔗

Our final tool, Mallows’ Cp, is another popular criterion used, especially in the context of linear regression, to help select the best subset of variables. It’s a fantastic tool because it explicitly tackles the trade-off between a model’s *precision* (low variance) and its *bias* (being systematically wrong).

Understanding the Cp statistic 🔗

Mallows’ Cp compares your “subset” model (with, say, 4 variables) to the “full” model (which includes *all* possible variables you’re considering). It tries to find a model that has a low total error by balancing these two components.

• A model with too few variables (an “under-fitted” model) is biased. It’s missing important information.
• A model with too many variables (an “over-fitted” model) has low bias but high variance. It’s fitting to the “noise” in the data, not just the “signal.”

Mallows’ Cp helps you find the sweet spot. The statistic is calculated by comparing the error of your subset model to an estimate of the error from the full, “true” model.

The golden rule for reading Mallows’ Cp 🔗

The best part of Mallows’ Cp is its simple, elegant rule of thumb. When you’re evaluating models, you should look for a model where the Cp value is:

1. Small (indicating low error)
2. Approximately equal to ‘p’, where ‘p’ is the number of parameters in the model (including the intercept).

A small Cp value indicates that the model is relatively precise. A Cp value that is close to ‘p’ indicates that the model is relatively *unbiased*. For example, if you are testing a model with 3 independent variables (plus 1 intercept, so p=4), you are looking for a model that has a Cp value close to 4.

If a model (with p=4) has a Cp of 50, this is a huge red flag. The Cp is much larger than ‘p’, which tells you the model is suffering from significant bias-it’s very likely missing one or more crucial variables. Your goal is to find the simplest model that has a Cp value close to its ‘p’.

So, which one do I use? 🔗

This is the most common question, and the answer is: “It depends, and probably all of them.” There is no single “best” criterion. They all measure slightly different things and have different strengths.

• Use Adjusted R-squared to get a quick, intuitive sense of in-sample explanatory power and to penalize simple model complexity.
• Use AIC and SIC when comparing a wide range of models (nested or non-nested) and when your goal is predictive power (AIC) or explanatory simplicity (SIC).
• Use Mallows’ Cp when you’re using subset selection in regression and want a clear signal about model bias.

The best approach is to use them together. If you find a model that has the highest Adjusted R-squared, the lowest AIC/SIC, and a Mallows’ Cp close to ‘p’, you can be very confident that you’ve found a robust, reliable, and parsimonious model. You’ve successfully moved from just “fitting” data to truly “selecting” a model.

What do you think? When you’re building a model, do you find yourself prioritizing goodness-of-fit (like R-squared), or are you more concerned with simplicity and predictive power (like AIC/SIC)? Have you ever been “tricked” by a model that had a high R-squared but turned out to be overfitted?

References

1. https://www.investopedia.com/ask/answers/012615/whats-difference-between-rsquared-and-adjusted-rsquared.asp
2. https://www.scribbr.com/statistics/akaike-information-criterion/
3. https://support.minitab.com/en-us/minitab/help-and-how-to/statistical-modeling/regression/supporting-topics/goodness-of-fit-statistics/what-is-mallows-cp/