Goodness of Fit: Mastering R-Squared in Regression

Imagine you run a small bakery. You’ve started tracking your daily advertising spend, wondering if it has any real effect on the number of cookies you sell. You plot your data: ad spend on the horizontal axis (X) and cookies sold on the vertical axis (Y). You draw a line through the data that seems to represent the trend. But how good is that line? How much of your cookie sales “boom” is *really* due to your ads, and how much is just… random, happy customers?

This is the central question of “goodness of fit.” In econometrics and statistics, we don’t just build models; we need to measure how well they actually explain the world. The most famous tool for this job is the Coefficient of Determination, much better known by its nickname: R-Squared ($r^2$). It’s a single number, a score from 0 to 1, that tells you how much “sense” your model makes. This post is all about understanding what that number *really* means.

So, what exactly is R-Squared? 🔗

At its core, R-Squared measures the proportion of the total variation in the dependent variable (your ‘Y’, e.g., cookie sales) that is explained by the independent variable (your ‘X’, e.g., ad spend) in your model.

Let’s translate that. Your cookie sales aren’t the same every day. They vary. They go up and down. This fluctuation is the “total variation.” What R-Squared tells you is what percentage of that fluctuation can be predictably linked to the changes in your ad spend.

If your $r^2$ is 0.70, it means 70% of the daily changes in cookie sales can be explained by your ad budget. The other 30% is unexplained-it’s the “residual” or “noise” caused by other things: the weather, a local holiday, a bad flour batch, or just random chance.

A high $r^2$ suggests your X is a good predictor for Y. A low $r^2$ suggests your X doesn’t have much to do with Y. But to truly grasp this, we need to look under the hood. We need to dissect “total variation” into its component parts.

To understand R-Squared, we must first understand variation 🔗

The “variation” we keep mentioning is simply the differences between data points. Not all days are average. The core idea of regression is to see if the ups and downs in Y (sales) move in time with the ups and downs in X (ads).

Before we even look at ‘X’ (ads), our *simplest* guess for tomorrow’s cookie sales (Y) would just be the average sales over the past year. This average is our baseline, our “dumb” model. The total variation is, essentially, a measure of how wrong that “dumb” model is.

This variation is broken into three key parts: TSS, ESS, and RSS.

Step 1: Total Sum of Squares (TSS) – The total problem 🔗

The Total Sum of Squares (TSS) represents the total variation in your dependent variable (Y). It’s the starting point. It captures the entire “problem” we’re trying to solve.

How it’s calculated: 1. You calculate the average value for Y (e.g., average cookie sales, let’s say 200). 2. For *every* day, you measure the difference (deviation) between the *actual* sales for that day (say, 250) and the average (200). So, $250 – 200 = 50$. 3. You square that difference ($50^2 = 2500$) to avoid negative numbers canceling out positive ones. 4. You do this for all your data points and add them all up.

This big number, TSS, is your total “data puzzle.” It’s a measure of how much your sales jump around the average. If all your sales were exactly 200 every day, your TSS would be 0.

Step 2: Residual Sum of Squares (RSS) – The unexplained part 🔗

Now, we introduce your regression model-the line you drew. This model makes a *prediction* ($\hat{Y}$) for every X value. But it’s not perfect. The Residual Sum of Squares (RSS) measures the part of the variation your model *failed* to explain. It is the sum of the “errors” or “residuals.”

How it’s calculated: 1. For *every* day, you look at the *actual* sales (e.g., 250 cookies). 2. Then you look at what your ad-spend model *predicted* for that day (e.g., 230 cookies). 3. The difference ($250 – 230 = 20$) is the “residual” or error. Your model was off by 20 cookies. 4. You square that error ($20^2 = 400$). 5. You add up these squared errors for all your data points.

This number, RSS, is the ‘unexplained’ part. It’s the variation that remains a mystery, the ‘noise’ your ad spend variable couldn’t account for. In a perfect model, RSS would be 0.

Step 3: Explained Sum of Squares (ESS) – The model’s victory 🔗

If TSS is the total puzzle and RSS is the part you couldn’t solve, what’s left over must be the Explained Sum of Squares (ESS). This is the portion of the total variation that your model *successfully* captured. It’s your model’s “victory.”

How it’s calculated: 1. For *every* day, you look at what your model *predicted* (e.g., 230 cookies). 2. You measure how far that prediction is from the simple *average* (e.g., 200 cookies). 3. The difference ($230 – 200 = 30$) represents the “improvement” your model made over just guessing the average. 4. You square that difference ($30^2 = 900$). 5. You add up these squared improvements for all your data points.

This number, ESS, represents the “signal” your model found in the noise. It’s the amount of variation in cookie sales that moves in a predictable way with your ad spend.

The big reveal: The decomposition (TSS = ESS + RSS) 🔗

This brings us to the most beautiful and simple identity in basic regression:

Total Variation (TSS) = Explained Variation (ESS) + Unexplained Variation (RSS)

Think of it like this:

• TSS: The total amount of “movement” in your cookie sales.
• ESS: The amount of that movement you can attribute to your ad spend.
• RSS: The amount of that movement that seems to be random (or caused by factors you aren’t measuring).

This simple breakdown is the key to everything. It allows us to finally, and clearly, calculate R-Squared.

Calculating and interpreting R-Squared 🔗

Given the equation above, R-Squared is just the ratio of the “good” part (ESS) to the “total” part (TSS).

The first formula: $r^2 = \text{ESS} / \text{TSS}$ 🔗

This is the most intuitive definition. $R^2$ is simply the (Explained Variation) / (Total Variation). If your TSS (total puzzle) is 1,000,000 and your model’s ESS (explained part) is 700,000, then: $r^2 = 700,000 / 1,000,000 = 0.70$

Your model explains 70% of the total variation.

The second formula: $r^2 = 1 – (\text{RSS} / \text{TSS})$ 🔗

This is the other common way to calculate it, and it gives the exact same result. It defines $R^2$ as 1 – (Unexplained Variation / Total Variation). Using the same numbers, your RSS (unexplained part) would be $\text{TSS} – \text{ESS}$, or $1,000,000 – 700,000 = 300,000$. $r^2 = 1 – (300,000 / 1,000,000)$ $r^2 = 1 – 0.30 = 0.70$

This version is useful because it shows that $R^2$ gets *bigger* as your error (RSS) gets *smaller*.

How to interpret the value (from 0 to 1) 🔗

The value of $R^2$ is always between 0 and 1 (or 0% and 100%), which makes it a wonderfully clear “score.”

• An $r^2$ of 1 (or 100%) means a “perfect fit.” This means $\text{RSS} = 0$. All variation is explained. Every single data point falls *exactly* on your regression line. In the real world, especially in economics, this is a giant red flag. It usually means you’ve made a mistake, like modeling a variable against itself (e.g., “predicting” sales using… sales).
• An $r^2$ of 0 (or 0%) means “no fit.” This means $\text{ESS} = 0$. Your model explains *nothing*. Your X variable (ad spend) has zero linear relationship with your Y variable (cookie sales). Knowing the ad spend gives you no help *at all* in predicting sales.
• An $r^2$ of 0.65 (or 65%) is a typical “real world” result. It means 65% of the variance in Y is explained by X. The other 35% is the residual, which could be driven by location, competition, weather, or other variables not yet in your model.

The big “so what?”: What is a *good* R-Squared? 🔗

This is the most common question, and the answer is frustratingly: “it depends.”

A “good” $r^2$ is entirely context-dependent.

• In physics or engineering, you might be modeling a precise physical law. You would expect an $r^2$ of 0.95 or higher. If you get 0.70, something is very wrong with your experiment.
• In the social sciences (like economics), you are modeling human behavior, which is incredibly “noisy.” An $r^2$ of 0.50 (50%) predicting individual consumer spending might be considered very strong. When modeling complex systems like national GDP growth, an $r^2$ of 0.30 (30%) might be valuable if it isolates the effect of one key policy.

A low $r^2$ doesn’t mean your model is useless. It just means your independent variable has low predictive power *by itself*. But if that variable is the only one you can *control* (like your ad budget), and the model shows it has a statistically significant positive effect, it’s still incredibly useful for making decisions.

The dark side: When R-Squared lies to you 🔗

R-Squared is a vital tool, but it’s a “dumb” one. It can be easily misused and misinterpreted. You must be aware of its critical limitations.

Limitation 1: R-Squared will always increase when you add more variables 🔗

This is the most dangerous trap. In a simple model, $r^2$ is fine. But what if you create a “multiple regression” model and add more X variables?
$\text{Sales} = \text{Ad Spend} + \text{Location} + \text{Weather} + \text{Price} + …$

$R^2$ has a fatal flaw: it *never* goes down when you add a new variable. Even if you add a completely useless variable (like “the number of blue cars that passed by”), it will add *some* tiny, random bit of explanatory power, and $r^2$ will nudge upward. This encourages “overfitting”-building a complex model that’s just memorizing noise, not finding a real trend.

The Solution: Always use Adjusted R-Squared when comparing models with different numbers of variables. Adjusted $R^2$ “punishes” you for adding variables that don’t add *significant* explanatory power. It’s a much more honest and reliable score for complex models.

Limitation 2: R-Squared says nothing about causation 🔗

A high $r^2$ does not prove that X *causes* Y. It only shows they are *correlated*. The classic example: Ice cream sales and crime rates have a very high $r^2$. They move together. Does this mean eating ice cream causes crime? No. A third “lurking” variable-hot weather-causes both.

Your $r^2$ for ad spend and cookie sales might be high, but it doesn’t *prove* ads *cause* sales. It’s a strong piece of evidence, but not the final word.

Limitation 3: R-Squared says nothing about statistical significance 🔗

You can have a high $r^2$ from a model built on pure chance (especially with a small dataset). $R^2$ doesn’t tell you if your results are “real” or a random fluke. For that, you must look at the p-values of your variables. A p-value tells you the probability that the relationship you’re seeing is just a coincidence. A good model needs *both* a reasonable $r^2$ *and* statistically significant p-values.

Limitation 4: R-Squared can’t see the pattern 🔗

The most famous warning about $r^2$ is Anscombe’s Quartet. This is a set of four different datasets that have the *exact same* $r^2$, the *exact same* average X, the *exact same* average Y, and the *exact same* regression line.

But when you plot them, one is a nice line, one is a clear curve (meaning your linear model is wrong), and the other two are distorted by single, powerful outliers. This proves that you must always plot your data. Never, ever trust $R^2$ alone.

R-Squared is the beginning of your analysis, not the end. It’s a fantastic “at a glance” summary of your model’s predictive power, but it’s only one number in a much larger detective story.

What do you think?

Have you ever been fooled by a high R-Squared in a business report or a news article? In your own work, what “unexplained” factors (the RSS) do you find are the most powerful and the hardest to measure?

References

1. https://online.stat.psu.edu/stat462/node/104/
2. https://www.investopedia.com/terms/r/r-squared.asp
3. https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/what-is-r-squared/