Measuring Goodness-of-Fit in Logit and Probit Models

Imagine you’re trying to predict a simple “yes” or “no” question. Will a customer click the “Buy” button? Will a borrower default on a loan? Will a student pass an exam? To tackle these binary (yes/no) questions, you’ve likely built a powerful logit or probit model. You’ve fed it data, run the numbers, and out come the results. But here’s the million-dollar question: Is your model any good? In the world of linear regression (predicting a continuous number, like a house price), we have a trusted friend: R-squared. It gives us a neat score from 0% to 100% telling us how much of the variation our model explains. It’s simple and intuitive. But try to use R-squared on your shiny new logit model, and the whole concept falls apart. This post is all about the ‘why’ and what to do instead. We’re diving into the world of Pseudo-R-squared, with a special focus on the most popular one, McFadden’s Pseudo-R-squared.

Why your trusty R-squared fails for logit and probit models 🔗

To understand why our old friend fails us, we need to remember what R-squared (also called the coefficient of determination) actually does. In a standard Ordinary Least Squares (OLS) regression, R-squared measures the proportion of the variance in your dependent variable (say, house_price) that is predictable from your independent variables (square_feet, location, etc.). It works because the output is a continuous number, and the model tries to minimize the sum of squared errors-the literal distance between your model’s prediction and the actual price.

Now, think about a binary choice model, like a logit model predicting if a person will buy a product (1 = Buy, 0 = No Buy). Here are the two big problems:

1. The output is a probability, not the actual value. Your logit model doesn’t predict “0” or “1”. It predicts the *probability* of being a “1”. For example, it might predict a 0.82 (or 82%) chance that a customer will buy. How do you measure the “distance” between a prediction of 0.82 and the actual outcome of 1? It’s not as straightforward.
2. The variance is not constant. The dependent variable is a Bernoulli distribution, and its variance is tied directly to its mean (p*(1-p)). This violates the assumptions of OLS, like homoscedasticity, making the very idea of “proportion of variance explained” conceptually flawed and mathematically unhelpful.

If you force a standard R-squared calculation, you’ll get numbers that are incredibly low and misleading. You’d end up throwing away excellent models simply because you were using the wrong measuring stick.

Welcome to the world of pseudo-R-squared 🔗

Since we can’t use R-squared, economists and statisticians developed alternatives. These are called pseudo-R-squared measures. The name “pseudo” is important: it means “false” or “imitation.” These are not R-squared. They do *not* tell you the “percentage of variance explained.”

So, what do they do? They are metrics designed to *mimic* the nice properties of R-squared:

• They usually range from 0 to 1 (though some, like Cox & Snell’s, can’t actually reach 1).
• A value of 0 means your model’s predictors are useless.
• A higher value suggests a better model fit than a lower value.

There isn’t just one pseudo-R-squared; there are many (Cox & Snell, Nagelkerke, Tjur’s, and more). They all use different formulas and have different interpretations. The key is to understand that these are primarily used to compare different models fit on the *same data*. For example, is Model A (using age and income) a better fit than Model B (using only age)? The pseudo-R-squared can help you decide.

The most popular kid on the block: McFadden’s pseudo-R-squared 🔗

The most common and arguably most intuitive of these measures is McFadden’s Pseudo-R-squared (also called the likelihood-ratio index). Instead of variance, McFadden’s metric is built around a concept called log-likelihood.

What is log-likelihood? 🔗

In simple terms, the log-likelihood (LL) is a number that measures how well your model, with its chosen parameters, “fits” the data. It’s based on the probability of observing the *actual outcomes* (the 0s and 1s in your dataset) given your model’s predictions. A higher log-likelihood value means your model does a better job of predicting the data you actually observed. (Note: Since log-likelihoods are the log of probabilities, which are less than 1, they are always negative. So “higher” means “less negative”-closer to 0.)

The magic formula (explained simply) 🔗

McFadden’s genius was to create a ratio using log-likelihoods that looks just like the logic of traditional R-squared. To calculate it, you need to run two models:

1. The Intercept-Only Model: This is your “dumb” or “null” model. It has no explanatory variables at all. It just predicts the overall sample proportion for everyone. For example, if 30% of your sample bought the product, this model predicts a 0.30 probability of buying for every single person. We get the log-likelihood from this model: LL(Intercept).
2. The Full Model: This is *your* model, the one you’re testing. It includes all your insightful variables (age, income, education, etc.). We get the log-likelihood from this model: LL(Full).

The formula for McFadden’s Pseudo-R-squared is:

R²_McF = 1 – ( LL(Full) / LL(Intercept) )

Think about this. LL(Full) will always be higher (less negative) than LL(Intercept) because your variables add *some* predictive power. This makes the fraction LL(Full) / LL(Intercept) a number between 0 and 1. By subtracting it from 1, McFadden created a measure of *improvement*. It tells you how much better your model is than a model with no predictive variables at all, based on this log-likelihood metric.

How to interpret McFadden’s R-squared 🔗

This is the most critical part, and the biggest trap for newcomers. You cannot interpret this value the same way you interpret an OLS R-squared. An OLS R-squared of 0.30 (30%) is often considered poor. A McFadden’s R-squared of 0.30 is fantastic!

Here’s a general rule of thumb, based on McFadden’s own comments:

• Values between 0.2 and 0.4 are considered a very good fit.
• Values above 0.4 are considered an excellent fit.

Why are the values so much lower? Because predicting human choice (a 0 or a 1) is inherently more difficult and “noisier” than predicting a continuous value like a house price. You are trying to predict a stark binary outcome, and there will always be a high degree of randomness that no model can capture. Therefore, we have to adjust our expectations. A model that improves our predictive power by 20-40% over the baseline is doing a very good job.

So, what’s the bottom line on “goodness-of-fit”? 🔗

No single pseudo-R-squared is a perfect, all-encompassing measure of model fit. McFadden’s is an excellent tool, but it shouldn’t be the *only* tool you use. The best practice is to look at a combination of metrics to get a holistic view of your model’s performance.

Always supplement your pseudo-R-squared with other measures, such as:

• A Classification Table (or Confusion Matrix): At a chosen probability threshold (like 0.5), how many “Buys” did your model correctly predict? How many “No Buys”? What is its overall percentage accuracy?
• Statistical Significance of Variables: Look at the p-values (or z-scores) for your individual explanatory variables. Does your model contain variables that are actually statistically significant? A good model is built on a foundation of meaningful predictors.
• The Likelihood Ratio (LR) Test: This is a formal statistical test that checks if your LL(Full) is *statistically significantly* better than LL(Intercept). Most software reports this, and it’s a good check to ensure your model as a whole is better than just guessing the average.
• Real-World Intuition: Do the results make sense? Do the coefficients have the signs you would expect (e.g., does higher income *increase* the probability of buying a luxury good)? A model that is statistically “good” but makes no economic sense is a dangerous one.

Ultimately, pseudo-R-squared measures are valuable additions to your econometric toolkit, but they are not a silver bullet. Use them wisely, understand their limitations, and never forget to pair their statistical output with sound economic theory and common sense.

What do you think? Have you ever been misled by comparing a pseudo-R-squared to a traditional R-squared? What’s your preferred way to check if your logit or probit model is actually doing a good job?

References

1. https://en.wikipedia.org/wiki/Logistic_regression
2. https://www.graphpad.com/guides/prism/latest/curve-fitting/reg_mult_logistic_gof_pseudo_r_squared.htm
3. https://stats.oarc.ucla.edu/other/mult-pkg/faq/general/faq-what-are-pseudo-r-squareds/
4. https://stats.stackexchange.com/questions/82105/mcfaddens-pseudo-r2-interpretation
5. https://en.wikipedia.org/wiki/Pseudo-R-squared