Testing Joint Significance in Logit and Probit Models

If you’ve ever run a standard linear regression (like an Ordinary Least Squares, or OLS, model), you’re probably familiar with the F-test. It’s that all-important number at the bottom of your output that answers one big question: “Does my model, as a whole, actually explain anything?” It tests the joint significance of all your variables at once. If the F-test fails, your model is essentially useless, regardless of what individual t-tests might say.

But what happens when your outcome isn’t a continuous number like ‘price’ or ‘income’? What if you’re trying to predict a ‘yes’ or ‘no’ answer? For example:

• Will a customer click this ad? (Yes/No)
• Will this borrower default on their loan? (Default/Repay)

– Will this student graduate? (Yes/No)

For these questions, we leave OLS behind and enter the world of qualitative response models, with the two most famous being the Logit and Probit models. These models are incredibly powerful, but they work differently under the hood. They don’t use the ‘least squares’ method; they use a concept called ‘maximum likelihood’. And because the math is different, they need a different test for joint significance. The F-test’s cousin for the Logit and Probit world is the Likelihood Ratio (LR) test.

First, what does ‘joint significance’ even mean? 🔗

Before we dive into the ‘how’, let’s be crystal clear on the ‘what’. When we test for joint significance, we are testing a very specific hypothesis. Let’s imagine we build a model to predict if someone will buy a new car (1=Yes, 0=No) based on their income, age, and current car’s age.

Our model looks something like this:

Prob(Buy Car = 1) = Some_Function(β₀ + β₁*Income + β₂*Age + β₃*Car_Age)

The null hypothesis (H₀) for the joint significance test is that *all* our explanatory variables have *zero* effect on the outcome. In math terms:

H₀: β₁ = β₂ = β₃ = 0

The alternative hypothesis (H₁) is that *at least one* of these coefficients is *not* zero.

H₁: At least one β ≠ 0

In simple English, the null hypothesis says our model is worthless. It claims that income, age, and car age, taken together as a team, have absolutely no power to predict who will buy a car. The LR test is our way of challenging this claim.

The core idea: Comparing two ‘stories’ 🔗

The Likelihood Ratio test is a beautifully logical concept. Its main idea is to compare two different ‘stories’ (or models) and see which one does a better job of explaining the data we actually observed.

The two models we build are:

1. The Unrestricted Model (or Full Model): This is your complete model with all the explanatory variables you think are important (income, age, car age, etc.). It is ‘unrestricted’ because you are *allowing* the data to estimate a coefficient for every single variable.
2. The Restricted Model (or Null Model): This is a very simple model that is ‘restricted’ by the null hypothesis. Since our null hypothesis is that all coefficients are zero, this model *only includes an intercept (β₀)*. It’s the most basic model possible, essentially just predicting the average probability of buying a car for everyone, with no custom information.

The logic is simple: If your variables (income, age, etc.) are genuinely useful, your Unrestricted Model should fit the data *significantly* better than the boring, one-size-fits-all Restricted Model. The LR test is what measures this “significantly better” part.

A quick detour: What is ‘likelihood’? 🔗

This is the one part that can feel a bit abstract. We need to know what “fit” means. In OLS, ‘fit’ is measured by minimizing the sum of squared errors. In Logit and Probit models, ‘fit’ is measured by maximizing the log-likelihood function (LLF).

Think of it this way: The likelihood function asks, “Given the data we *actually* saw (e.g., ‘Person 1 did not buy’, ‘Person 2 did buy’), how ‘likely’ is it that we would observe this exact dataset if our model’s coefficients (our β’s) were the ‘true’ ones?”

The estimation process (Maximum Likelihood Estimation, or MLE) is a computer algorithm that tries out millions of different combinations of β’s until it finds the set that results in the highest possible ‘likelihood’ of having produced your data. The computer then reports this single, best-fit number as the log-likelihood (LL). This number is always negative, so a ‘higher’ value is one that is *closer to zero* (e.g., -150 is much better than -200).

So, when we compare our two models, we are just comparing their two final log-likelihood values:

• LL_UR: The log-likelihood of the full, unrestricted model.
• LL_R: The log-likelihood of the simple, restricted (intercept-only) model.

Because the full model has more variables, it will *always* fit at least as well as the restricted model, so we know that LL_UR ≥ LL_R. The real question is: is it *a lot* better, or just a tiny bit better by random chance?

Calculating and interpreting the LR statistic 🔗

This is where the magic happens. We can now build our test statistic, which is surprisingly simple. The Likelihood Ratio (LR) test statistic is calculated as:

LR = 2 * (LL_UR – LL_R)

Let’s break that down. `(LL<sub>UR</sub> – LL<sub>R</sub>)` is the difference in the log-likelihoods. Since LL_UR is a larger number (less negative) than LL_R, this difference will be positive. We multiply it by 2 for statistical reasons, and this resulting LR statistic conveniently follows a well-known statistical distribution: the Chi-squared (χ²) distribution.

This is the key! Just like the F-statistic follows an F-distribution, the LR statistic follows a Chi-squared distribution. This allows us to find a p-value and make a decision.

H3: Degrees of freedom 🔗

To find our p-value, we need one more piece of information: the degrees of freedom (df). For the LR test of joint significance, the degrees of freedom are simply the *number of restrictions* we imposed. In our car example, our null hypothesis was H₀: β₁ = β₂ = β₃ = 0. We set 3 parameters to zero, so our degrees of freedom are 3.

In short: df = number of explanatory variables in your full model.

H3: Making the decision 🔗

Now we put it all together. Here’s the step-by-step process:

1. Run the Unrestricted Model: Run your Logit or Probit with all your variables (Income, Age, Car_Age). Your software (like Stata, R, or Python) will give you a final log-likelihood. Let’s say: LL_UR = -150.2.
2. Run the Restricted Model: Run the same model with *only* an intercept (no explanatory variables). Your software will give you another log-likelihood. Let’s say: LL_R = -198.7.
3. Calculate the LR Statistic:

   LR = 2 * (LL_UR – LL_R)

   LR = 2 * ((-150.2) – (-198.7))

   LR = 2 * (48.5) = 97.0

4. Find the p-value: We look up our LR statistic (97.0) on a Chi-squared distribution with 3 degrees of freedom.

Almost all statistical software does this for you automatically. You’ll see a line in your output that looks something like this:

LR chi2(3) = 97.00
Prob > chi2 = 0.0000

Interpretation:

• LR chi2(3) = 97.00: This is our LR statistic (97.0) and our degrees of freedom (3).
• Prob > chi2 = 0.0000: This is our p-value.

A p-value of 0.0000 is far, far less than the standard significance levels (like 0.05 or 0.01). This tiny p-value means it is *extremely unlikely* that we would see such a large improvement in fit (from -198.7 to -150.2) if our variables were truly useless (i.e., if the null hypothesis were true).

Therefore, we reject the null hypothesis (H₀). We conclude that our variables, taken together, are jointly significant. Our model, as a whole, has real explanatory power.

Why not just look at the individual p-values (z-tests)? 🔗

This is a common and excellent question. When you run your full model, your software also gives you a z-statistic (or t-statistic) and a p-value for *each individual variable*. Why do we need an *overall* test?

The individual z-tests check a different hypothesis. For Income, the z-test checks H₀: β₁ = 0, *holding Age and Car_Age constant*. It tests the *unique* contribution of that one variable.

It’s possible (especially with high multicollinearity) to have a model where no single variable is significant on its own, but they are powerful *as a group*. Think of a football team’s offensive line. If you test the individual significance of just the left guard, you might find he has no significant effect on winning. Same for the center. Same for the right tackle. But if you test them *jointly* (H₀: all offensive linemen coefficients = 0), you would find they are *highly* significant. You can’t just remove them all!

The LR test for joint significance is your “team” test. It’s the very first thing you should look at. If this test fails (i.e., you get a high p-value), your model as a whole is not useful, and you shouldn’t even bother interpreting the individual p-values. It’s the gatekeeper for model validity.

What about the Wald and Lagrange Multiplier (LM) tests? 🔗

You may also see other tests in your software output, most commonly the Wald test and the Lagrange Multiplier (LM) test (also called the score test). In econometrics, these are often called the “Holy Trinity” of hypothesis tests.

Here’s the simple version:

• Likelihood Ratio (LR) Test: Compares the log-likelihood of *both* the full and restricted models. It requires estimating two models.
• Wald Test: Only requires estimating the *full, unrestricted model*. It then checks how far the estimated coefficients are from zero. Most software reports individual p-values using z-tests, which are a form of Wald test.
• Lagrange Multiplier (LM) Test: Only requires estimating the *simple, restricted model*. It then checks if allowing the coefficients to be non-zero would cause a significant improvement.

All three tests are designed to answer the exact same question (testing the same null hypothesis) and are asymptotically equivalent, meaning that with a large enough sample size, they will all give you the same conclusion. In smaller samples, they might differ slightly. The LR test is often preferred for its strong theoretical properties, but the Wald test is very common because it’s what’s used for the individual coefficient tests you see on every output.

For the purpose of *joint significance*, the “LR chi2” statistic reported at the top of your Logit or Probit output is the most direct and reliable tool for the job. It is the F-test’s true equivalent, built on the solid foundation of comparing the likelihood of two nested models.

What do you think? When you’re building a model, do you find yourself focusing more on the overall model significance (like the LR test) or on the significance of individual variables? Have you ever had a model that was jointly significant, but none of the individual predictors were?

References

1. https://en.wikipedia.org/wiki/Likelihood-ratio_test
2. https://www.scribbr.com/statistics/chi-square-tests/
3. https://stats.oarc.ucla.edu/stata/output/logistic-regression-analysis/
4. https://www.rand.org/content/dam/rand/pubs/papers/2008/P6756.pdf