Imagine you’re trying to understand what makes a company successful. You collect data on hundreds of firms over 10 years. You track their profits, R&D spending, and advertising budgets. This is panel data: it tracks the same entities (firms) across time. Now, what if you believe that a firm’s profit *this* year is directly influenced by how profitable it was *last* year? This idea of “history mattering” makes your model dynamic. You’ve just entered the powerful, but tricky, world of dynamic panel data models.
These models are incredible for answering questions about “state dependence.” Does being unemployed last month make you more likely to be unemployed this month? Does a state’s high GDP last year give it momentum for growth this year? But as soon as we add this “history” variable (known as the lagged dependent variable), we run into a massive statistical puzzle. Our standard tools break down, and two common “fixes” we try to apply can actually make the problem worse, leading to biased and unreliable answers.
Table of Contents
The two-headed dragon of dynamic panels
When we build a model to explain a firm’s profit, we can measure things like R&D (X) and last year’s profit (y_it-1). But what about the things we *can’t* measure? What about a firm’s “unbeatable company culture” or its “super-savvy, unshakeable CEO”? These are stable, persistent qualities that make a firm unique. In econometrics, we call this unobserved individual heterogeneity, or more simply, a fixed effect (often written as $\mu_i$).
This fixed effect is our first problem. That “savvy CEO” (the $\mu_i$) probably influenced *last year’s* profit (y_it-1) and is *also* influencing *this year’s* profit (y_it). This connection creates a bias. To get a true estimate of how R&D spending affects profit, we *must* get rid of this unobserved fixed effect. So, how do we remove an invisible factor?
The second, related problem is the lagged variable itself. Because this year’s outcome is a function of last year’s, any random “shock” or error (like a sudden supply chain disruption) that happened last year gets bundled up inside y_it-1 and carried forward. This creates a feedback loop where our predictor variable is tangled up with the error terms, a classic case of endogeneity. This violates a core assumption of standard regression (OLS), rendering its results useless.
Economists have two clever tricks to eliminate the fixed effect $\mu_i$: the “Within Transformation” and “First-Differencing.” But as we’ll see, in a dynamic model, these tricks backfire.
Attempt 1: The ‘within transformation’ and its fatal flaw
The Within Transformation is the engine behind the standard Fixed Effects (FE) estimator. The logic is simple: if the fixed effect ($\mu_i$) is “fixed” for a firm over all 10 years, what if we just subtract that firm’s 10-year *average* from its data for every single year?
Imagine Firm A has a “fixed effect” value of 50 (representing its great CEO). Its 10-year average for this fixed effect is… still 50. When we subtract the average (50) from the value (50), we get zero. The fixed effect vanishes! This is called “de-meaning” the data. We’re no longer comparing Firm A to Firm B; we are only looking at how Firm A *deviates from its own average self* over time. This technique is a cornerstone of modern panel data analysis for static models.
So, we de-mean our entire equation:
- This year’s profit (minus its 10-year average)
- Last year’s profit (minus its 10-year average)
- R&D spending (minus its 10-year average)
- The error term (minus its 10-year average)
The fixed effect is gone. Victory? Not quite. We’ve created a new, subtle problem. Look at the new “de-meaned” lagged variable (y_it-1 minus its average). By definition, the 10-year average of y_it-1 includes values from *all* years. Now look at the new “de-meaned” error term. Its 10-year average also includes the error shocks from *all* years.
Nickell’s bias: The ghost in the machine
Here’s the trap: the de-meaned error term for this year (error_it - average_error) is now, by construction, mathematically correlated with the de-meaned lagged profit (y_it-1 - average_y). Why? Because the “average_error” part of the new error term contains last year’s shock (error_it-1). And last year’s profit (y_it-1), which is part of the new lagged variable, was *also* influenced by last year’s shock (error_it-1).
This new, artificial correlation between our transformed predictor and our transformed error term is exactly the kind of endogeneity we were trying to avoid. This specific problem was identified in a famous 1981 paper by Stephen Nickell. It’s now known as Nickell’s Bias.
The most frustrating part? This bias is worst when our time period (T) is short. If we only have 5-10 years of data for each firm (which is very common), the bias is significant and our estimates are inconsistent-meaning, even with a million firms, our answer won’t converge to the *truth*. The Within Transformation, so brilliant in other contexts, fails us completely here.
Attempt 2: ‘First-differencing’ and a different kind of trap
So, de-meaning is out. What’s the other trick? Instead of subtracting the 10-year average, let’s just subtract *last year’s* data from *this year’s* data. This is called First-Differencing.
The logic is just as clean:
- (This year’s profit) – (Last year’s profit)
- (This year’s R&D) – (Last year’s R&D)
What happens to the fixed effect ($\mu_i$)? It was in this year’s data, and it was in last year’s data. So, ($\mu_i – \mu_i$) = 0. Again, the fixed effect vanishes! We are now modeling the *change* in profit as a function of the *change* in R&D and the *change* in last year’s profit. This seems promising.
But again, we must look at our new variables. Let’s focus on the new lagged variable and the new error term.
- New Lagged Variable ($\Delta y_{it-1}$): This is $(y_{it-1} – y_{it-2})$. It’s the *change* in profit from two years ago to last year.
- New Error Term ($\Delta \epsilon_{it}$): This is $(\epsilon_{it} – \epsilon_{it-1})$. It’s the *change* in the random shock from last year to this year.
Do you see the problem? The original variable for last year’s profit, $y_{it-1}$, was partly determined by last year’s random shock, $\epsilon_{it-1}$. This means our New Lagged Variable is correlated with $\epsilon_{it-1}$. And our New Error Term is *also* correlated with $\epsilon_{it-1}$ by its very definition.
Once again, we have created a model where our predictor variable ($\Delta y_{it-1}$) is correlated with our error term ($\Delta \epsilon_{it}$). We’ve successfully eliminated the fixed effect but created a *new* source of endogeneity. Standard OLS regression on this differenced data will *still* produce biased and inconsistent estimates. We’re 0-for-2.
We aren’t stuck, we just need a new tool
This might feel like a hopeless situation. Both of our standard methods for dealing with fixed effects-methods that are taught in every econometrics course-fail when the model is dynamic. They can’t handle the combination of unobserved heterogeneity ($\mu_i$) and state dependence ($y_{it-1}$). As Nickell’s insight powerfully demonstrated, the transformations themselves introduce a new bias that makes our results unreliable.
Trying to use these estimators is like trying to measure a grain of sand with a ruler marked only in feet. The tool is simply not right for the job, and the measurements will be wrong.
This very problem is what spurred the development of more advanced estimators. Economists realized they needed a tool that could *simultaneously* handle fixed effects *and* the endogeneity of the lagged variable. This led to the creation of the Generalized Method of Moments (GMM) estimators, specifically designed for dynamic panels, such as the famous Arellano-Bond estimator. These methods use clever “instruments”-like “deeper” lags of the variables-to sidestep the endogeneity problem. They effectively solve the puzzle that the Within and First-Difference transformations could not.
Understanding *why* these simpler methods fail is the crucial first step to appreciating the power and necessity of the more complex tools that define modern econometric research.
What do you think? When you analyze trends in your own life or work, do you find that “hidden” fixed traits (like personality or ingrained culture) are more powerful, or does the momentum from “what happened last year” seem to be the bigger driver?
Leave a Reply