Common Data Analysis Misconceptions: Correlation, Simpson’s Paradox, and Dredging

We live in a world obsessed with data. From business forecasts to health advice, we’re told to “trust the numbers.” But what happens when the numbers lie? Or more accurately, what happens when our intuition misinterprets what the numbers are actually saying? Data analysis is an incredibly powerful tool, but it’s also a minefield of cognitive traps and statistical illusions. We see a pattern and our brain immediately wants to build a story, to find a simple cause for a complex effect. Unfortunately, this natural impulse can lead us wildly astray.

Understanding the common fallacies in data analysis isn’t just for data scientists; it’s a critical thinking skill for everyone. It helps us become better consumers of news, more skeptical employees, and more informed citizens. We’re going to explore three of the most common and misleading data misconceptions: the classic confusion between correlation and causation, the mind-bending Simpson’s Paradox, and the sneaky practice of data dredging. These aren’t just academic curiosities; they shape decisions in medicine, policy, and business every single day.

The classic trap: correlation does not imply causation 🔗

This is the most famous warning in all of statistics, and for good reason. It’s a phrase you’ve almost certainly heard before, but its implications run deep. A correlation is simply a statistical relationship between two variables. When one changes, the other tends to change, too. Causation, on the other hand, means that a change in one variable *directly causes* a change in another. Our brains are wired to see correlation and immediately assume causation, but the world is rarely that simple.

Let’s take the example from the user’s prompt: a strong correlation between student attendance and final marks. It’s easy to look at this and conclude, “Aha! Coming to class *causes* high scores. The solution to better grades is just to enforce attendance.” But does it? While attending class is certainly helpful, this conclusion jumps the gun. It ignores a potential third variable, sometimes called a lurking variable or confounding factor.

In this case, the lurking variable might be ‘student motivation’ or ‘study hours’. A highly motivated student is likely to do two things: attend all their classes *and* spend a lot of time studying. An unmotivated student will likely skip classes *and* not study. In this scenario, the high marks are not caused by attendance, but by the study hours. The attendance and marks just move together because they are *both* results of that third factor, motivation. If you forced an unmotivated student to attend class, but they still didn’t study, their marks would likely not improve.

When ice cream causes crime 🔗

A famous, more clear-cut example is the strong positive correlation between ice cream sales and crime rates. As ice cream sales rise, so does crime. If we mistook this correlation for causation, we’d have to conclude that eating ice cream makes people become criminals, or perhaps that robbing people gives them a craving for a sweet treat. Both are absurd.

The obvious lurking variable here is ‘warm weather’ (or the season, ‘summer’). When it’s hot, more people are outside, creating more opportunities for social interaction and, unfortunately, conflict. More people are also buying ice cream. The weather is the true causal factor influencing both variables independently. This is a perfect example of a spurious correlation-a relationship that appears real but is just a coincidence or the result of a third, unseen factor.

There are other reasons for this fallacy, too. Sometimes the causation is reversed. For example, a study might find that people who use wheelchairs have a higher-than-average incidence of injuries. Does this mean wheelchairs are dangerous and cause injuries? No, it’s the other way around: a pre-existing injury *causes* the person to need a wheelchair. Always question the direction of the arrow. Is A causing B, is B causing A, or is a hidden C causing both?

Simpson’s Paradox: when combining data tells a lie 🔗

This one is perhaps the most fascinating and counter-intuitive of all data traps. Simpson’s Paradox is a statistical phenomenon where a trend appears in several different groups of data but disappears or, more shockingly, *reverses* when those groups are combined. It’s a powerful warning that averages can be dangerously misleading and that grouping data incorrectly can hide the truth.

The most famous real-world example comes from a 1973 lawsuit against the University of California, Berkeley. The university was worried they might be sued for gender bias in graduate admissions. When they looked at the overall numbers, it seemed they had a serious problem. Across the entire university, men had a noticeably higher admission rate than women. It looked like clear evidence of discrimination.

But when a statistician dug deeper, he found something astonishing. He looked at the data department by department. And what he found was that, within most individual departments, the admission rate for women was *equal to or even slightly higher* than for men. So how could this be? How could women be doing better in most departments, but worse overall?

The lurking variable strikes again 🔗

The answer, once again, was a lurking variable. In this case, the lurking variable was ‘department choice’ or ‘department difficulty’. It turned out that women tended to apply to the most competitive departments, like English, which had very low admission rates for everyone (both men and women). Men, on the other hand, tended to apply in greater numbers to less competitive departments, like engineering, which had much higher admission rates for everyone.

Here’s a simplified breakdown:

• Competitive Departments (e.g., English): High number of female applicants, very low acceptance rate for *both* genders.
• Less Competitive Departments (e.g., Engineering): High number of male applicants, high acceptance rate for *both* genders.

When the data was combined, it *looked* like women were being rejected at a higher rate. But this “effect” was purely an illusion. The ‘discrimination’ wasn’t happening at the admissions level; it was a structural issue reflecting the application patterns of men and women at the time. The overall average was completely misleading because it was mixing apples (hard departments) and oranges (easy departments). Without accounting for the lurking variable of department choice, the university would have drawn the exact wrong conclusion.

This paradox is a critical lesson: before you trust an “overall” number, always ask if there are subgroups within the data that might be telling a different story. What you find might be the complete opposite of what the aggregate data suggests. As the Stanford Encyclopedia of Philosophy notes, the paradox highlights that the statistical relationships we find are highly dependent on which variables we choose to include and “control for” in our analysis.

Data dredging: finding patterns that aren’t real 🔗

The final misconception is one that has become supercharged in the age of Big Data. It’s known by many names: data dredging, data snooping, or (my personal favorite) p-hacking. The core idea is this: if you have a massive dataset and you test enough different hypotheses, you are *guaranteed* to find correlations that look statistically significant, even though they are just the product of random chance.

It’s like torturing the data until it confesses to something. A proper scientific analysis works like this:

1. You form a hypothesis (e.g., “This new drug will lower blood pressure”).
2. You design an experiment to test *that specific hypothesis*.
3. You collect the data and see if your hypothesis was supported.

Data dredging flips this on its head. It works like this:

1. You collect a massive dataset (e.g., a survey with 500 questions about diet, lifestyle, health, and opinions).
2. You have no specific hypothesis. Instead, you feed all 500 variables into a computer and test every possible correlation against every other variable. (Diet vs. political opinion? Sleep hours vs. favorite color? Zip code vs. cholesterol?).
3. This process creates *thousands* or even *millions* of tests.
4. Eventually, the computer flags a few correlations that are “statistically significant” (meaning they have a low p-value, or a low probability of occurring by chance).
5. You then publish a paper acting as if you had intended to test this specific correlation all along.

Margarine and the divorce rate in Maine 🔗

This practice is how we get absurd, spurious correlations that are technically “real” in the data but utterly meaningless in the real world. A famous example from a website that tracks these is the near-perfect correlation (over 99%!) between the per capita consumption of margarine in the United States and the divorce rate in the state of Maine. Does this mean margarine is destroying marriages in Maine? Or that divorce makes people crave margarine? Of course not. It’s a spurious correlation-a pattern that emerged from a massive sea of data purely by dumb luck.

This is a huge problem. With modern computing power, anyone can dredge a dataset for these chance findings. A “statistically significant” result is typically one that has less than a 5% chance (a p-value of < 0.05) of happening randomly. But if you run 100 tests, you should *expect* to get 5 significant-looking results just by chance. If you run 1,000 tests, you’ll get 50. Data dredging is the practice of running those 1,000 tests, throwing away the 950 that showed nothing, and publishing the 50 “discoveries” as if they are meaningful breakthroughs.

This is why we must be so skeptical of headlines that announce shocking, single-study “discoveries” about food, health, or behavior. Was the hypothesis stated *before* the data was collected, or was it “discovered” *after* dredging the data? Without knowing the answer, we can’t know if the finding is a genuine effect or just a statistical ghost.

Conclusion: think like a data detective 🔗

These three misconceptions-confusing correlation with causation, getting fooled by Simpson’s Paradox, and falling for data dredging-all share a common theme. They arise when we treat numbers as absolute facts rather than as clues in a complex investigation. Data doesn’t speak for itself; it has to be interpreted. And that interpretation is where bias and error creep in.

To protect yourself, you don’t need to be a statistician, but you do need to be a critical thinker. When you see a claim backed by data, ask yourself these questions:

• Correlation vs. Causation: Are they just saying two things are linked, or are they claiming one *causes* the other? If it’s causation, could there be a third, lurking variable that’s really responsible for the pattern?
• Simpson’s Paradox: Is this an “overall” average? Could there be hidden subgroups (like age, gender, location, or department) that, if analyzed separately, would tell a completely different story?
• Data Dredging: Does this finding seem random or bizarre (like margarine and divorce)? Was the study designed to test this one specific hypothesis, or does it feel like a “fishing expedition” that just got lucky?

By asking these questions, you can move from being a passive consumer of data to an active, critical participant in the conversation. You’ll be able to spot the story behind the statistics and, more importantly, spot when the story is pure fiction.

What do you think? Which of these data traps have you seen most often in news reports or at your own workplace? What other data misconceptions do you think are dangerously common?

References

1. https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation
2. https://www.geeksforgeeks.org/machine-learning/reasons-why-correlation-does-not-imply-causation/
3. https://www.britannica.com/topic/Simpsons-paradox
4. https://plato.stanford.edu/entries/paradox-simpson/
5. https://en.wikipedia.org/wiki/Data_dredging