Foundations of AI: A Review of Probability Theory

Ever wondered how your favourite AI recommendation engine, be it for movies, music, or online shopping, manages to make such smart guesses? It all boils down to one powerful mathematical tool: Probability Theory. This isn’t just about flipping coins or rolling dice; it’s the language AI uses to handle the messy, uncertain real world. If you want to understand the *why* behind machine learning’s smart decisions, you need to start with the basics of probability. Let’s peel back the curtain and explore the foundational concepts that form the bedrock of artificial intelligence.

The starting line: Trials, sample space, and events 🔗

To talk about uncertainty, we first need a framework for organizing all the potential outcomes. This framework starts with three core concepts that transform a vague situation into a measurable one.

Random experiments and trials 🔗

A random experiment is essentially any process where the outcome can’t be predicted with certainty beforehand, even if we know all the possible results. Think of it like a scientist running an experiment, but in the world of probability. A single performance of this experiment is called a trial.

• Experiment: Flipping a standard coin.
• Trial: A single flip of that coin.
• Experiment: Checking if a loan applicant defaults on their payment.
• Trial: Processing one specific loan application.

The universe of outcomes: Sample space 🔗

The sample space ($\text{S}$ or $\Omega$) is the collection of *all* possible outcomes of a random experiment. It’s the entire universe of possibilities. Every result from your trial must be an element of the sample space.

• For a single coin flip, the sample space is $\text{S} = \{ \text{Heads}, \text{Tails} \}$.
• For rolling a standard six-sided die, the sample space is $\text{S} = \{ 1, 2, 3, 4, 5, 6 \}$.
• For the experiment of tossing *two* coins, the sample space expands to $\text{S} = \{ \text{HH}, \text{HT}, \text{TH}, \text{TT} \}$, where H is Heads and T is Tails.

In AI, the sample space might be the set of all possible classifications for an image (e.g., \{‘Cat’, ‘Dog’, ‘Bird’\}) or the entire range of potential stock prices on a given day. Defining this space clearly is the first step toward building a successful probabilistic model.

The focus area: Events 🔗

An event ($\text{E}$) is simply any subset of the sample space. It’s a particular outcome, or a collection of outcomes, that we are interested in. If the outcome of a trial is included in the event, we say the event has “occurred.”

• From the die roll sample space $\text{S} = \{ 1, 2, 3, 4, 5, 6 \}$, the event “rolling an even number” is $\text{E} = \{ 2, 4, 6 \}$.
• The event “rolling a 5” is a simple event: $\text{E} = \{ 5 \}$.

In an AI context, an event could be “the customer clicks on the ad” or “the temperature exceeds 40 degrees Celsius.” Understanding the relationship between these events is crucial for probabilistic reasoning.

—

The logic puzzle: The algebra of events 🔗

Just as numbers have arithmetic (addition, subtraction), events have an “algebra” that allows us to combine them logically using set operations. This is how AI systems reason about complex situations built from simpler ones.

Union ($\text{E} \cup \text{F}$): The ‘or’ operation 🔗

The union of two events, $\text{E}$ and $\text{F}$, written as $\text{E} \cup \text{F}$, is the event that occurs if at least one of the events $\text{E}$ or $\text{F}$ occurs. Think of the symbol $\cup$ as standing for ‘or’.

Consider rolling a single die:

• Event $\text{E}$: Rolling an even number ($\{ 2, 4, 6 \}$).
• Event $\text{F}$: Rolling a number less than or equal to 3 ($\{ 1, 2, 3 \}$).
• The union $\text{E} \cup \text{F}$ is the event of rolling an even number *or* a number less than or equal to 3: $\{ 1, 2, 3, 4, 6 \}$.

If an AI is trying to predict if a user will engage with content, the event could be: “User clicks on the video $\cup$ User shares the article.” If either one happens, the union event is a success.

Intersection ($\text{E} \cap \text{F}$): The ‘and’ operation 🔗

The intersection of two events, $\text{E}$ and $\text{F}$, written as $\text{E} \cap \text{F}$, is the event that occurs if both events $\text{E}$ and $\text{F}$ occur simultaneously. Think of the symbol $\cap$ as standing for ‘and’.

Using the same die roll example:

• $\text{E} = \{ 2, 4, 6 \}$
• $\text{F} = \{ 1, 2, 3 \}$
• The intersection $\text{E} \cap \text{F}$ is the event of rolling an even number *and* a number less than or equal to 3. The only outcome that satisfies both is $\{ 2 \}$.

In a medical diagnosis application, the intersection could be: “Patient exhibits Symptom A $\cap$ Patient tests positive for Condition B.” This dual occurrence is often the key to high-confidence decision-making.

No common ground: Mutually exclusive events 🔗

Two events, $\text{E}$ and $\text{F}$, are mutually exclusive (or disjoint) if they cannot occur at the same time in a single trial. In terms of their intersection, this means $\text{E} \cap \text{F}$ is the empty set ($\emptyset$), and the probability of their intersection is zero, $\text{P}(\text{E} \cap \text{F}) = 0$. This is a vital concept because it greatly simplifies the calculation of their union: $\text{P}(\text{E} \cup \text{F}) = \text{P}(\text{E}) + \text{P}(\text{F})$.

Relatable Example:

Imagine you’re checking a deck of 52 playing cards. The event “drawing a King” and the event “drawing an Ace” are mutually exclusive; a single card can’t be both a King and an Ace. They have no overlap. However, the event “drawing a King” and the event “drawing a Heart” are *not* mutually exclusive because the King of Hearts exists-that’s their intersection!

—

Quantifying the chance: Measuring likelihood with relative frequency 🔗

Once we define our events, how do we assign a number to their likelihood? One of the foundational ways to interpret the probability of an event is through its relative frequency.

The frequentist view of probability 🔗

The relative frequency approach, also known as the frequentist interpretation, defines the probability of an event based on how often it occurs over a large number of trials. We don’t need to know the ‘theoretical’ probability (like the $1/6$ for a die roll); we find the probability by actually observing the outcome over and over again.

The formula for the relative frequency of an event $\text{E}$ after $n$ trials is:

$$\text{Relative Frequency} = \frac{\text{Frequency of Event E}}{\text{Total number of trials } (n)} = \frac{f_n(\text{E})}{n}$$

The law of large numbers in action 🔗

The magic happens as we increase the number of trials, $n$. The Law of Large Numbers states that as $n$ approaches infinity, the relative frequency of the event tends to stabilize and get closer and closer to the true, theoretical probability of that event. This stabilization gives us an empirical, data-driven way to estimate probability.

Story Time: The Coin Toss Experiment

Suppose you flip a fair coin. Theoretically, the probability of getting Heads is $0.5$ (or $50\%$). Let’s track the results:

Notice how the relative frequency is erratic at first (0.00, then 0.70), but as the number of trials grows to $1,000$ and then $10,000$, the ratio gets incredibly close to the theoretical probability of $0.5$. This convergence is precisely why relative frequency is a key estimator for probability.

Relevance to AI and Machine Learning 🔗

In Machine Learning (ML), we often don’t have a simple theoretical formula for probability. For instance, what’s the theoretical probability of a particular image being a dog? It’s impossible to calculate a simple ratio. Instead, ML models use the relative frequency approach:

• The trials are the massive datasets of images an AI system is trained on.
• The relative frequency of a feature (e.g., a specific texture or shape) is observed across thousands of images labeled as ‘Dog’.
• This observed relative frequency is then used as the *estimate* of the probability that the next image with similar features is also a dog.

In India, for example, the Ministry of Commerce or the Reserve Bank of India might use the relative frequency of certain economic indicators (like business failures or stock market volatility) observed over decades of data to model future economic uncertainty. Probability theory, therefore, isn’t an abstract concept; it is the essential mathematical bridge between raw data and informed decision-making in the world of AI.

What do you think? Can you think of a real-world decision (like choosing an investment or planning your day) that is essentially a calculation of a union or intersection of events? How does the concept of relative frequency apply to something you learn from repeatedly doing a task?

References

1. https://www.blueskydigitalassets.com/understanding-what-is-probability-theory-in-ai-a-simple-guide/
2. https://stats.libretexts.org/Bookshelves/Introductory_Statistics/Mostly_Harmless_Statistics_(Webb)/04%3A_Probability/4.04%3A_Union_and_Intersection
3. https://www.mathsisfun.com/data/probability-events-mutually-exclusive.html
4. https://www.scribbr.com/frequently-asked-questions/relative-frequency-vs-probability/
5. https://www.rbi.org.in/scripts/BS_SpeechesView.aspx?Id=1236