Logistic Regression for Binary Classification in Machine Learning

We make hundreds of “Yes” or “No” decisions every day. Will I buy this coffee? Is this email spam? Will this customer click the ad? This type of binary (two-option) question is one of the most common problems we try to solve with data. But how do you build a machine learning model that answers a “Yes/No” question? You can’t just draw a straight line. If you try, you’ll get nonsensical answers. Instead, you need a cleverer tool, one that’s built specifically for this job: Logistic Regression. Despite its name, it’s not actually for regression; it’s the go-to algorithm for classification.

Why a straight line just won’t work (Logistic vs. Linear Regression) 🔗

Let’s first remember its cousin, Linear Regression. Linear regression is fantastic for predicting continuous values-numbers that can fall anywhere on a spectrum. For example, you could use it to predict:

• A house’s price based on its square footage.
• A student’s exam score based on how many hours they studied.
• The number of ice creams sold based on the day’s temperature.

In all these cases, you’re fitting a straight line to your data. The equation is the classic y = b0 + b1x. If you study one more hour (x), your score (y) goes up by b1 points. Simple and effective.

Now, let’s try to use this “straight line” logic for a “Yes/No” problem. Imagine a bank wants to predict if a loan applicant will default (1) or not default (0) based on their credit score. This is a binary classification problem. If we try to fit a linear regression line to this data, we run into two massive problems:

1. The “Nonsense” Output: A straight line doesn’t stop. It will eventually predict values above 1 and below 0. What does a 1.2 (120%) probability of default mean? Or a -0.3 (-30%) probability? It’s meaningless. We need our answer to be strictly between 0 and 1, just like a real probability.
2. The “Bad Fit” Problem: The model will be heavily skewed by outliers, trying to draw a line through a set of points clustered at 0 and 1, leading to poor predictions.

We don’t want to predict a *value*; we want to predict a *probability*. This is the core difference. Logistic Regression is designed to answer: “What is the probability that this new instance belongs to the ‘1’ class (e.g., ‘Yes’, ‘Spam’, ‘Default’)?”

The magic S-curve: Understanding the Sigmoid Function 🔗

So, how do we take a linear equation (which can output any number from negative infinity to positive infinity) and “squish” its result into a clean 0-to-1 probability score? We use a special mathematical function called the Sigmoid Function, also known as the logistic function.

The equation for the sigmoid function looks like this:

σ(z) = 1 / (1 + e-z)

Let’s break down what this z is. Surprise! It’s just our old friend, the linear regression equation:

z = b0 + b1X1 + b2X2 + ... + bnXn

Here, the X variables are our features (e.g., credit score, income, age) and the b values are the coefficients the model learns. This linear equation outputs a single number, z, which we can think of as a “score.”

The sigmoid function then takes this score z and transforms it. Look at its three special properties:

• If z is a large positive number (e.g., 10), e-10 is a tiny, tiny number (close to 0). The formula becomes 1 / (1 + 0), which is 1.
• If z is a large negative number (e.g., -10), e-(-10) is e10, a massive number. The formula becomes 1 / (1 + large number), which is 0.
• If z is exactly 0, e0 is 1. The formula becomes 1 / (1 + 1), which is 0.5.

And just like that, we have a function that maps any real number into a 0-1 range. This output is our probability.

Putting it all together: The decision boundary 🔗

The model now has a clear job. It calculates the z score using its linear equation and plugs it into the sigmoid function to get a probability.

For example, if the model calculates a score z = 2.2 for an applicant, the probability of default is:

p(default) = 1 / (1 + e-2.2) ≈ 0.90 or 90%.

If another applicant gets a score z = -1.5:

p(default) = 1 / (1 + e-(-1.5)) ≈ 0.18 or 18%.

We then set a decision boundary, which is typically 0.5. If the model outputs a probability > 0.5, we classify it as ‘1’ (Default). If it’s < 0.5, we classify it as '0' (Not Default). Because the sigmoid function outputs 0.5 when z=0, our decision boundary is simply the line where b0 + b1X1 + ... = 0.

The equation that runs the show 🔗

So, we know the probability (p) is 1 / (1 + e-z). This is great for prediction, but it’s terrible for interpretation. In linear regression, we could say, “A 1-unit increase in X₁ changes Y by b₁.” Here, a 1-unit increase in X₁ changes the *input to a sigmoid function* by b₁. That’s not very intuitive for a stakeholder.

To understand what the coefficients *mean*, we have to do a little algebra and reverse-engineer the sigmoid function. When we solve that sigmoid equation for z, we get a new, beautiful equation. This is the true form of logistic regression:

log(p / (1 - p)) = b0 + b1X1 + b2X2 + ... + bnXn

This equation might look scarier, but it’s actually much more interpretable. Let’s break down that left side: log(p / (1 - p)).

Understanding Odds and Log-Odds 🔗

This equation introduces two crucial concepts: Odds and Log-Odds.

1. Probability (p): The chance of an event happening.
Example: The probability of rain is 0.8 (or 80%). The probability of no rain is 1 - p = 0.2 (or 20%).

2. Odds: This is a different way of expressing likelihood, common in statistics and betting. It’s the ratio of the probability of success to the probability of failure.
Formula: Odds = p / (1 - p)
Example: If the probability of rain is 0.8, the odds are 0.8 / 0.2 = 4. We would say the odds are “4 to 1 in favor” of rain.

3. Log-Odds (or “Logit”): This is simply the natural logarithm of the odds.
Formula: log(Odds) = log(p / (1 - p))
Example: The log-odds of rain are log(4) ≈ 1.386.

Now, look at our logistic regression equation again!

Log-Odds = b0 + b1X1 + b2X2 + ... + bnXn

The model is stating that the Log-Odds of an event are a linear combination of the input features. This is the breakthrough. We’ve connected our non-linear probability problem back to a simple, linear equation.

How to interpret the coefficients (the b values) 🔗

This “log-odds” form is what lets us understand our model’s “thinking.” A 1-unit increase in feature X₁ changes the log-odds of success by b1.

That’s still a bit abstract. So, we usually take it one step further by exponentiating the coefficient (eb1). This gives us the Odds Ratio. This tells us how a 1-unit increase in a feature *multiplies* the odds.

Example:
Imagine we’re predicting if a student will pass (1) or fail (0) based on hours studied (X₁). The model learns a coefficient b1 = 0.693.

• Interpretation 1 (Log-Odds): “For every additional hour a student studies, the log-odds of them passing increase by 0.693.” (Not very clear.)
• Interpretation 2 (Odds Ratio): Let’s calculate eb1.
  e0.693 ≈ 2.0
  This means: “For every additional hour a student studies, their odds of passing double.” Now *that’s* an insight you can share with a university dean!

This ability to move between probability, odds, and log-odds is what makes logistic regression both a powerful predictive tool and an excellent explanatory model.

Going beyond ‘Yes’ or ‘No’ 🔗

While this post has focused on Binary Logistic Regression (two outcomes), the algorithm is flexible. It can be extended to handle problems with more than two categories.

Multinomial Logistic Regression 🔗

This is used when you have three or more categories that have no natural order.
Examples:

• Predicting which brand of cereal a customer will buy (“Brand A”, “Brand B”, “Brand C”).
• Classifying a news article into a topic (“Sports”, “Politics”, “Technology”, “Business”).
• Identifying the species of a flower (“Setosa”, “Versicolor”, “Virginica”).

In this setup, “Sports” is not “higher” or “lower” than “Politics.” The model handles this by calculating the probability for each class simultaneously, ensuring all probabilities sum to 1. It’s like it runs multiple binary regressions in the background (e.g., “Brand A vs. All Others,” “Brand B vs. All Others”).

Ordinal Logistic Regression 🔗

This is used when you have three or more categories that do have a meaningful order, but the spacing between them isn’t necessarily equal.
Examples:

• Survey responses (“Strongly Disagree”, “Disagree”, “Neutral”, “Agree”, “Strongly Agree”).
• Patient condition (“Good”, “Fair”, “Poor”, “Critical”).
• Product ratings (“1 Star”, “2 Star”, “3 Star”, “4 Star”, “5 Star”).

We know “Agree” is more than “Neutral,” and “5 Stars” is more than “4 Stars.” An ordinal model is more powerful here than a multinomial one because it uses this ordering information. It learns a single linear equation but also finds the “cut-off” points (thresholds) on the log-odds scale that separate one category from the next.

From a simple “Yes/No” question, logistic regression opens a door to modeling complex human choices, medical outcomes, and customer behaviors, all while remaining one of the most interpretable and foundational algorithms in machine learning.

What do you think? Can you think of a “Yes/No” question in your daily life or work that could potentially be modeled with logistic regression? What features (the ‘X’ variables) would you use to predict that outcome?

References

1. https://www.ibm.com/topics/logistic-regression
2. https://www.analyticsvidhya.com/blog/2021/08/logistic-regression-in-machine-learning/
3. https://towardsdatascience.com/logistic-regression-detailed-overview-46c4050636d5
4. https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression