Simple Linear Regression

1. Introduction

Imagine you run a small ice cream cart. You’ve noticed something over the summer: on hotter days, you sell more ice creams. On 25°C days you sell about 20 cones, on 35°C days you sell about 45 cones. You don’t have an exact formula in your head, but your gut tells you “hotter temperature → more sales, roughly in a straight-line kind of way.”

Now imagine tomorrow’s forecast says 30°C. Can you predict how many cones you’ll sell — not just guess, but calculate a number based on the pattern you’ve seen before? That’s exactly the problem Simple Linear Regression (SLR) solves.

Formally: Simple Linear Regression is a statistical method used to model the relationship between one independent variable (input, like temperature) and one dependent variable (output, like ice cream sales) by fitting a straight line through the data points.

It’s called “simple” because there’s only one input variable involved (as opposed to multiple linear regression, which uses several inputs), and “linear” because we assume the relationship looks like a straight line, not a curve.

At its heart, SLR answers one question: “If I know X, what’s my best guess for Y, assuming a straight-line relationship?”

2. History

The story of linear regression doesn’t start with computers or data scientists — it starts with astronomers trying to track comets and planets in the late 1700s.

Scientists at the time were taking multiple measurements of the same celestial object’s position, but every measurement had small errors — telescopes weren’t perfect, human reaction times varied, atmospheric conditions changed. If you had 10 different measurements of a planet’s orbit, which one was “correct”? None of them, exactly. So mathematicians needed a way to find the best-fitting line or curve through noisy, imperfect data.

In 1805, a French mathematician named Adrien-Marie Legendre published the method of least squares — a technique to find the line that minimizes the total squared distance between the line and the actual data points. This became the mathematical backbone of what we now call linear regression.

Shortly after, Carl Friedrich Gauss, the famous German mathematician, claimed he had actually been using the same method since 1795 — a decade before Legendre published it — to predict the position of the asteroid Ceres. This sparked a famous priority dispute between the two, but today Gauss’s deeper theoretical justification (connecting least squares to probability and the normal distribution) is considered foundational, alongside Legendre’s original publication.

The term “regression” itself came later, from an unexpected source: biology. In the 1880s, Sir Francis Galton, a British scientist studying heredity, noticed something curious while studying the heights of parents and their children. Very tall parents tended to have children who were tall — but usually not as tall as the parents. Very short parents had children who were short — but usually not as short. Galton called this phenomenon “regression toward mediocrity” (later renamed “regression toward the mean”). The statistical technique he used to describe this pattern borrowed the same least-squares math from Legendre and Gauss, and the name “regression” stuck to the whole method, even though most modern uses of regression have nothing to do with height inheritance.

From there, regression analysis exploded in use — economists used it to model supply and demand, engineers used it to model stress and strain, and by the 20th century, it became one of the most widely used tools in statistics, and eventually one of the first algorithms taught in machine learning.

3. Core Concepts

3.1 The Building Blocks

Before diving into formulas, let’s define the vocabulary you’ll see everywhere in regression.

3.2 What Does “Best Fit” Mean?

If you scatter-plot temperature vs. ice cream sales, the points won’t sit perfectly on a line — real-world data is noisy. So there isn’t one obviously “correct” line; there are infinitely many lines you could draw.

Simple Linear Regression picks the one specific line that minimizes the total error between the line’s predictions and the actual data points. This is the “best fit” line — and how we measure “total error” precisely is what the math section below explains.

3.3 The Model Itself

Every simple linear regression model has the same basic shape:

Where:

• $\hat{y}$ (read “y-hat”) is the predicted value of Y (not the actual observed value — we use the hat symbol specifically to mark it as a prediction)
• $x$ is the input
• $\beta_0$ is the intercept
• $\beta_1$ is the slope

This should look familiar — it’s just the equation of a straight line, $y = mx + c$, wearing statistics clothing.

4. The Math — Building It Step by Step

4.1 Step 1: Defining Error for a Single Point

For any data point $(x_i, y_i)$, the model predicts $\hat{y}_i = \beta_0 + \beta_1 x_i$. But the actual value is $y_i$. The difference between them is called the residual:

This tells us how wrong the line is for that one specific point. A positive residual means the actual point is above the line; a negative residual means it’s below.

4.2 Step 2: Why We Can’t Just Add Up the Errors

You might think: “just add up all the residuals across every point, and pick the line that makes that sum smallest.” But there’s a problem — some residuals are positive (points above the line) and some are negative (points below the line). They can cancel each other out, making a genuinely bad line look artificially good on paper.

Fix: Square each residual before summing. Squaring makes every term positive, so errors can’t cancel out — and it also punishes large errors more heavily than small ones, which is usually what we want (a line that’s wildly wrong for one point should be penalized more than a line that’s slightly off for many points).

4.3 Step 3: The Cost Function — Sum of Squared Errors

This gives us the quantity we want to minimize, called the Sum of Squared Errors (SSE), or equivalently the least squares cost function:

Our whole goal now becomes: find the values of $\beta_0$ and $\beta_1$ that make this SSE as small as possible. This is the “least squares” idea from Legendre and Gauss, mentioned in the History section.

4.4 Step 4: Solving for the Slope and Intercept

Using calculus (taking derivatives of SSE with respect to $\beta_0$ and $\beta_1$, and setting them to zero to find the minimum), we get clean closed-form formulas:

Where $\bar{x}$ and $\bar{y}$ are the means (averages) of all X and Y values.

Why does the slope formula make sense in plain words?

• The numerator, $\sum (x_i - \bar{x})(y_i - \bar{y})$, measures how X and Y move together. If X above-average tends to pair with Y above-average (and vice versa), this sum is large and positive — meaning “as X goes up, Y tends to go up.” This is essentially covariance.
• The denominator, $\sum (x_i - \bar{x})^2$, measures how spread out X is — the variance of X.
• So slope = (how much X and Y move together) ÷ (how much X varies on its own). In plain English: “for every unit X typically moves, how many units does Y typically move along with it?”

Why does the intercept formula make sense? Once we know the slope, the intercept just makes sure the line passes through the “center of mass” of the data — the point $(\bar{x}, \bar{y})$. It’s the anchor that positions the line correctly on the graph after the slope has fixed its angle.

4.5 Geometric Interpretation

Picture a scatter plot of all your (X, Y) points. The regression line is the one straight line where, if you drew a vertical segment from every point down (or up) to the line and measured its length, then squared and summed all those lengths — no other line would give you a smaller total. It’s literally the line of “least resistance” through the cloud of points, balanced so the squared vertical distances are minimized on both sides.

4.6 Measuring How Good the Fit Is: R²

Once you have your line, how do you know if it’s actually a good fit, or just the “least bad among bad options”? We use R² (R-squared), also called the coefficient of determination:

In plain words: it compares “how much error your line has” to “how much error you’d have if you just guessed the average $\bar{y}$ every single time, ignoring X entirely.” $R^2$ ranges from 0 to 1 — closer to 1 means your line explains the pattern in the data very well; closer to 0 means X isn’t helping you predict Y much better than just guessing the average.

Adjusted R²

Adjusted R² is a modified version of R² that penalizes you for adding predictors that don’t actually help the model.

The problem with plain R²

Regular R² measures how much variance in y is explained by your model:

where $SS_{res} = \sum(y_i - \hat{y}_i)^2$ (residual sum of squares) and $SS_{tot} = \sum(y_i - \bar{y})^2$ (total sum of squares).

The issue: R² never decreases when you add more features — even totally useless, random ones. Add enough junk predictors and R² will keep creeping toward 1, making your model look better than it really is. This is misleading, especially when comparing models with different numbers of features.

The fix: Adjusted R²

where:

• n = number of data points (sample size)
• k = number of predictors (independent variables), not counting the intercept

What it does differently

Adjusted R² adds a penalty term based on the number of predictors relative to the sample size. If a new feature actually improves the model meaningfully, adjusted R² increases. If you add a feature that’s essentially noise, adjusted R² decreases — even though plain R² would still go up (or stay flat).

Quick intuition

Why it matters

• Use R² to describe how well your current model fits the data.
• Use Adjusted R² when comparing models with different numbers of features — it tells you whether adding a variable was actually worth it, or if you’re just overfitting.

It’s especially important in multiple linear regression, where it’s tempting to throw in lots of features hoping to boost R².

5. Types / Variants of Regression

Simple Linear Regression is the entry point into a whole family of regression techniques. Here’s how they relate:

• Simple Linear Regression is like predicting your commute time using just the distance you’re driving.
• Multiple Linear Regression is like predicting commute time using distance and time of day and weather — multiple inputs feeding one output.
• Polynomial Regression is like realizing your commute time doesn’t increase in a straight line with distance (maybe traffic makes it worse disproportionately for longer distances), so you fit a curve instead of a line.
• Logistic Regression is like switching the question from “how long will my commute take” (a number) to “will I be late or not” (yes/no) — despite the name, it’s used for classification, not predicting continuous numbers.

6. Limitations and Trade-offs

6.1 Assumes a Straight-Line Relationship

Worked example: Suppose you’re modeling plant growth vs. days of sunlight. In reality, a plant grows fast at first, then growth slows down and plateaus once it matures. If you force a straight line through this curved data, the line will predict the plant keeps growing forever at a constant rate — clearly wrong, and it will systematically under-predict growth early on and over-predict it later.

Fix that came later: This limitation motivated polynomial regression and other non-linear models that can bend and curve to match the true shape of the data.

6.2 Extremely Sensitive to Outliers

Worked example: Say 9 out of 10 houses in your dataset are priced between ₹40–60 lakhs based on size, but one mansion is priced at ₹5 crore due to a data entry error. Because SLR minimizes squared error, that one outlier gets squared into a massive penalty, and the whole line tilts to try to accommodate it — dragging predictions for all the normal houses off track.

Fix that came later: Techniques like robust regression or simply removing/investigating outliers before fitting were developed to reduce this sensitivity.

6.3 Assumes the Relationship Is Truly Causal-Looking, But It Isn’t Causation

Just because ice cream sales and temperature move together doesn’t prove temperature causes sales (though here it plausibly does). In many real datasets, two variables move together due to a hidden third factor, and SLR can’t tell the difference — it only reports correlation, never true causation, no matter how good the fit looks.

6.4 Only Handles One Input Variable

By definition, SLR ignores every other factor that might influence Y. If ice cream sales are also affected by whether it’s a weekend, SLR using temperature alone will systematically miss part of the picture.

Fix that came later: This is precisely why Multiple Linear Regression was developed — to let more than one input variable contribute to the prediction.

7. Connecting to What You Already Know

If you’ve already learned about neurons, perceptrons, machine learning types, and deep learning, here’s how SLR fits into that bigger picture:

• The perceptron connection: A single perceptron computes a weighted sum of inputs plus a bias, i.e., $z = w \cdot x + b$. Compare that to SLR’s $\hat{y} = \beta_1 x + \beta_0$ — it’s structurally the exact same equation. The slope $\beta_1$ plays the role of the weight (w), and the intercept $\beta_0$ plays the role of the bias (b). In fact, a perceptron with no activation function, doing regression instead of classification, is essentially simple linear regression.

• The biological neuron connection: Just as a biological neuron takes weighted electrical/chemical signals from other neurons and combines them before firing, SLR takes an input X, multiplies it by a “strength” (the slope), and adds a baseline offset (the intercept) to produce an output. It’s the same core idea of “combine inputs with learned weights” scaled down to its simplest possible form — one input, one output, no activation function, no layers.

• The deep learning connection: Deep learning networks are essentially stacks of these weighted-sum-plus-bias units (like SLR/perceptrons), layered on top of each other with non-linear activation functions in between, so the network can learn curved, complex relationships instead of just straight lines. In other words, SLR is the single-cell version of what deep learning does with millions of interconnected cells.

• The machine learning types connection: SLR is a classic example of supervised learning — specifically regression (predicting a continuous number), as opposed to classification (predicting a category, like logistic regression does). It’s usually the very first supervised learning algorithm taught, precisely because its math is fully solvable by hand (via the least-squares formulas above), unlike most deep learning models which require iterative optimization (like gradient descent) instead of a clean closed-form answer.

8. Quick Recap Table

9. Key Takeaway

Simple Linear Regression is, at its core, the simplest possible way to teach a machine to draw a straight line through noisy, real-world data so it can predict one number from another — it’s the same “weighted input plus bias” idea that powers a single perceptron, and by extension the same core building block that, when stacked into millions of connected units with non-linear twists, becomes deep learning. Understanding SLR deeply — the line, the least-squares math behind it, and why it can fail on curved or outlier-heavy data — gives you the foundation on which almost every more advanced predictive model, from polynomial regression to neural networks, is ultimately built.