# The Mathematics of Ordinary Least Squares (OLS)

## 0. Overview

Ordinary Least Squares (OLS) is one of the most elegant examples in all of applied mathematics — a place where **calculus, geometry, and statistics intersect**.

It can be understood in two equivalent ways:

1. **Optimization View:** Minimize the sum of squared errors (MSE-based derivation).
2. **Covariance View:** Find coefficients such that residuals are **uncorrelated** with regressors.

Both lead to the same estimator:

\[ \hat{\beta} = (X'X)^{-1} X'y \]

---

## 1. The Model

We start with the standard linear model:

\[ y = X\beta + \varepsilon \]

where:

| Symbol | Description |
| --- | --- |
| y | n×1 vector of dependent variables |
| X | n×p matrix of regressors (columns = features) |
| β | p×1 vector of coefficients |
| ε | n×1 vector of random errors |

---

## 2. The Optimization View: Minimizing Squared Error

OLS minimizes the **sum of squared errors** (SSE):

\[ S(\beta) = (y - X\beta)'(y - X\beta) \]

Expanding gives:

\[ S(\beta) = y'y - 2y'X\beta + \beta'X'X\beta \]

Taking the derivative with respect to \( \beta \):

\[ \frac{\partial S}{\partial \beta} = -2X'y + 2X'X\beta \]

Set derivative to zero:

\[ X'X\hat{\beta} = X'y \]

Solve for \( \hat{\beta} \):

\[ \hat{\beta} = (X'X)^{-1} X'y \]

These are the **normal equations** of OLS.

---

## 3. The Covariance View: Orthogonality and Uncorrelated Residuals

OLS enforces that residuals are **orthogonal to the regressors**:

\[ X'(y - X\hat{\beta}) = 0 \]

In sample covariance terms:

\[ \operatorname{Cov}(x_j, \hat{\varepsilon}) = 0 \quad \forall j \]

This means:

> The best-fitting line is the one where the residuals have zero sample covariance with every regressor.

---

## 4. Simple Linear Regression Case

For a single predictor:

\[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i \]

The OLS slope \( \hat{\beta}_1 \) can be expressed as a ratio of covariances:

\[ \hat{\beta}_1 = \frac{\operatorname{Cov}(x, y)}{\operatorname{Var}(x)} \]

and the intercept:

\[ \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} \]

This compact form encapsulates the same condition:

\[ \sum_i x_i \hat{\varepsilon}_i = 0 \]

Residuals are uncorrelated with \( x \), and their mean is zero.

---

## 5. Error Variance and Coefficient Covariance

Once \( \hat{\beta} \) is known, residuals are:

\[ \hat{\varepsilon} = y - X\hat{\beta} \]

Estimated variance of the error term:

\[ \hat{\sigma}^2 = \frac{\hat{\varepsilon}'\hat{\varepsilon}}{n - p} \]

Covariance matrix of coefficient estimates:

\[ \operatorname{Var}(\hat{\beta}) = \hat{\sigma}^2 (X'X)^{-1} \]

Standard errors for each coefficient are the square roots of the diagonal elements.

---

## 6. Geometric View: Projection and Orthogonality

OLS has a beautiful geometric interpretation.

Define the **projection matrix**:

\[ P = X(X'X)^{-1} X' \]

Then:

\[ \hat{y} = Py \quad \text{and} \quad \hat{\varepsilon} = (I - P)y \]

Properties:
- \( P \) is symmetric and idempotent: \( P = P' = P^2 \)
- Residuals are orthogonal to fitted values: \( \hat{y}'\hat{\varepsilon} = 0 \)
- \( X'\hat{\varepsilon} = 0 \)

Thus, OLS is literally **the orthogonal projection of \( y \)** onto the column space of \( X \).

---

## 7. The Duality: Derivative = Orthogonality

Both derivations are equivalent:

| View | Condition | Interpretation |
| --- | --- | --- |
| **Derivative (Optimization)** | \( \frac{\partial S}{\partial \beta} = 0 \) | Minimize MSE |
| **Covariance (Orthogonality)** | \( X'(y - X\hat{\beta}) = 0 \) | Residuals uncorrelated with regressors |

They yield identical normal equations because:

\[ X'(y - X\beta) = 0 \Leftrightarrow \text{Cov}(X, \varepsilon) = 0 \]

So whether you minimize error energy or enforce zero correlation — you end up at the same \( \hat{\beta} \).

---

## 8. Statistical Properties (Gauss–Markov Conditions)

For OLS to be **BLUE** (Best Linear Unbiased Estimator):

1. \( E[\varepsilon | X] = 0 \) (unbiased errors)
2. \( \operatorname{Var}(\varepsilon | X) = \sigma^2 I \) (homoskedasticity)
3. No multicollinearity (invertibility of \( X'X \))

Then:

\[ E[\hat{\beta}] = \beta, \quad \operatorname{Var}(\hat{\beta}) = \sigma^2 (X'X)^{-1} \]

---

## 9. Summary Table

| Concept | Formula | Meaning |
| --- | --- | --- |
| OLS estimate | \( \hat{\beta} = (X'X)^{-1}X'y \) | Coefficients minimizing squared error |
| Residuals | \( \hat{\varepsilon} = y - X\hat{\beta} \) | Errors between prediction and data |
| Fitted values | \( \hat{y} = X\hat{\beta} \) | Projection of \( y \) onto column space of \( X \) |
| Error variance | \( \hat{\sigma}^2 = \frac{\hat{\varepsilon}'\hat{\varepsilon}}{n - p} \) | Estimate of noise variance |
| Covariance of \( \hat{\beta} \) | \( \hat{\sigma}^2 (X'X)^{-1} \) | Coefficient uncertainty |
| Orthogonality | \( X'\hat{\varepsilon} = 0 \) | Residuals uncorrelated with regressors |

---

## 10. TL;DR

OLS solves the problem of finding the line (or hyperplane) that:

- Minimizes the mean squared error
- Ensures residuals are orthogonal to all regressors
- Yields coefficients proportional to the covariance between \( X \) and \( y \)

You can view it as:

> **Derivative perspective:** “Find the stationary point of the MSE.”
>
> **Covariance perspective:** “Find the coefficients that make errors uncorrelated with inputs.”
>
> Both are just different lenses on the same underlying geometry:
>
> The orthogonal projection of \( y \) onto the space spanned by \( X \).

---

**Inertia7 Insight:**

OLS isn’t merely an algorithm — it’s a unification of calculus, geometry, and statistics.

Its strength lies in how elegantly these three views converge to the same simple truth:

\[ \hat{\beta} = (X'X)^{-1} X'y \]