The Mathematics of Ordinary Least Squares (OLS)

0. Overview

Ordinary Least Squares (OLS) is one of the most elegant examples in all of applied mathematics — a place where calculus, geometry, and statistics intersect.

It can be understood in two equivalent ways:

1. Optimization View: Minimize the sum of squared errors (MSE-based derivation).
2. Covariance View: Find coefficients such that residuals are uncorrelated with regressors.

Both lead to the same estimator:

[ \hat{\beta} = (X'X)^{-1} X'y ]

1. The Model

We start with the standard linear model:

[ y = X\beta + \varepsilon ]

where:

2. The Optimization View: Minimizing Squared Error

OLS minimizes the sum of squared errors (SSE):

[ S(\beta) = (y - X\beta)'(y - X\beta) ]

Expanding gives:

[ S(\beta) = y'y - 2y'X\beta + \beta'X'X\beta ]

Taking the derivative with respect to ( \beta ):

[ \frac{\partial S}{\partial \beta} = -2X'y + 2X'X\beta ]

Set derivative to zero:

[ X'X\hat{\beta} = X'y ]

Solve for ( \hat{\beta} ):

[ \hat{\beta} = (X'X)^{-1} X'y ]

These are the normal equations of OLS.

3. The Covariance View: Orthogonality and Uncorrelated Residuals

OLS enforces that residuals are orthogonal to the regressors:

[ X'(y - X\hat{\beta}) = 0 ]

In sample covariance terms:

[ \operatorname{Cov}(x_j, \hat{\varepsilon}) = 0 \quad \forall j ]

This means:

The best-fitting line is the one where the residuals have zero sample covariance with every regressor.

4. Simple Linear Regression Case

For a single predictor:

[ y_i = \beta_0 + \beta_1 x_i + \varepsilon_i ]

The OLS slope ( \hat{\beta}_1 ) can be expressed as a ratio of covariances:

[ \hat{\beta}_1 = \frac{\operatorname{Cov}(x, y)}{\operatorname{Var}(x)} ]

and the intercept:

[ \hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x} ]

This compact form encapsulates the same condition:

[ \sum_i x_i \hat{\varepsilon}_i = 0 ]

Residuals are uncorrelated with ( x ), and their mean is zero.

5. Error Variance and Coefficient Covariance

Once ( \hat{\beta} ) is known, residuals are:

[ \hat{\varepsilon} = y - X\hat{\beta} ]

Estimated variance of the error term:

[ \hat{\sigma}^2 = \frac{\hat{\varepsilon}'\hat{\varepsilon}}{n - p} ]

Covariance matrix of coefficient estimates:

[ \operatorname{Var}(\hat{\beta}) = \hat{\sigma}^2 (X'X)^{-1} ]

Standard errors for each coefficient are the square roots of the diagonal elements.

6. Geometric View: Projection and Orthogonality

OLS has a beautiful geometric interpretation.

Define the projection matrix:

[ P = X(X'X)^{-1} X' ]

Then:

[ \hat{y} = Py \quad \text{and} \quad \hat{\varepsilon} = (I - P)y ]

Properties:

• ( P ) is symmetric and idempotent: ( P = P' = P^2 )
• Residuals are orthogonal to fitted values: ( \hat{y}'\hat{\varepsilon} = 0 )
• ( X'\hat{\varepsilon} = 0 )

Thus, OLS is literally the orthogonal projection of ( y ) onto the column space of ( X ).

7. The Duality: Derivative = Orthogonality

Both derivations are equivalent:

They yield identical normal equations because:

[ X'(y - X\beta) = 0 \Leftrightarrow \text{Cov}(X, \varepsilon) = 0 ]

So whether you minimize error energy or enforce zero correlation — you end up at the same ( \hat{\beta} ).

8. Statistical Properties (Gauss–Markov Conditions)

For OLS to be BLUE (Best Linear Unbiased Estimator):

1. ( E[\varepsilon | X] = 0 ) (unbiased errors)
2. ( \operatorname{Var}(\varepsilon | X) = \sigma^2 I ) (homoskedasticity)
3. No multicollinearity (invertibility of ( X'X ))

Then:

[ E[\hat{\beta}] = \beta, \quad \operatorname{Var}(\hat{\beta}) = \sigma^2 (X'X)^{-1} ]

9. Summary Table

10. TL;DR

OLS solves the problem of finding the line (or hyperplane) that:

• Minimizes the mean squared error
• Ensures residuals are orthogonal to all regressors
• Yields coefficients proportional to the covariance between ( X ) and ( y )

You can view it as:

Derivative perspective: “Find the stationary point of the MSE.”

Covariance perspective: “Find the coefficients that make errors uncorrelated with inputs.”

Both are just different lenses on the same underlying geometry:

The orthogonal projection of ( y ) onto the space spanned by ( X ).

Inertia7 Insight:

OLS isn’t merely an algorithm — it’s a unification of calculus, geometry, and statistics.

Its strength lies in how elegantly these three views converge to the same simple truth:

[ \hat{\beta} = (X'X)^{-1} X'y ]