# Linear Algebra: The Geometry of Space and Transformation

Technology: **Math**  
Skill: **Mathing**

## Overview

Linear algebra is the **language of structure and transformation**.

While calculus studies _change_, linear algebra studies the _spaces where change happens_ — how vectors, planes, and higher-dimensional structures interact, align, and evolve.

It underlies computer graphics, quantum mechanics, data science, and machine learning — wherever quantities live in multidimensional space.

This document moves from vectors and matrices to orthogonality, eigenvalues, and the singular value decomposition.

The appendix then shows how to compute each of these concepts step by step.

## 1. Vectors and Spaces — The Building Blocks of Structure

### 1.1 Vectors as Geometric Objects

A **vector** is both _direction_ and _magnitude_.

\[
v = \begin{bmatrix} v_x \\ v_y \end{bmatrix}, \quad \mathbf{v} = \begin{bmatrix} v_x \\ v_y \\ v_z \end{bmatrix}
\]

The **length** of a vector is its Euclidean norm:

\[
\|\mathbf{v}\| = \sqrt{v_x^2 + v_y^2 + v_z^2}
\]

The **unit vector** in the same direction is:

\[
hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|}
\]

Geometrically, vectors describe positions, velocities, forces, or any quantity that has both magnitude and direction.

### 1.2 Vector Operations

**Addition:**

\[
\mathbf{u} + \mathbf{v} = \begin{bmatrix} u_x + v_x \\ u_y + v_y \end{bmatrix}
\]

Combines directions head-to-tail.

**Scalar Multiplication:**

\[
c\mathbf{v} = \begin{bmatrix} cv_x \\ cv_y \end{bmatrix}
\]

Scales magnitude, preserves direction.

**Dot Product:**

\[
\mathbf{u} \cdot \mathbf{v} = u_xv_x + u_yv_y + u_zv_z = \|\mathbf{u}\|\|\mathbf{v}\|\cos \theta
\]

Measures _alignment_.

- If = 0, vectors are **orthogonal**.
- If positive, acute angle; if negative, obtuse.

**Cross Product (3D only):**

\[
\mathbf{u} \times \mathbf{v} = \begin{bmatrix} u_yv_z - u_zv_y \\ u_zv_x - u_xv_z \\ u_xv_y - u_yv_x \end{bmatrix}
\]

Gives a vector perpendicular to both — defines orientation and area.

### 1.3 Vector Spaces and Subspaces

A **vector space** is a set of vectors closed under addition and scalar multiplication.

The **span** of {v1,v2} is all combinations:

\[
\text{Span}\{\mathbf{v}_1, \mathbf{v}_2\} = \{ a\mathbf{v}_1 + b\mathbf{v}_2 \mid a,b \in \mathbb{R} \}
\]

A **subspace** is any subset that’s also a vector space (e.g., a line or plane through the origin).

A **basis** is a minimal set of linearly independent vectors that span the space.

The **dimension** equals the number of basis vectors.

## 2. Matrices — Linear Transformations and Systems

### 2.1 Matrices as Transformations

A matrix encodes a **linear transformation**:

\[
A\mathbf{x} = \mathbf{b}
\]

If A is m×n, it maps \( \mathbb{R}^n \to \mathbb{R}^m \).

Each column of A shows how a basis vector transforms.

Matrices stretch, rotate, reflect, and shear space.

**Example:**

\[
A = \begin{bmatrix} 2 & 0 \\ 0 & 3 \end{bmatrix} \Rightarrow A\begin{bmatrix} 1 \\ 1 \end{bmatrix} = \begin{bmatrix} 2 \\ 3 \end{bmatrix}
\]

Space is stretched by 2 along x and 3 along y.

### 2.2 Matrix Operations

| Operation  | Meaning                                    |
|------------|--------------------------------------------|
| A + B      | Elementwise addition                        |
| AB         | Composition of transformations              |
| A^TA      | Reflect across diagonal (swap rows/cols)  |
| A^{-1}    | Reverse transformation (if exists)         |
| det(A)    | Volume scaling factor                       |

If det(A) = 0, the transformation collapses space — no inverse.

### 2.3 Solving Systems of Equations

Systems of linear equations can be written compactly as \( A\mathbf{x} = \mathbf{b} \).

If det(A)≠0, the unique solution is:

\[
\mathbf{x} = A^{-1}\mathbf{b}
\]

If A is not invertible or overdetermined, approximate with **least squares**:

\[
A^TA\mathbf{x} = A^T\mathbf{b}
\]

## 3. Orthogonality and Projections

### 3.1 Orthogonality

Two vectors are **orthogonal** if their dot product is zero:

\[
\mathbf{u} \cdot \mathbf{v} = 0
\]

### 3.2 Projection

To find the component of a \( \mathbf{a} \) along \( \mathbf{b} \):

\[
\text{proj}\_{\mathbf{b}}(\mathbf{a}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\mathbf{b}
\]

The residual (error) orthogonal to \( \mathbf{b} \) is:

\[
\mathbf{a}_\perp = \mathbf{a} - \text{proj}\_{\mathbf{b}}(\mathbf{a})
\]

### 3.3 Orthonormal Bases

An **orthonormal basis** satisfies:

\[
\mathbf{v}_i\cdot\mathbf{v}_j = \begin{cases} 1, & i=j \\ 0, & i \neq j \end{cases}
\]

Any vector \( \mathbf{x} \) decomposes uniquely as:

\[
\mathbf{x} = \sum_i (\mathbf{x} \cdot \mathbf{v}_i) \mathbf{v}_i
\]

## 4. Eigenvalues and Eigenvectors — Directions That Stay Put

### 4.1 Definition

For a square matrix A:

\[
A\mathbf{v} = \lambda\mathbf{v}
\]

- \( \mathbf{v} \) is an **eigenvector** (direction stays the same)
- \( \lambda \) is an **eigenvalue** (scaling factor)

They satisfy the **characteristic equation**:

\[
\det(A - \lambda I) = 0
\]

### 4.2 Interpretation

- Eigenvectors are _invariant directions_ of transformation.
- Eigenvalues describe _how much_ those directions are stretched or compressed.
- If \( |\lambda| < 1 \), repeated applications shrink the vector (stability).
- If \( |\lambda| > 1 \), the system amplifies along that direction.

## 5. Diagonalization and the Spectral Theorem

If a matrix has \( n \) independent eigenvectors:

\[
A = PDP^{-1}
\]

- \( P \) contains eigenvectors as columns
- \( D \) is diagonal with eigenvalues on the diagonal

Then powers of \( A \) are easy:

\[
A^k = P D^k P^{-1}
\]

If A is **symmetric**, all eigenvalues are real and eigenvectors can be chosen orthonormal.

This is the **Spectral Theorem** — the backbone of PCA and quantum mechanics.

## 6. Least Squares and Projections Revisited

When \( A\mathbf{x} = \mathbf{b} \) has no exact solution (overdetermined system), minimize the squared error:

\[
\min_{\mathbf{x}} \|A\mathbf{x} - \mathbf{b}\|^2
\]

Setting derivative = 0 gives:

\[
A^TA\mathbf{x} = A^T\mathbf{b}
\]

## 7. Singular Value Decomposition (SVD)

Every matrix A (even non-square) can be decomposed as:

\[
A = U\Sigma V^T
\]

where:

- U — orthogonal basis for outputs
- V — orthogonal basis for inputs
- \( \Sigma \) — diagonal with nonnegative **singular values** \( \sigma_i \)

## 8. Determinants and Volume

For a square matrix:

\[
\det(A)
\]

measures how A scales _volume_ and changes _orientation_.

| \( \det(A) \)  | Interpretation      |
|------------------|----------------------|
| 1                | Preserves volume     |
| >1               | Expands space        |
| <0               | Reflects (flips orientation) |
| 0                | Collapses to lower dimension (no inverse) |

## 9. The Unified Picture

| Concept      | Meaning                            | Application                     |
|--------------|------------------------------------|---------------------------------|
| Vector       | Quantity with direction/magnitude   | Forces, data points             |
| Matrix       | Linear transformation               | Rotations, scaling              |
| Dot Product  | Alignment measure                   | Orthogonality                   |
| Cross Product| Perpendicular vector                | Torque, normals                 |
| Eigenvalues  | Invariant scaling                   | PCA, stability                  |
| SVD          | Universal factorization             | Compression, ML                 |
| Least Squares| Best linear fit                    | Regression                       |

Linear algebra defines **how space transforms**, how **data decomposes**, and how **systems align**.

It is the structural engine of all continuous mathematics.

# Appendix: Linear Algebra in Practice

The appendix turns abstract concepts into **computable procedures** — showing how to perform, interpret, and reason through each operation in linear algebra.

Each subsection connects the **symbolic process** to its **geometric intuition**.

## A1. Vector Computations

Vectors are the atoms of linear algebra. Every operation — matrix multiplication, projection, eigenanalysis — ultimately manipulates vectors.

### A1.1 Magnitude and Unit Vector

The **magnitude** of a vector is its length in space.

The **unit vector** gives the same direction but normalizes length to 1.

#### Example:

\[
\mathbf{v} = (3,4)
\]

Compute its magnitude:

\[
\|\mathbf{v}\| = \sqrt{3^2 + 4^2} = 5
\]

Normalize:

\[
\hat{\mathbf{v}} = \frac{\mathbf{v}}{\|\mathbf{v}\|} = \left(\frac{3}{5}, \frac{4}{5}\right)
\]

### A1.2 Dot Product — Measuring Alignment

The **dot product** measures how much one vector “points” in the direction of another.

#### Formula:

\[
\mathbf{u} \cdot \mathbf{v} = \|\mathbf{u}\|\|\mathbf{v}\|\cos \theta
\]

### A1.3 Cross Product — Perpendicular Direction (3D Only)

The **cross product** gives a new vector that is _perpendicular_ to both \( \mathbf{u} \) and \( \mathbf{v} \).

## A2. Matrix Computations

Matrices describe transformations of space — scaling, rotation, projection, or shearing.

### A2.1 Matrix Multiplication — Composition of Transformations

Matrix multiplication is **function composition in disguise**.

### A2.2 Determinant and Inverse — Volume and Reversibility

The **determinant** tells how much a transformation **scales volume** and whether it **flips orientation**.

### A3. Eigenvalues and Eigenvectors — The DNA of a Transformation

Eigenvectors are the **directions that don’t rotate** under a matrix transformation — they only scale.

### A4. Projection — Decomposing Vectors

Projection splits a vector into **parallel** and **perpendicular** parts relative to another vector or subspace.

### A5. Least Squares — Best Approximation in Overdetermined Systems

The **least squares** solution finds the vector \( \mathbf{x} \) that minimizes the squared error between \( A\mathbf{x} \) and \( \mathbf{b} \).