Linear Algebra: The Geometry of Space and Transformation

Technology: Math
Skill: Mathing

Overview

Linear algebra is the language of structure and transformation.

While calculus studies change, linear algebra studies the spaces where change happens — how vectors, planes, and higher-dimensional structures interact, align, and evolve.

It underlies computer graphics, quantum mechanics, data science, and machine learning — wherever quantities live in multidimensional space.

This document moves from vectors and matrices to orthogonality, eigenvalues, and the singular value decomposition.

The appendix then shows how to compute each of these concepts step by step.

1. Vectors and Spaces — The Building Blocks of Structure

1.1 Vectors as Geometric Objects

A vector is both direction and magnitude.

[ v = \begin{bmatrix} v_x \ v_y \end{bmatrix}, \quad \mathbf{v} = \begin{bmatrix} v_x \ v_y \ v_z \end{bmatrix} ]

The length of a vector is its Euclidean norm:

[ |\mathbf{v}| = \sqrt{v_x^2 + v_y^2 + v_z^2} ]

The unit vector in the same direction is:

[ hat{\mathbf{v}} = \frac{\mathbf{v}}{|\mathbf{v}|} ]

Geometrically, vectors describe positions, velocities, forces, or any quantity that has both magnitude and direction.

1.2 Vector Operations

Addition:

[ \mathbf{u} + \mathbf{v} = \begin{bmatrix} u_x + v_x \ u_y + v_y \end{bmatrix} ]

Combines directions head-to-tail.

Scalar Multiplication:

[ c\mathbf{v} = \begin{bmatrix} cv_x \ cv_y \end{bmatrix} ]

Scales magnitude, preserves direction.

Dot Product:

[ \mathbf{u} \cdot \mathbf{v} = u_xv_x + u_yv_y + u_zv_z = |\mathbf{u}||\mathbf{v}|\cos \theta ]

Measures alignment.

• If = 0, vectors are orthogonal.
• If positive, acute angle; if negative, obtuse.

Cross Product (3D only):

[ \mathbf{u} \times \mathbf{v} = \begin{bmatrix} u_yv_z - u_zv_y \ u_zv_x - u_xv_z \ u_xv_y - u_yv_x \end{bmatrix} ]

Gives a vector perpendicular to both — defines orientation and area.

1.3 Vector Spaces and Subspaces

A vector space is a set of vectors closed under addition and scalar multiplication.

The span of {v1,v2} is all combinations:

[ \text{Span}{\mathbf{v}_1, \mathbf{v}_2} = { a\mathbf{v}_1 + b\mathbf{v}_2 \mid a,b \in \mathbb{R} } ]

A subspace is any subset that’s also a vector space (e.g., a line or plane through the origin).

A basis is a minimal set of linearly independent vectors that span the space.

The dimension equals the number of basis vectors.

2. Matrices — Linear Transformations and Systems

2.1 Matrices as Transformations

A matrix encodes a linear transformation:

[ A\mathbf{x} = \mathbf{b} ]

If A is m×n, it maps ( \mathbb{R}^n \to \mathbb{R}^m ).

Each column of A shows how a basis vector transforms.

Matrices stretch, rotate, reflect, and shear space.

Example:

[ A = \begin{bmatrix} 2 & 0 \ 0 & 3 \end{bmatrix} \Rightarrow A\begin{bmatrix} 1 \ 1 \end{bmatrix} = \begin{bmatrix} 2 \ 3 \end{bmatrix} ]

Space is stretched by 2 along x and 3 along y.

2.2 Matrix Operations

If det(A) = 0, the transformation collapses space — no inverse.

2.3 Solving Systems of Equations

Systems of linear equations can be written compactly as ( A\mathbf{x} = \mathbf{b} ).

If det(A)≠0, the unique solution is:

[ \mathbf{x} = A^{-1}\mathbf{b} ]

If A is not invertible or overdetermined, approximate with least squares:

[ A^TA\mathbf{x} = A^T\mathbf{b} ]

3. Orthogonality and Projections

3.1 Orthogonality

Two vectors are orthogonal if their dot product is zero:

[ \mathbf{u} \cdot \mathbf{v} = 0 ]

3.2 Projection

To find the component of a ( \mathbf{a} ) along ( \mathbf{b} ):

[ \text{proj}_{\mathbf{b}}(\mathbf{a}) = \frac{\mathbf{a}\cdot\mathbf{b}}{\mathbf{b}\cdot\mathbf{b}}\mathbf{b} ]

The residual (error) orthogonal to ( \mathbf{b} ) is:

[ \mathbf{a}_\perp = \mathbf{a} - \text{proj}_{\mathbf{b}}(\mathbf{a}) ]

3.3 Orthonormal Bases

An orthonormal basis satisfies:

[ \mathbf{v}_i\cdot\mathbf{v}_j = \begin{cases} 1, & i=j \ 0, & i \neq j \end{cases} ]

Any vector ( \mathbf{x} ) decomposes uniquely as:

[ \mathbf{x} = \sum_i (\mathbf{x} \cdot \mathbf{v}_i) \mathbf{v}_i ]

4. Eigenvalues and Eigenvectors — Directions That Stay Put

4.1 Definition

For a square matrix A:

[ A\mathbf{v} = \lambda\mathbf{v} ]

• ( \mathbf{v} ) is an eigenvector (direction stays the same)
• ( \lambda ) is an eigenvalue (scaling factor)

They satisfy the characteristic equation:

[ \det(A - \lambda I) = 0 ]

4.2 Interpretation

• Eigenvectors are invariant directions of transformation.
• Eigenvalues describe how much those directions are stretched or compressed.
• If ( |\lambda| < 1 ), repeated applications shrink the vector (stability).
• If ( |\lambda| > 1 ), the system amplifies along that direction.

5. Diagonalization and the Spectral Theorem

If a matrix has ( n ) independent eigenvectors:

[ A = PDP^{-1} ]

• ( P ) contains eigenvectors as columns
• ( D ) is diagonal with eigenvalues on the diagonal

Then powers of ( A ) are easy:

[ A^k = P D^k P^{-1} ]

If A is symmetric, all eigenvalues are real and eigenvectors can be chosen orthonormal.

This is the Spectral Theorem — the backbone of PCA and quantum mechanics.

6. Least Squares and Projections Revisited

When ( A\mathbf{x} = \mathbf{b} ) has no exact solution (overdetermined system), minimize the squared error:

[ \min_{\mathbf{x}} |A\mathbf{x} - \mathbf{b}|^2 ]

Setting derivative = 0 gives:

[ A^TA\mathbf{x} = A^T\mathbf{b} ]

7. Singular Value Decomposition (SVD)

Every matrix A (even non-square) can be decomposed as:

[ A = U\Sigma V^T ]

where:

• U — orthogonal basis for outputs
• V — orthogonal basis for inputs
• ( \Sigma ) — diagonal with nonnegative singular values ( \sigma_i )

8. Determinants and Volume

For a square matrix:

[ \det(A) ]

measures how A scales volume and changes orientation.

9. The Unified Picture

Linear algebra defines how space transforms, how data decomposes, and how systems align.

It is the structural engine of all continuous mathematics.

Appendix: Linear Algebra in Practice

The appendix turns abstract concepts into computable procedures — showing how to perform, interpret, and reason through each operation in linear algebra.

Each subsection connects the symbolic process to its geometric intuition.

A1. Vector Computations

Vectors are the atoms of linear algebra. Every operation — matrix multiplication, projection, eigenanalysis — ultimately manipulates vectors.

A1.1 Magnitude and Unit Vector

The magnitude of a vector is its length in space.

The unit vector gives the same direction but normalizes length to 1.

Example:

[ \mathbf{v} = (3,4) ]

Compute its magnitude:

[ |\mathbf{v}| = \sqrt{3^2 + 4^2} = 5 ]

Normalize:

[ \hat{\mathbf{v}} = \frac{\mathbf{v}}{|\mathbf{v}|} = \left(\frac{3}{5}, \frac{4}{5}\right) ]

A1.2 Dot Product — Measuring Alignment

The dot product measures how much one vector “points” in the direction of another.

Formula:

[ \mathbf{u} \cdot \mathbf{v} = |\mathbf{u}||\mathbf{v}|\cos \theta ]

A1.3 Cross Product — Perpendicular Direction (3D Only)

The cross product gives a new vector that is perpendicular to both ( \mathbf{u} ) and ( \mathbf{v} ).

A2. Matrix Computations

Matrices describe transformations of space — scaling, rotation, projection, or shearing.

A2.1 Matrix Multiplication — Composition of Transformations

Matrix multiplication is function composition in disguise.

A2.2 Determinant and Inverse — Volume and Reversibility

The determinant tells how much a transformation scales volume and whether it flips orientation.

A3. Eigenvalues and Eigenvectors — The DNA of a Transformation

Eigenvectors are the directions that don’t rotate under a matrix transformation — they only scale.

A4. Projection — Decomposing Vectors

Projection splits a vector into parallel and perpendicular parts relative to another vector or subspace.

A5. Least Squares — Best Approximation in Overdetermined Systems

The least squares solution finds the vector ( \mathbf{x} ) that minimizes the squared error between ( A\mathbf{x} ) and ( \mathbf{b} ).