Mathematical Foundations
Mathematical foundations supplies the notation and mechanisms used throughout the wiki. Linear algebra explains data as vectors, matrices, projections, ranks, and decompositions. Calculus and optimization explain how models are trained. Information theory explains losses, uncertainty, and distribution mismatch.
The goal is not abstract proof for its own sake. Each page answers a modeling question: what does this operation preserve, what can go wrong numerically, and how does it appear in machine learning systems?
Knowledge map
The section has three trunks — linear algebra, calculus/optimization, and information theory — that recur across every later area. Arrows point from a prerequisite to what it enables.
flowchart TD LA[Linear Algebra] --> Det[Determinants and Rank] Det --> Decomp[Eigen and SVD Decompositions] Decomp --> Graph[Graph Laplacians] Decomp --> LowRank[Low-Rank Approximation] LA --> Calc[Calculus and Gradients] Calc --> Opt[Optimization] Opt --> GD[Gradient and Stochastic Descent] GD --> NumStab[Numerical Stability] Info[Information Theory] --> Losses[Entropy, Cross-Entropy, KL, Mutual Information]
Reading path
Read linear algebra first, then calculus and optimization, then information theory.
- Linear Algebra: the overview of vectors, matrices, and the operations that follow.
- Vectors and Matrices: the basic objects and how shapes compose.
- Matrix Multiplication: the core operation behind linear maps and layers.
- Determinants: signed volume scaling for square matrices.
- Rank: how many independent directions a matrix actually spans.
- Orthogonality: perpendicular directions, projections, and orthonormal bases.
- Norms and Distances: measuring size and similarity of vectors.
- Eigenvalues and Eigenvectors: directions a matrix only stretches.
- Graph Laplacian: degree-minus-adjacency matrices for graph structure.
- Matrix Decompositions: factorizations that expose structure.
- Singular Value Decomposition: the decomposition behind embeddings and compression.
- Low-Rank Approximation: keeping the dominant structure and discarding noise.
- Calculus: derivatives as the language of change.
- Gradients: multivariate derivatives that point training in a direction.
- Jacobians and Hessians: first- and second-order derivative matrices.
- Optimization: finding parameters that minimize a loss.
- Convex Optimization: the well-behaved case with a single global optimum.
- Constrained Optimization: optimizing subject to equalities and inequalities.
- Gradient Descent: the workhorse iterative optimizer.
- Stochastic Gradient Descent: mini-batch updates that scale to large data.
- Numerical Stability: avoiding overflow, underflow, and catastrophic cancellation.
- Information Theory: quantifying uncertainty and information.
- Entropy: the average surprise of a distribution.
- Cross Entropy: the standard classification training loss.
- KL Divergence: a directed measure of distribution mismatch.
- Mutual Information: shared information between variables.
Connections
- Probability and Statistics builds estimation and inference on this notation.
- Classical Machine Learning and Deep Learning turn these operations into models and training loops.
Nav
Learning path — Foundations