Orthogonality

Orthogonality means two directions have zero dot product. It is the algebraic version of perpendicularity and is what lets projections, SVD, QR decomposition, and PCA separate variation into non-overlapping directions.

Defining math

Vectors $x,y\in {\mathbb{R}}^d$ are orthogonal when

$$x^{\top }y=0.$$

Here $x^{\top }y$ is the dot product. A zero dot product means the signed component of $x$ in the direction of $y$ cancels out, so neither vector carries length along the other direction.

A matrix $Q$ has orthonormal columns when

$$Q^{\top }Q=I.$$

The columns of $Q$ are the basis vectors, and $I$ is the identity matrix. Diagonal ones mean each column has unit length; off-diagonal zeros mean different columns are mutually orthogonal.

Then multiplication by $Q$ preserves Euclidean lengths inside its column space:

$$\Vert Qz{\Vert }_2^2=z^{\top }{Q}^{\top }Qz=\Vert z{\Vert }_2^2.$$

That length preservation is why orthogonal bases are numerically convenient in matrix decompositions and why residuals in least squares are orthogonal to fitted directions.

Worked example

The vectors $u=(1,1,0)$ and $v=(1,-1,0)$ are orthogonal because their dot product cancels:

$$u^{\top }v=1\cdot 1+1\cdot (-1)+0\cdot 0=0.$$

Each has length $\sqrt{2}$, so normalizing gives the orthonormal pair $\hat{u}=\frac{1}{\sqrt{2}}(1,1,0)$ and $\hat{v}=\frac{1}{\sqrt{2}}(1,-1,0)$. Stacking them as columns of $Q=[\hat{u}\hat{v}]$,

$$Q^{\top }Q=\left[\begin{array}{cc}{\hat{u}}^{\top }\hat{u}& {\hat{u}}^{\top }\hat{v}\\ {\hat{v}}^{\top }\hat{u}& {\hat{v}}^{\top }\hat{v}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]=I,$$

where the unit diagonal comes from the normalization and the zero off-diagonal from orthogonality. A QR decomposition of any two independent columns produces exactly such an orthonormal basis automatically.

Caveats

Orthogonality depends on the inner product. Standard Euclidean orthogonality may be the wrong geometry when features have different units or correlated noise; then norms and distances or whitening need to define the metric first.

References

• MIT OpenCourseWare: 18.06 Linear Algebra

Nav

Section — Mathematical Foundations

← Rank Norms and Distances →