Numerical Stability

Numerical stability is about whether an algorithm preserves useful accuracy under finite-precision arithmetic. Two formulas can be algebraically identical but behave differently when exponentials overflow, nearly equal numbers subtract, or a matrix is ill-conditioned.

Defining math

For softmax,

$$\mathrm{softmax}(z{)}_i=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}.$$

Subtracting a constant $c$ from every logit leaves the result unchanged:

$$\frac{e^{z_i-c}}{{\sum }_j{e}^{z_j-c}}=\frac{e^{z_i}}{{\sum }_j{e}^{z_j}}.$$

Choosing $c={\max }_j{z}_j$ prevents the largest exponent from exceeding $1$. The same idea underlies stable log-sum-exp and stable cross-entropy implementations. For matrices, condition numbers and singular values from matrix decompositions describe how much input error can be amplified.

Executed demo

This snippet contrasts a naive softmax on very large logits with the max-shifted stable version that avoids overflow.

import numpy as np
 
z = np.array([1000., 1001., 1002.])
naive = np.exp(z) / np.exp(z).sum()
stable = np.exp(z-z.max()) / np.exp(z-z.max()).sum()
print("naive_softmax", naive)
print("stable_softmax", np.round(stable, 6))
print("finite_stable", np.isfinite(stable).all())

Observed output:

naive_softmax [nan nan nan]
stable_softmax [0.090031 0.244728 0.665241]
finite_stable True

The naive expression overflows and prints [nan nan nan] because exponentials near $e^{1000}$ exceed floating-point range. Subtracting the maximum before exponentiating returns finite probabilities, [0.090031,0.244728,0.665241], with the same mathematical softmax value.

Caveats

Stability fixes should preserve the target computation, not silently change it. Clipping probabilities, adding epsilons, or switching precision can mask bugs if the altered objective no longer matches the intended optimization problem.

References

• SciPy documentation: scipy.special.softmax
• NumPy documentation: numpy.linalg.cond

Nav

Section — Mathematical Foundations

← Stochastic Gradient Descent Information Theory →