Alternating Least Squares

Alternating least squares trains matrix factorization by freezing one side of the model and solving the other side exactly. With item factors fixed, each user factor is a small ridge regression; with user factors fixed, each item factor is the symmetric ridge problem. This is why ALS is a natural fit for sparse utility matrices and distributed computation.

The ALS objective

For explicit ratings, ALS minimizes the usual observed-entry objective

$$\sum _{(u,i)\in \Omega }(r_{ui}-p_u^{\top }{q}_i{)}^2+\lambda (\Vert p_u{\Vert }_2^2+\Vert q_i{\Vert }_2^2).$$

Holding $Q$ fixed, user $u$ solves

$$p_u=(Q_{{\Omega }_u}^{\top }{Q}_{{\Omega }_u}+\lambda I{)}^{-1}{Q}_{{\Omega }_u}^{\top }{r}_u.$$

Weighted matrix factorization inserts a confidence matrix into the same normal equation, which is the common ALS form for implicit feedback.

Worked example

This snippet alternates user and item least-squares updates on an observed rating matrix and reports how RMSE and user-0 predictions change.

import numpy as np
rng = np.random.default_rng(4)
R = np.array([[5., 4., np.nan, 1.], [4., np.nan, 1., 1.],
              [1., 1., 5., 4.], [np.nan, 1., 4., 5.]])
obs, k, lam = ~np.isnan(R), 2, 0.1
P = 0.1 * rng.normal(size=(4, k)); Q = 0.1 * rng.normal(size=(4, k))
def rmse(): return np.sqrt(np.mean((R[obs] - (P @ Q.T)[obs]) ** 2))
print("initial_rmse", round(float(rmse()), 3))
for _ in range(8):
    for u in range(4):
        idx = np.where(obs[u])[0]
        P[u] = np.linalg.solve(Q[idx].T @ Q[idx] + lam*np.eye(k), Q[idx].T @ R[u, idx])
    for i in range(4):
        idx = np.where(obs[:, i])[0]
        Q[i] = np.linalg.solve(P[idx].T @ P[idx] + lam*np.eye(k), P[idx].T @ R[idx, i])
print("final_rmse", round(float(rmse()), 3))
print("pred_user0", np.round((P @ Q.T)[0], 2).tolist())

Observed output:

initial_rmse 3.338
final_rmse 0.279
pred_user0 [5.01, 3.96, 0.97, 0.98]

The closed-form half-steps rapidly recover the two taste blocks. Compared with Funk SVD, ALS is less sensitive to SGD step size but requires solving many small linear systems.

Caveats

ALS is still optimizing logged observations, so it inherits exposure bias and cold-start gaps. It can overfit rare users unless regularization increases for small $|{\Omega }_u|$. For ranking-heavy products, validate ALS with top-k evaluation, not only reconstruction RMSE.

References

• Koren, Bell, and Volinsky, 2009, Matrix Factorization Techniques for Recommender Systems
• Hu, Koren, and Volinsky, 2008, Collaborative Filtering for Implicit Feedback Datasets

Nav

Section — Recommendation Systems and Personalization

← Funk SVD Weighted Matrix Factorization →