Evaluation of Recommenders

Recommender evaluation asks whether ranked lists are useful, robust, and healthy for users and inventory. Accuracy metrics such as recall@k and NDCG are necessary, but they miss novelty, diversity, coverage, calibration, and long-term feedback loops.

Top-k ranking metrics

For a top-$k$ list $L_k$ and relevant set $G_u$,

$$Precision@k=\frac{|L_k\cap G_u|}{k},Recall@k=\frac{|L_k\cap G_u|}{|G_u|}.$$

NDCG discounts hits by rank and normalizes by the ideal list. The same family appears in ranking and retrieval metrics.

Worked example

This snippet ranks items by predicted score and computes top-3 precision, recall, and NDCG against binary relevance labels.

import numpy as np
from sklearn.metrics import ndcg_score
y_true = np.array([[0,0,1,1,0]])
y_score = np.array([[.9,.2,.8,.4,.1]])
top3 = np.argsort(-y_score[0])[:3]
hits = y_true[0, top3].sum()
print("top3", top3.tolist())
print("precision_at_3", round(float(hits / 3), 3))
print("recall_at_3", round(float(hits / y_true.sum()), 3))
print("ndcg_at_3", round(float(ndcg_score(y_true, y_score, k=3)), 3))

Observed output:

top3 [0, 2, 3]
precision_at_3 0.667
recall_at_3 1.0
ndcg_at_3 0.693

Both relevant items appear in the top three, but one irrelevant item ranks first, so NDCG penalizes the ordering. Offline versus online evaluation decides whether this historical score predicts live behavior.

Caveats

Random train-test splits leak future behavior; time-based splits are usually more realistic. Missing interactions are not guaranteed negatives. Report segment metrics, coverage, and list quality, not only a single average score.

References

• Herlocker et al., 2004, Evaluating Collaborative Filtering Recommender Systems
• scikit-learn documentation: ndcg_score

Nav

Section — Recommendation Systems and Personalization

← Retrieval and Ranking Architectures Offline Versus Online Evaluation →

Learning path — Recommender systems

← SVD versus Matrix Factorization