Content-Based Image Retrieval

Content-based image retrieval represents images as feature vectors and returns nearby vectors for a query image. In recommendation systems, it is often a retrieval source for visually similar products, artworks, recipes, or media thumbnails before ranking applies personalization and constraints.

Image similarity scoring

Given an image encoder $f(\cdot )$ and query image $q$, retrieve items by

$$s(q,i)=\frac{f(q{)}^{\top }f(i)}{\Vert f(q)\Vert \Vert f(i)\Vert }.$$

At scale the vectors are stored in vector indexes, similar to dense retrieval.

Worked example

This snippet computes cosine similarity between a query image descriptor and candidate descriptors, then returns the nearest image index.

import numpy as np
from sklearn.metrics.pairwise import cosine_similarity
emb = np.array([[.9,.1,.1], [.85,.05,.2], [.1,.9,.2], [.2,.1,.95]])
q = np.array([[.88,.08,.12]])
sim = cosine_similarity(q, emb)[0]
print("similarities", np.round(sim, 3).tolist())
print("nearest_image", int(np.argmax(sim)))

Observed output:

similarities [0.999, 0.995, 0.223, 0.342]
nearest_image 0

The query retrieves the nearest visual vector. Image-based recommendation adds user preference or collaborative scores so the result is not merely a near-duplicate search.

In the example, items 0 and 1 are almost collinear with the query vector, so their cosine scores are near 1. Items 2 and 3 point toward different feature dimensions, so they are much less similar even if their raw vector magnitudes were comparable. This is why embedding normalization and nearest-neighbor indexing are part of the retrieval contract, not incidental implementation details.

Caveats

Visual similarity can overemphasize color, background, camera angle, or brand presentation. A user may want complements rather than substitutes. Safety filters, inventory constraints, and deduplication belong in the downstream retrieval and ranking architecture.

References

• He and McAuley, 2015, VBPR: Visual Bayesian Personalized Ranking from Implicit Feedback
• scikit-learn documentation: cosine_similarity

Nav

Section — Recommendation Systems and Personalization

← Image-Based Recommendation