Content-Based Image Retrieval

Content-based image retrieval finds images by visual similarity rather than only text metadata. The core dependency is feature extraction: the retrieval system can only rank by what the descriptor or embedding preserves. It connects directly to vector search in vector indexes.

Ranking by embedding similarity

Given indexed image embeddings $v_i=\varphi (x_i)$ and a query embedding $q=\varphi (x_q)$, rank by cosine similarity

$$s_i=\frac{v_i^{\top }q}{\Vert v_i{\Vert }_2\Vert q{\Vert }_2}.$$

Here $v_i$ is the stored embedding for image $i$, $q$ is the query embedding, and $s_i$ is the similarity score used for ranking. The denominator normalizes vector lengths, so ranking depends on direction in embedding space rather than raw magnitude.

Classical systems may use color histograms or local descriptors; modern systems often use pretrained visual embeddings from self-supervised visual learning or multimodal models.

Worked example

The code below builds four toy image embeddings and ranks them by cosine similarity to a query. It illustrates the retrieval contract without needing a full image model: nearest neighbors are selected by embedding geometry.

import numpy as np
 
vecs = np.array([[1,0,0], [.9,.1,0], [0,1,0], [.2,.8,0]], float)
query = np.array([[.8,.2,0.]])
vecs = vecs / np.linalg.norm(vecs, axis=1, keepdims=True)
q = query / np.linalg.norm(query)
sims = (vecs @ q.T).ravel()
order = np.argsort(-sims)
print("cosine_scores", np.round(sims, 3).tolist())
print("ranked_ids", order.tolist())

Observed output:

cosine_scores [0.97, 0.991, 0.243, 0.471]
ranked_ids [1, 0, 3, 2]

The query is closest to item 1 because its direction in embedding space is most similar, even though item 0 has a larger first coordinate.

Caveats

Nearest neighbors expose embedding bias quickly: results may cluster by background, lighting, crop style, or digitization source rather than semantic content. A museum archive should evaluate retrieval alongside document image analysis and field extraction, because visual similarity and catalog-entity similarity are different signals.

References

• Computer Vision: Algorithms and Applications, 2nd ed.
• scikit-learn documentation: Nearest Neighbors

Nav

Section — Computer Vision

← Self Supervised Visual Learning OCR Pipelines →