Image Representation

Image representation defines the contract between visual data and algorithms: pixel grid, channel semantics, dtype, scale, metadata, patches, feature maps, or embeddings. Most failures in image classification, classical image processing, and vision transformers become harder to debug when this contract is implicit.

Tensors and normalization

A common tensor representation is $X\in {\mathbb{R}}^{C\times H\times W}$ or batched $X\in {\mathbb{R}}^{N\times C\times H\times W}$. Normalization maps integer pixels into numeric ranges suitable for optimization:

$$x_{c,u,v}^{\prime }=\frac{x_{c,u,v}/255-{\mu }_c}{{\sigma }_c}.$$

Patch-based models reshape an image into $P$ flattened patches $X_p\in {\mathbb{R}}^{P\times (C{p}_h{p}_w)}$ before projection into tokens.

Worked example

This snippet converts an image array into channel-first tensor form, extracts flattened patches, and reports channel means and a patch summary.

import numpy as np
 
rgb = np.arange(3 * 4 * 4, dtype=np.uint8).reshape(3, 4, 4)
chw = rgb.astype("float32") / 255.0
patches = chw.reshape(3, 2, 2, 2, 2).transpose(1, 3, 0, 2, 4).reshape(4, -1)
print("tensor_shape", tuple(chw.shape), "dtype", str(chw.dtype), "range", (round(float(chw.min()),3), round(float(chw.max()),3)))
print("patch_matrix_shape", patches.shape)
print("channel_means", np.round(chw.mean(axis=(1,2)), 3).tolist())
print("first_patch_sum", round(float(patches[0].sum()), 3))

Observed output:

tensor_shape (3, 4, 4) dtype float32 range (0.0, 0.184)
patch_matrix_shape (4, 12)
channel_means [0.028999999165534973, 0.09200000017881393, 0.1550000011920929]
first_patch_sum 0.871

The same 4-by-4 image becomes four 12-value patch vectors. That representation is natural for a vision transformer, while a CNN architecture would preserve local spatial neighborhoods through convolution.

Caveats

RGB/BGR swaps, missing alpha handling, lossy resizing, and wrong dtype ranges can silently poison a pipeline. Medical images add spacing, orientation, windowing, and sequence metadata; treating a voxel volume as an ordinary PNG stack can invalidate MRI segmentation.

References

• Torchvision transforms documentation
• Computer Vision: Algorithms and Applications, 2nd ed.

Nav

Section — Computer Vision

Classical Image Processing →

Learning path — Computer vision

← Computer Vision Object Detection →