Tokenization

Tokenization chooses the units consumed by NLP models. A word tokenizer is natural for sparse text classification; subword or byte tokenizers are essential for bert-style encoders, decoder-only transformers, and open-vocabulary language modelling. The tokenizer defines vocabulary size, sequence length, and which errors are even representable.

Granularity of the unit

Tokenizers differ in how coarse their units are, which trades vocabulary size against sequence length:

Granularity	Unit	Trade-off
Word	whole words	short sequences, but a large brittle vocabulary and out-of-vocabulary gaps
Subword (BPE)	frequent character chunks	balances vocabulary size against out-of-vocabulary coverage
Byte / character	bytes or characters	no out-of-vocabulary words, but much longer sequences

Subword schemes such as byte-pair encoding are the common default because they interpolate between the two extremes.

Byte-pair encoding

Byte-pair encoding style tokenization starts with characters and repeatedly merges the most frequent adjacent pair:

$$(a^{\text{*}},b^{\text{*}})=\arg \underset{(a,b)}{\max }\mathrm{count}(a,b).$$

After learning merges, tokenization usually applies them greedily to new text. The model then embeds token ids, so a different tokenizer changes the input distribution even if the visible sentence is unchanged.

Worked example

This small BPE loop learns merges from a toy corpus, so the output shows how repeated character pairs become reusable subword tokens.

import numpy as np
from collections import Counter
 
np.random.seed(7)
corpus = ["low lower lowest", "newer wider lower"]
vocab = [tuple(list(w) + ["</w>"]) for sent in corpus for w in sent.split()]
 
def pair_counts(words):
    c = Counter()
    for w in words:
        for a, b in zip(w, w[1:]):
            c[(a, b)] += 1
    return c
 
merges = []
for _ in range(4):
    pair, count = pair_counts(vocab).most_common(1)[0]
    merges.append((pair, count))
    merged = "".join(pair)
    new_vocab = []
    for w in vocab:
        out, i = [], 0
        while i < len(w):
            if i < len(w) - 1 and (w[i], w[i + 1]) == pair:
                out.append(merged); i += 2
            else:
                out.append(w[i]); i += 1
        new_vocab.append(tuple(out))
    vocab = new_vocab
print("merges", merges)
print("lower_tokens", list(vocab[1]))
print("vocab_size_after", len(set(t for w in vocab for t in w)))

Observed output:

merges [(('l', 'o'), 4), (('lo', 'w'), 4), (('e', 'r'), 4), (('er', '</w>'), 4)]
lower_tokens ['low', 'er</w>']
vocab_size_after 10

The learned pieces reuse low across low, lower, and lowest, reducing unknown-word pressure while keeping sequence length shorter than pure characters.

Caveats

Token counts are not word counts. A rare name, code identifier, or OCR error may explode into many subwords and be truncated away. Tokenization must be inspected together with text preprocessing, embeddings, and downstream evaluation, because a clean aggregate score can hide failures on languages, names, or technical strings.

References

• Manning, Raghavan, and Schutze, Introduction to Information Retrieval: Tokenization
• Jurafsky and Martin, Speech and Language Processing, 3rd ed. draft

Nav

Section — Natural Language Processing

← Text Preprocessing Embeddings →

Learning path — Natural language processing

← Natural Language Processing Embeddings →