BERT-Style Encoders

BERT-style encoders read an entire token sequence bidirectionally and produce contextual representations for tokens or pooled text. They are strong for sequence labelling, semantic textual similarity, classification, reranking, and feature extraction. Unlike decoder-only transformers, they are not trained to generate left-to-right completions.

Encoder versus decoder

The choice between a BERT-style encoder and a decoder-only transformer comes down to which direction attention may look, which decides what the model is good at:

	Encoder (BERT-style)	Decoder (GPT-style)
Attention	bidirectional — sees both sides	causal — past tokens only
Pretraining	predict masked tokens	predict the next token
Best for	understanding: labeling, classification, retrieval	generation: completion, chat

Bidirectional self-attention

Encoder self-attention has no causal future mask:

$$H^{(\ell +1)}=\mathrm{TransformerEncoderLayer}(H^{(\ell )}),$$

so token $i$ can attend to tokens on both sides. BERT-style pretraining commonly uses masked language modelling:

$${\mathcal{L}}_{\text{MLM}}=-\sum _{i\in M}\log P_{\theta }(t_i\mid t_{\setminus M}),$$

where $M$ is the set of masked positions. Tokenization defines those positions, and the resulting contextual embeddings can be fine-tuned or reused.

Worked example

This snippet computes bidirectional self-attention weights for token embeddings and shows that the first token can attend to later positions.

import math, torch
 
torch.manual_seed(7)
X = torch.randn(4, 3)
Wq, Wk, Wv = torch.randn(3, 3), torch.randn(3, 3), torch.randn(3, 2)
weights = torch.softmax((X @ Wq) @ (X @ Wk).T / math.sqrt(3), dim=-1)
print("row0_weights", torch.round(weights[0], decimals=3).tolist())
print("row0_future_mass_positions_1_to_3", round(float(weights[0,1:].sum()), 3))
print("all_rows_sum", torch.round(weights.sum(dim=1), decimals=3).tolist())

Observed output:

row0_weights [0.257999986410141, 0.019999999552965164, 0.04399999976158142, 0.6769999861717224]
row0_future_mass_positions_1_to_3 0.742
all_rows_sum [1.0, 1.0, 1.0, 1.0]

Position 0 places most of its attention mass on later positions because no causal mask blocks them. That is useful for understanding tasks but invalid for next-token generation.

Caveats

Encoder outputs are sensitive to truncation, pooling choice, and domain mismatch. A classifier head can overfit annotation artifacts even when the base encoder is strong. For multilingual or noisy inputs, inspect tokenizer fragmentation and slice metrics before trusting pooled vectors.

References

• Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Vaswani et al., Attention Is All You Need
• PyTorch documentation: MultiheadAttention

Nav

Section — Natural Language Processing

← Language Modelling Decoder-Only Transformers →