Decoder-Only Transformers

Decoder-only transformers are causal sequence models: at position $i$, the hidden state may use tokens $1,\dots ,i$ but not future tokens. That makes them natural for language modelling, completion, chat, and generative summarization. They share attention machinery with bert-style encoders, but the mask changes what information can flow.

Causal self-attention

Causal self-attention applies a triangular mask before softmax:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }+M}{\sqrt{d_k}}\right)V,$$

where $M_{ij}=0$ if $j\le i$ and $M_{ij}=-\infty$ if $j>i$. The model is trained with next-token cross-entropy:

$$\mathcal{L}=-\sum _i\log P_{\theta }(t_i\mid t_{<i}).$$

Tokenization fixes the sequence being predicted, so both perplexity and latency depend on tokenizer choice.

Worked example

The code constructs one attention head by hand, applies the upper-triangular causal mask, and then checks that masked future positions receive zero probability after softmax.

import math, torch
 
torch.manual_seed(7)
X = torch.randn(4, 3)
Wq, Wk, Wv = torch.randn(3, 3), torch.randn(3, 3), torch.randn(3, 2)
Q, K, V = X @ Wq, X @ Wk, X @ Wv
scores = Q @ K.T / math.sqrt(3)
mask = torch.triu(torch.ones(4, 4, dtype=torch.bool), diagonal=1)
weights = torch.softmax(scores.masked_fill(mask, float("-inf")), dim=-1)
context = weights @ V
print("attention_weights", torch.round(weights, decimals=3).tolist())
print("future_weight_sum", round(float(weights[0,1:].sum() + weights[1,2:].sum() + weights[2,3:].sum()), 6))
print("context_last", torch.round(context[-1], decimals=3).tolist())

Observed output:

attention_weights [[1.0, 0.0, 0.0, 0.0], [0.9010000228881836, 0.0989999994635582, 0.0, 0.0], [0.796999990940094, 0.04899999871850014, 0.15399999916553497, 0.0], [0.0820000022649765, 0.25, 0.13099999725818634, 0.5370000004768372]]
future_weight_sum 0.0
context_last [-0.27000001072883606, -0.9359999895095825]

The first row is [1.0, 0.0, 0.0, 0.0] because position 0 can attend only to itself. Row 2 assigns weight to positions 0, 1, and 2 but zero to position 3, and the printed future_weight_sum 0.0 confirms that every illegal future edge was removed by the mask before softmax.

Caveats

A missing or misaligned causal mask leaks answers during training or evaluation. Long contexts stress memory through the attention matrix and the key-value cache. Decoding settings change outputs, so compare systems with fixed prompt format, tokenizer, stop rules, and sampling parameters.

References

• Vaswani et al., Attention Is All You Need
• PyTorch documentation: MultiheadAttention
• Jurafsky and Martin, Speech and Language Processing, 3rd ed. draft

Nav

Section — Natural Language Processing

← BERT-Style Encoders Text Classification →

Learning path — Natural language processing

← Embeddings Text Classification →