Development of Attention and Transformers
Attention and transformers grew out of a concrete translation problem: early neural sequence-to-sequence systems could map one sequence to another, but they compressed the whole source sentence into a fixed vector. That made long-range information hard to preserve even when recurrent neural networks and LSTMs were strong enough to train.
Verified chronology
| Year | Milestone | Why it followed |
|---|---|---|
| 2014 | Sutskever, Vinyals, and Le published “Sequence to Sequence Learning with Neural Networks,” an encoder-decoder LSTM translation model. | It showed that neural translation could be trained end to end, but the fixed-length encoded vector was still a narrow communication channel. |
| 2014/2015 | Bahdanau, Cho, and Bengio introduced neural machine translation with learned soft alignment. | The decoder no longer had to rely only on one compressed sentence vector; at each step it could attend to relevant source positions. |
| 2017 | Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin introduced the Transformer in “Attention Is All You Need.” | If attention was the useful routing mechanism, self-attention could replace recurrence in the main sequence operation and make training much more parallel. |
| 2018 | Devlin, Chang, Lee, and Toutanova introduced BERT as a bidirectional Transformer encoder for language understanding. | Once self-attention made contextual representation scalable, masked-token pretraining turned unlabeled text into supervision for BERT-style encoders. |
| 2018-2020 | GPT-style decoder-only Transformers scaled autoregressive next-token prediction from GPT to GPT-3. | The same transformer block, with a causal mask, became a general-purpose generator for language-model architectures. |
Historical mechanism
The decisive mechanism was not “more layers” by itself. It was replacing a serial hidden-state bottleneck with attention: content-based lookup from queries to keys and values. Bahdanau attention first used this idea across encoder and decoder states, making alignment differentiable and task-driven. The Transformer then made every token exchange information with every other token through self-attention, followed by feed-forward layers, residual connections, normalization, and positional information.
That change altered the scaling path. Recurrent models process tokens in order, so training over long sequences is hard to parallelize. Transformers expose the whole sequence to matrix operations, so hardware can trade memory for throughput. BERT used bidirectional self-attention when the goal was representation and classification; GPT-style decoder-only transformers used causal self-attention when the goal was generation without seeing future tokens.
The historical caveat is that attention did not remove sequence modeling difficulties; it moved them. Positional encoding, masking, context length, data scale, and evaluation became central. The attention map is a routing computation, not a guaranteed explanation of a model’s answer.
References
- Sutskever et al., 2014, Sequence to Sequence Learning with Neural Networks
- Bahdanau et al., 2014, Neural Machine Translation by Jointly Learning to Align and Translate
- Vaswani et al., 2017, Attention Is All You Need
- Devlin et al., 2018, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
- Brown et al., 2020, Language Models are Few-Shot Learners
Nav