Attention Is All You Need

Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin
Published: arXiv:1706.03762, first submitted 12 June 2017; source version v7 dated 2 August 2023
Primary source: arXiv abstract · Full paper

Takeaway

The paper’s central move is simple: for sequence-to-sequence tasks such as translation, replace recurrent and convolutional layers with attention. The resulting architecture, the Transformer, lets every input position directly gather relevant information from every other position while training positions in parallel. The decoder still generates output tokens one at a time. [Source: Abstract; §§1, 3]

Three points to remember:

1. Remove the training-time recurrence. An RNN must carry state through a chain; self-attention can process all positions of a training example together. [Source: §§1, 4]
2. Let each token look across the sequence. Attention computes a relevance-weighted mixture of information, and multiple heads can learn different relationships at once. [Source: §3.2]
3. The 2017 evidence was strong but narrow. The model set new reported translation results on two WMT 2014 tasks and transferred well to constituency parsing, but the experiments did not establish performance on today’s long-context, multimodal, or broad language-model workloads. [Source: §§6–7; the final clause is interpretation]

Visual 1 — Why the architectural change matters. The lower path removes the within-example sequential chain during training; it does not mean that autoregressive decoding produces all output tokens simultaneously. Original teaching visual based on §§1, 3, and 4.

Problem

The sequential bottleneck

Before this work, leading translation systems commonly used recurrent encoder–decoders, often with attention added on top. A recurrent layer computes the representation at position t from the state at position t − 1. That dependency prevents all positions within one example from being computed at once, which becomes especially costly for long sequences. Convolutional alternatives parallelized better, but connecting distant positions could still require several layers. [Source: §§1–2]

Long-distance relationships

Language often links words that are far apart. In a recurrent network, information may cross a path proportional to sequence length; in the paper’s comparison, unrestricted self-attention connects any two positions through a constant number of sequential operations. The tradeoff is important: full self-attention costs O(n²d) work per layer and memory grows with all position pairs, so the advantage is clearest when sequence length n is smaller than representation width d. [Source: §4, Table 1]

Method

Attention as selective reading

For each position, the model forms a query—roughly, “what information do I need?” Every available position supplies a key, used for matching, and a value, the information to retrieve. The model compares the query with all keys, scales the dot products by 1/√dₖ, applies softmax to obtain weights, and returns a weighted sum of the values. The scaling is intended to keep large dot products from pushing softmax into regions with very small gradients. [Source: §3.2.1, Eq. 1]

Visual 2 — One attention operation. Larger circles represent larger learned weights, not measured values. Original teaching visual based on §3.2.1.

Multiple views at once

Instead of one attention calculation, multi-head attention projects queries, keys, and values into several learned subspaces, runs attention in parallel, concatenates the results, and projects them again. The paper uses eight heads in its base model. This gives the model multiple opportunities to capture relationships at different positions or in different representation subspaces, at a total computational cost similar to one full-width head. [Source: §3.2.2]

Encoder, decoder, and order

The encoder stacks self-attention with a small feed-forward network applied independently at every position. The decoder adds two controls: it attends to encoder outputs, and it masks future target positions so a prediction cannot peek ahead. Residual connections and layer normalization wrap the sublayers. Because attention alone has no built-in notion of order, the model adds sinusoidal positional encodings to token embeddings; learned position embeddings gave nearly identical development results in the reported comparison. [Source: §§3.1, 3.3, 3.5; Table 3 row E]

Visual 3 — The Transformer’s information flow. The triangular motif represents causal masking: later target positions are unavailable. The drawing compresses repeated layers and is not a parameter-by-parameter reproduction of the paper’s Figure 1. Original teaching visual based on §§3.1–3.5.

Evidence and limits

What the experiments showed

The strongest evidence is machine translation on WMT 2014. The “big” Transformer reached 28.4 BLEU on English→German, versus 26.36 for the best prior ensemble listed in Table 2, and 41.8 BLEU on English→French, versus 41.29 for the best listed ensemble. The big models trained for 300,000 steps over 3.5 days on eight NVIDIA P100 GPUs. The smaller base models trained for 100,000 steps in about 12 hours on the same number of GPUs. [Source: Abstract; §§5.2, 6.1; Table 2]

The datasets contained about 4.5 million English–German sentence pairs and 36 million English–French sentence pairs. As a transfer check, a four-layer Transformer reached 91.3 F1 on WSJ-only English constituency parsing and 92.7 F1 with semi-supervised data. It was competitive, but not the best number in every parsing setting. [Source: §§5.1, 6.3; Table 4]

What remains uncertain

These results come from two translation benchmarks and one parsing task, with 2017-era tokenization, hardware, baselines, and tuning. BLEU is an automatic overlap metric, not a direct measure of factuality, safety, or human preference. The paper does not test very long sequences, memory use at modern context lengths, multilingual coverage beyond the reported pairs, robustness, or deployment latency. Its own conclusion names local attention for large inputs and less-sequential generation as future work. [Source: §7; benchmark interpretation clearly separated]

The ablations are useful but limited: a single head was 0.9 development BLEU below the best head setting, learned and sinusoidal position representations were almost tied, and larger models performed better. These comparisons vary selected design choices rather than proving which component alone causes the final test-set gains. [Source: §6.2, Table 3]

Practical meaning

When the idea helps

The paper offers a durable design pattern: when many items need context from one another, replace a forced step-by-step state update with direct, learned routing among items. For product teams, the practical gain is not “attention solves language,” but that parallelizable training can make larger experiments feasible while short information paths help the model combine distant context. This is an interpretation of the reported architecture and experiments, not a claim tested across all products.

Questions before using it

• Is the sequence short enough that all-pairs attention fits the latency and memory budget?
• Must generation remain autoregressive, and will decoding speed dominate the user experience?
• Do the evaluation set and metric reflect real user quality, including failures that BLEU cannot see?
• Which parts of the gain survive under the intended data size, hardware, language, and context length?

Verification guide. Check the official title and abstract on the arXiv record; the architecture and attention equations in §§3.1–3.5; the complexity comparison in §4 and Table 1; training data and hardware in §§5.1–5.2; translation numbers in §6.1 and Table 2; ablations in §6.2 and Table 3; parsing in §6.3 and Table 4; and the authors’ stated future work in §7.