Attention Is All You Need

The paper introduced the Transformer: an encoder-decoder architecture that replaces recurrent and convolutional sequence modeling with attention-based token mixing.

Metadata

Field	Value
Paper	Attention Is All You Need
Authors	Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin
Year	2017
Venue	NeurIPS 2017
arXiv	1706.03762
NeurIPS	Proceedings page
Status	verified

Question

Before this paper, strong sequence transduction systems were usually built around recurrent or convolutional encoder-decoder models, often with attention added on top. The question was whether recurrence and convolution were necessary for high-quality sequence modeling, or whether attention alone could do the sequence mixing.

The deeper architecture question is about sequence dependence. RNNs process tokens sequentially, CNNs process local windows with depth-dependent receptive fields, and attention lets every token directly address every other token in one layer. The paper tests whether that direct token-token interaction can become the main computational primitive.

Main Claim

The paper’s central claim is that a sequence transduction model can be built entirely from attention and feed-forward blocks, without recurrence or convolution, while improving translation quality and training parallelism on the reported machine translation benchmarks.

Narrowed claim:

$$\text{Transformer}\Rightarrow \text{strong WMT translation results under the paper’s training and evaluation protocol}$$

This is not the same as proving that recurrence or convolution are never useful. It shows that attention-only sequence modeling is a strong architecture class under the tested settings.

Architecture Contract

Item	Contract
Input	source token sequence and target prefix sequence
Output	next-token distribution over target vocabulary
Backbone	stacked encoder blocks and decoder blocks
Token mixing	self-attention in encoder and decoder; cross-attention from decoder to encoder
Position information	sinusoidal or learned positional encodings
Parallelism	encoder tokens and teacher-forced decoder tokens can be processed in parallel during training
Causal constraint	decoder self-attention masks future target tokens

The translation training objective is autoregressive over the target sequence:

$$\mathcal{L}=-\sum _{t=1}^{T_y}\log p_{\theta }(y_t\mid y_{<t},x)$$

where $x$ is the source sequence and $y_{<t}$ is the target prefix.

At inference time:

$$p_{\theta }(y\mid x)=\prod _{t=1}^{T_y}{p}_{\theta }(y_t\mid y_{<t},x)$$

The encoder builds source representations; the decoder repeatedly queries those representations while generating target tokens.

Method

The Transformer uses stacked encoder and decoder blocks. Each block combines:

• scaled dot-product attention;
• multi-head self-attention;
• encoder-decoder cross-attention in the decoder;
• position-wise feed-forward networks;
• residual connections;
• normalization;
• positional encoding.

The core attention formula is:

$$\mathrm{Attention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}\right)V$$

where $Q$ contains queries, $K$ contains keys, $V$ contains values, and $d_k$ is the key dimension. The scale factor reduces extreme dot-product magnitudes before softmax.

Multi-head attention projects the same input into several attention subspaces:

$${\mathrm{head}}_i=\mathrm{Attention}(Q{W}_i^Q,K{W}_i^K,V{W}_i^V)$$ $$\mathrm{MultiHead}(Q,K,V)=\mathrm{Concat}({\mathrm{head}}_1,\dots ,{\mathrm{head}}_h)W^O$$

Because the architecture has no recurrence, positional information is injected with positional encodings:

$${\mathrm{PE}}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{\mathrm{model}}}}\right)$$ $${\mathrm{PE}}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{\mathrm{model}}}}\right)$$

Encoder Block Walkthrough

An encoder layer starts from token states:

$$H^{(l-1)}\in {\mathbb{R}}^{T_x\times d_{\text{model}}}$$

Self-attention mixes source tokens:

$$A^{(l)}=\mathrm{MultiHead}(H^{(l-1)},H^{(l-1)},H^{(l-1)})$$

The paper uses residual connections and layer normalization around each sublayer:

$${\tilde{H}}^{(l)}=\mathrm{LayerNorm}(H^{(l-1)}+A^{(l)})$$

Then a position-wise feed-forward network is applied independently at each token:

$$\mathrm{FFN}(h)=\max (0,h{W}_1+b_1)W_2+b_2$$ $$H^{(l)}=\mathrm{LayerNorm}({\tilde{H}}^{(l)}+\mathrm{FFN}({\tilde{H}}^{(l)}))$$

The important separation is:

Sublayer	Mixes tokens?	Mixes channels?
self-attention	yes	yes, through projections
feed-forward network	no	yes
residual/normalization	no	stabilizes representation scale

This is why Transformer blocks are often described as alternating token mixing and channel mixing.

Decoder Block Walkthrough

The decoder has three sublayers.

Sublayer	Query	Key/Value	Constraint
masked self-attention	target prefix states	target prefix states	cannot attend to future target positions
cross-attention	target states	encoder source states	can attend to all source positions
feed-forward network	each target position	same target position	position-wise

The causal mask changes decoder self-attention:

$$\mathrm{MaskedAttention}(Q,K,V)=\mathrm{softmax}\left(\frac{Q{K}^{\top }}{\sqrt{d_k}}+M\right)V$$

where:

$$M_{ij}=\{\begin{array}{ll}0& j\le i\\ -\infty & j>i\end{array}$$

Cross-attention then lets each target position query the encoded source sequence:

$$\mathrm{CrossAttn}(H_y,H_x)=\mathrm{MultiHead}(Q=H_y,K=H_x,V=H_x)$$

This encoder-decoder structure is different from later encoder-only and decoder-only Transformer backbones.

Complexity and Inductive Bias

The paper compares self-attention, recurrent layers, and convolutional layers by path length and computational structure.

For sequence length $n$ and hidden dimension $d$, dense self-attention has:

$$O(n^{2}d)$$

time/multiply cost for attention interactions, while recurrent layers have sequential path length:

$$O(n)$$

and self-attention has maximum token-token path length:

$$O(1)$$

The tradeoff is direct global interaction versus quadratic sequence-length cost.

Architecture	Main Bias	Parallel Over Positions	Long-Range Path
RNN	temporal recurrence	no	long
CNN	locality and translation sharing	yes	grows with depth/dilation
Transformer	content-based global addressing	yes	short

This is the core reason the paper became more than a translation result: it changed the default assumption about how sequence positions should communicate.

Training Recipe

The architecture is not the only thing that matters. The reported results also depend on the training recipe.

Ingredient	Role
residual dropout	regularizes sublayers and embeddings
label smoothing	avoids overconfident target distributions
Adam with warmup schedule	stabilizes early optimization
byte-pair encoding	controls vocabulary and open-word handling
beam search	affects reported translation quality

The learning-rate schedule is part of the paper’s practical recipe:

$$\mathrm{lrate}=d_{\text{model}}^{-0.5}\cdot \min \left({\mathrm{step}}^{-0.5},\mathrm{step}\cdot {\mathrm{warmup}}^{-1.5}\right)$$

For this wiki, architecture conclusions should not ignore these recipe choices. A weaker training setup may make the same block look worse; a stronger setup may improve baselines too.

Evidence

Claim	Evidence in paper	Caveat
Attention-only encoder-decoder can outperform strong translation baselines	WMT 2014 English-German and English-French BLEU results	claim is under machine translation protocol
Transformer trains more parallelly than recurrent sequence models	architecture removes sequential recurrence across token positions	wall-clock depends on hardware and implementation
Multi-head attention and feed-forward blocks are useful components	ablations over heads, key/value dimensions, model size, and positional encoding	ablations are mostly within translation setup
Model transfers beyond translation	English constituency parsing experiments	not a broad proof of universal transfer

Ablation Reading

The most useful ablations are about which architectural pieces are necessary.

Ablation Axis	What it tests	Reading
number of heads	whether multiple attention subspaces help	too few heads can bottleneck relation types; too many reduce per-head dimension
key/value dimension	capacity and attention sharpness	smaller dimensions can reduce compute but lose quality
model size	depth/width scaling	improvement is not solely from attention form
positional encoding	fixed sinusoidal vs learned	position injection matters, but the paper does not settle all later position methods
feed-forward dimension	channel-mixing capacity	attention alone is not the whole architecture

The paper supports the Transformer block as a package: attention, FFN, residuals, normalization, position encoding, optimization, and enough scale.

Benchmark Card

Field	WMT translation setting
Task	sequence-to-sequence machine translation
Input/output unit	source sentence to target sentence
Main metric	BLEU
Main comparison	recurrent/convolutional sequence transduction baselines and prior state of the art
Generalization claim	translation quality under the benchmark’s train/test setup
Not directly tested	modern LLM pretraining, retrieval, tool use, multimodal reasoning, protein modeling

What Later Models Kept and Changed

Kept	Changed Later
scaled dot-product attention	sparse, local, linear, grouped-query, and flash attention variants
multi-head projection pattern	[[papers/architectures/gqa
residual block structure	pre-norm and other normalization placement choices
feed-forward sublayer	GELU, SwiGLU, gated FFNs, mixture-of-experts FFNs
positional signal	relative position, rotary embeddings, ALiBi, learned position schemes
encoder-decoder idea	encoder-only BERT and decoder-only GPT-style models

This matters because “Transformer” now names a family. The 2017 paper is the root, but many modern systems are not exact copies of the original encoder-decoder stack.

Implementation Notes

• Attention masks are part of the architecture contract; encoder padding masks, decoder causal masks, and cross-attention masks must be handled separately.
• Tensor shapes should be explicit: batches, heads, sequence length, and head dimension are easy sources of silent bugs.
• Positional encoding must be added or otherwise represented; self-attention alone is permutation-equivariant over token order.
• LayerNorm placement changes training stability, especially in deeper stacks.
• Teacher forcing during training differs from autoregressive decoding at inference.
• BLEU improvements can be affected by tokenization, beam size, checkpoint averaging, and evaluation scripts.

Limitations

• The paper’s evidence is centered on translation and parsing, not today’s broad LLM setting.
• BLEU is useful for translation comparison but does not measure all downstream language-model behavior.
• The architecture changes compute structure as well as modeling inductive bias, so architecture, hardware efficiency, and implementation are intertwined.
• Long-context behavior beyond trained/evaluated lengths requires separate evidence.
• Later Transformer variants changed normalization placement, positional encoding, activation, scaling, objective, data, and training infrastructure.
• Dense attention has quadratic memory and compute in sequence length.
• The original architecture does not directly address retrieval, tool use, factual grounding, or alignment.
• Many later “Transformer” claims depend more on data and scale than on the original block alone.

Why It Matters

This paper is the anchor for modern Transformer-based systems. For this wiki, it should be treated as:

• an architecture paper;
• a prerequisite for Transformer;
• a bridge from Attention to encoder-decoder architectures;
• a historical foundation for Agents, while not itself being a modern LLM paper.

The reusable architecture pattern is:

$$\text{token embedding}+\text{position signal}\to {\left[\text{attention token mixing}+\text{FFN channel mixing}\right]}^L\to \text{task head or decoder distribution}$$

That pattern later branches into BERT, GPT-2, Vision Transformer, Swin Transformer, and many biological sequence or structure models.

Connections

• Attention
• Transformer
• Encoder-decoder architectures
• Encoder-only Transformer
• Decoder-only Transformer
• Cross-attention
• Feed-forward network
• Residual connection
• Normalization placement
• Positional encoding
• Softmax
• Tokenization
• Computational complexity
• Benchmark claim contract
• AI architectures
• BERT
• GPT-2
• Vision Transformer
• Architecture papers