Main topic
Background & Motivation
Dominant sequence transduction models (LSTM, GRU) process tokens sequentially, generating hidden states $h_t = f(h_{t-1}, x_t)$. This recurrence prevents parallelization during training and forces gradient signals to traverse $O(n)$ steps for long-range dependencies.
Attention mechanisms had already existed, but were always used *alongside* RNNs as a soft alignment add-on (Bahdanau et al., 2014). The Transformer is the first architecture to rely entirely on attention, discarding recurrence and convolutions entirely — allowing full parallelization and $O(1)$ path length between any two positions.
Attention mechanisms had already existed, but were always used *alongside* RNNs as a soft alignment add-on (Bahdanau et al., 2014). The Transformer is the first architecture to rely entirely on attention, discarding recurrence and convolutions entirely — allowing full parallelization and $O(1)$ path length between any two positions.
Original §1, p.1
Recurrent models typically factor computation along the symbol positions of the input and output sequences... This inherently sequential nature precludes parallelization within training examples, which becomes critical at longer sequence lengths.
Main topic
Model Architecture: Encoder-Decoder Stacks
The Transformer uses an encoder-decoder structure (Fig. 1). Both encoder and decoder are stacks of $N=6$ identical layers.
Encoder: Each layer has two sublayers — (1) multi-head self-attention, (2) position-wise FFN. Each sublayer is wrapped with residual connection + layer norm:
$$\text{output} = \text{LayerNorm}(x + \text{Sublayer}(x))$$
All sublayers and embeddings produce dimension $d_{\text{model}} = 512$.
Decoder: Same 6-layer stack, but adds a third sublayer for cross-attention over encoder output. Self-attention in the decoder is causally masked (upper-triangle set to $-\infty$) to preserve the auto-regressive property.
Encoder: Each layer has two sublayers — (1) multi-head self-attention, (2) position-wise FFN. Each sublayer is wrapped with residual connection + layer norm:
$$\text{output} = \text{LayerNorm}(x + \text{Sublayer}(x))$$
All sublayers and embeddings produce dimension $d_{\text{model}} = 512$.
Decoder: Same 6-layer stack, but adds a third sublayer for cross-attention over encoder output. Self-attention in the decoder is causally masked (upper-triangle set to $-\infty$) to preserve the auto-regressive property.
Original §3.1, p.3
The encoder is composed of a stack of N=6 identical layers. Each layer has two sub-layers... We employ a residual connection around each of the two sub-layers, followed by layer normalization.
Main topic
Scaled Dot-Product Attention
The attention function maps a query and a set of key-value pairs to an output — a weighted sum of values where the weight is the compatibility of the query with each key:
$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
$Q \in \mathbb{R}^{n \times d_k}$, $K \in \mathbb{R}^{m \times d_k}$, $V \in \mathbb{R}^{m \times d_v}$. The scaling factor $\tfrac{1}{\sqrt{d_k}}$ prevents large dot-product magnitudes from saturating the softmax (see §3.2.1 footnote). Compared to additive attention (which uses a feed-forward net for compatibility), dot-product attention is faster and more space-efficient via matrix multiplication, with identical quality after scaling.
$$\text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
$Q \in \mathbb{R}^{n \times d_k}$, $K \in \mathbb{R}^{m \times d_k}$, $V \in \mathbb{R}^{m \times d_v}$. The scaling factor $\tfrac{1}{\sqrt{d_k}}$ prevents large dot-product magnitudes from saturating the softmax (see §3.2.1 footnote). Compared to additive attention (which uses a feed-forward net for compatibility), dot-product attention is faster and more space-efficient via matrix multiplication, with identical quality after scaling.
Original §3.2.1, p.4
We call our particular attention 'Scaled Dot-Product Attention'. The input consists of queries and keys of dimension d_k, and values of dimension d_v.
Main topic
Multi-Head Attention
Instead of one attention function over $d_{\text{model}}$-dimensional vectors, the model projects Q, K, V $h$ times into smaller subspaces and runs attention in parallel:
$$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)\, W^O$$
$$\text{head}_i = \text{Attention}(QW_i^Q,\; KW_i^K,\; VW_i^V)$$
Projection matrices: $W_i^Q, W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}$, $W^O \in \mathbb{R}^{hd_v \times d_{\text{model}}}$.
In this work: $h=8$, $d_k = d_v = d_{\text{model}}/h = 512/8 = 64$. Total compute matches single-head full-dim attention (each head is cheaper; combined they are equivalent).
$$\text{MultiHead}(Q,K,V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h)\, W^O$$
$$\text{head}_i = \text{Attention}(QW_i^Q,\; KW_i^K,\; VW_i^V)$$
Projection matrices: $W_i^Q, W_i^K \in \mathbb{R}^{d_{\text{model}} \times d_k}$, $W_i^V \in \mathbb{R}^{d_{\text{model}} \times d_v}$, $W^O \in \mathbb{R}^{hd_v \times d_{\text{model}}}$.
In this work: $h=8$, $d_k = d_v = d_{\text{model}}/h = 512/8 = 64$. Total compute matches single-head full-dim attention (each head is cheaper; combined they are equivalent).
Original §3.2.2, p.4
Multi-head attention allows the model to jointly attend to information from different representation subspaces at different positions. With a single attention head, averaging inhibits this.
Main topic
FFN, Embeddings & Positional Encoding
Position-wise FFN (applied identically and independently to each position):
$$\text{FFN}(x) = \max(0,\; xW_1 + b_1)\,W_2 + b_2$$
$d_{\text{model}} = 512 \to d_{ff} = 2048 \to 512$. Equivalent to two convolutions with kernel size 1.
Positional Encoding (sinusoidal, fixed — no learned parameters):
$$PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$
Wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$. The key property: for any fixed offset $k$, $PE_{pos+k}$ is a linear function of $PE_{pos}$ (via angle-addition), allowing the model to attend by relative offset. Learned embeddings gave nearly identical results (Table 3, row E).
$$\text{FFN}(x) = \max(0,\; xW_1 + b_1)\,W_2 + b_2$$
$d_{\text{model}} = 512 \to d_{ff} = 2048 \to 512$. Equivalent to two convolutions with kernel size 1.
Positional Encoding (sinusoidal, fixed — no learned parameters):
$$PE_{(pos,\,2i)} = \sin\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right), \qquad PE_{(pos,\,2i+1)} = \cos\!\left(\frac{pos}{10000^{2i/d_{\text{model}}}}\right)$$
Wavelengths form a geometric progression from $2\pi$ to $10000 \cdot 2\pi$. The key property: for any fixed offset $k$, $PE_{pos+k}$ is a linear function of $PE_{pos}$ (via angle-addition), allowing the model to attend by relative offset. Learned embeddings gave nearly identical results (Table 3, row E).
Original §3.5, p.6
Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens.
Main topic
Why Self-Attention? Complexity Analysis
Three criteria motivate self-attention over recurrent/convolutional layers:
1. Complexity per layer — total compute
2. Parallelizable operations — minimum sequential steps (bottleneck for training speed)
3. Maximum path length — shorter paths = easier long-range dependency learning
Self-attention: $O(n^2 \cdot d)$ compute, $O(1)$ sequential, $O(1)$ path. Recurrent: $O(n \cdot d^2)$ compute, $O(n)$ sequential, $O(n)$ path. Self-attention is better when $n < d$ — true for typical sentence-level NLP where $n \approx 50{-}200$ and $d_{\text{model}} = 512$. For very long sequences ($n \gg d$), self-attention becomes the bottleneck.
1. Complexity per layer — total compute
2. Parallelizable operations — minimum sequential steps (bottleneck for training speed)
3. Maximum path length — shorter paths = easier long-range dependency learning
Self-attention: $O(n^2 \cdot d)$ compute, $O(1)$ sequential, $O(1)$ path. Recurrent: $O(n \cdot d^2)$ compute, $O(n)$ sequential, $O(n)$ path. Self-attention is better when $n < d$ — true for typical sentence-level NLP where $n \approx 50{-}200$ and $d_{\text{model}} = 512$. For very long sequences ($n \gg d$), self-attention becomes the bottleneck.
Original §4, p.6
In this section we compare various aspects of self-attention layers to the recurrent and convolutional layers commonly used for mapping one variable-length sequence of symbol representations... Motivating our use of self-attention we consider three desiderata.
Table 1: Layer-Type Complexity Comparison
| Layer Type | Complexity / Layer | Sequential Ops | Max Path Length |
|---|---|---|---|
| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
| Self-Attn (restricted, nbhd $r$) | $O(r \cdot n \cdot d)$ | $O(1)$ | $O(n/r)$ |
*n = sequence length, d = representation dim, k = conv kernel size, r = neighborhood size*
|---|---|---|---|
| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
| Self-Attn (restricted, nbhd $r$) | $O(r \cdot n \cdot d)$ | $O(1)$ | $O(n/r)$ |
*n = sequence length, d = representation dim, k = conv kernel size, r = neighborhood size*
Main topic
Training Setup
Data: WMT 2014 EN→DE (~4.5M sentence pairs, BPE vocab 37K shared source-target); EN→FR (36M pairs, word-piece vocab 32K). Batched by approximate sequence length; each batch ~25K source + 25K target tokens.
Hardware: 8× NVIDIA P100 GPUs. Base model: 100K steps at ~0.4 s/step (~12 hrs). Big model: 300K steps at ~1.0 s/step (~3.5 days).
Optimizer: Adam ($\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-9}$) with warmup schedule:
$$lrate = d_{\text{model}}^{-0.5} \cdot \min(step^{-0.5},\; step \cdot warmup^{-1.5}), \quad warmup = 4000 \text{ steps}$$
Regularization: Residual Dropout ($P_{drop}=0.1$) on each sublayer output before Add & Norm + on embedding sums. Label Smoothing ($\epsilon_{ls}=0.1$).
Hardware: 8× NVIDIA P100 GPUs. Base model: 100K steps at ~0.4 s/step (~12 hrs). Big model: 300K steps at ~1.0 s/step (~3.5 days).
Optimizer: Adam ($\beta_1=0.9$, $\beta_2=0.98$, $\epsilon=10^{-9}$) with warmup schedule:
$$lrate = d_{\text{model}}^{-0.5} \cdot \min(step^{-0.5},\; step \cdot warmup^{-1.5}), \quad warmup = 4000 \text{ steps}$$
Regularization: Residual Dropout ($P_{drop}=0.1$) on each sublayer output before Add & Norm + on embedding sums. Label Smoothing ($\epsilon_{ls}=0.1$).
Original §5.2–5.3, p.7
We trained our models on one machine with 8 NVIDIA P100 GPUs... We used the Adam optimizer with β1=0.9, β2=0.98 and ε=10^{−9}. We varied the learning rate over the course of training, according to the formula lrate = d_model^{−0.5}·min(step_num^{−0.5}, step_num·warmup_steps^{−1.5}).
Main topic
Results: Machine Translation
EN→DE (newstest2014): Transformer (big) achieves 28.4 BLEU, surpassing all prior models including ensembles by >2.0 BLEU. Prior best: ConvS2S Ensemble at 26.36. Training cost: $2.3 \times 10^{19}$ FLOPs vs. $7.7 \times 10^{19}$ for ConvS2S Ensemble.
EN→FR (newstest2014): 41.8 BLEU, surpassing all prior single models at less than 1/4 the training cost of GNMT+RL ($1.4 \times 10^{20}$ FLOPs).
Constituency Parsing (WSJ §23 F1): 4-layer Transformer achieves 92.7 (semi-supervised), outperforming all RNN seq2seq baselines and matching specialized parsers — demonstrating task generalization.
EN→FR (newstest2014): 41.8 BLEU, surpassing all prior single models at less than 1/4 the training cost of GNMT+RL ($1.4 \times 10^{20}$ FLOPs).
Constituency Parsing (WSJ §23 F1): 4-layer Transformer achieves 92.7 (semi-supervised), outperforming all RNN seq2seq baselines and matching specialized parsers — demonstrating task generalization.
Original §6.1, p.8
On the WMT 2014 English-to-German translation task, the big transformer model outperforms the best previously reported models, including ensembles, by more than 2.0 BLEU, establishing a new state-of-the-art BLEU score of 28.4.
Table 2: BLEU vs Training Cost (from paper)
| Model | EN-DE BLEU | EN-FR BLEU | FLOPs EN-DE | FLOPs EN-FR |
|---|---|---|---|---|
| ByteNet | 23.75 | — | — | — |
| GNMT + RL | 24.6 | 39.92 | $2.3\times10^{19}$ | $1.4\times10^{20}$ |
| ConvS2S | 25.16 | 40.46 | $9.6\times10^{18}$ | $1.5\times10^{20}$ |
| MoE | 26.03 | 40.56 | $2.0\times10^{19}$ | $1.2\times10^{20}$ |
| ConvS2S Ensemble | 26.36 | 41.29 | $7.7\times10^{19}$ | $1.2\times10^{21}$ |
| Transformer (base) | 27.3 | 38.1 | $\mathbf{3.3\times10^{18}}$ | — |
| Transformer (big) | 28.4 | 41.8 | $2.3\times10^{19}$ | — |
|---|---|---|---|---|
| ByteNet | 23.75 | — | — | — |
| GNMT + RL | 24.6 | 39.92 | $2.3\times10^{19}$ | $1.4\times10^{20}$ |
| ConvS2S | 25.16 | 40.46 | $9.6\times10^{18}$ | $1.5\times10^{20}$ |
| MoE | 26.03 | 40.56 | $2.0\times10^{19}$ | $1.2\times10^{20}$ |
| ConvS2S Ensemble | 26.36 | 41.29 | $7.7\times10^{19}$ | $1.2\times10^{21}$ |
| Transformer (base) | 27.3 | 38.1 | $\mathbf{3.3\times10^{18}}$ | — |
| Transformer (big) | 28.4 | 41.8 | $2.3\times10^{19}$ | — |
Table 3: Ablation Study (EN-DE dev set, newstest2013)
| Variant | N | $d_{\text{model}}$ | h | $d_k$ | PPL(dev) | BLEU(dev) | Params(M) |
|---|---|---|---|---|---|---|---|
| base | 6 | 512 | 8 | 64 | 4.92 | 25.8 | 65 |
| (A) h=1 | — | — | 1 | 512 | 5.29 | 24.9 | — |
| (A) h=4 | — | — | 4 | 128 | 5.00 | 25.5 | — |
| (A) h=32 | — | — | 32 | 16 | 5.01 | 25.4 | — |
| (B) $d_k$=16 | — | — | — | 16 | 5.16 | 25.1 | 58 |
| (C) N=2 | 2 | — | — | — | 6.11 | 23.7 | 36 |
| (C) $d_{\text{model}}$=1024 | — | 1024 | — | — | 4.66 | 26.0 | 168 |
| (D) no dropout | — | — | — | — | 5.77 | 24.6 | — |
| (E) learned PE | — | — | — | — | 4.92 | 25.7 | — |
| big | 6 | 1024 | 16 | — | 4.33 | 26.4 | 213 |
|---|---|---|---|---|---|---|---|
| base | 6 | 512 | 8 | 64 | 4.92 | 25.8 | 65 |
| (A) h=1 | — | — | 1 | 512 | 5.29 | 24.9 | — |
| (A) h=4 | — | — | 4 | 128 | 5.00 | 25.5 | — |
| (A) h=32 | — | — | 32 | 16 | 5.01 | 25.4 | — |
| (B) $d_k$=16 | — | — | — | 16 | 5.16 | 25.1 | 58 |
| (C) N=2 | 2 | — | — | — | 6.11 | 23.7 | 36 |
| (C) $d_{\text{model}}$=1024 | — | 1024 | — | — | 4.66 | 26.0 | 168 |
| (D) no dropout | — | — | — | — | 5.77 | 24.6 | — |
| (E) learned PE | — | — | — | — | 4.92 | 25.7 | — |
| big | 6 | 1024 | 16 | — | 4.33 | 26.4 | 213 |
Main topic
Conclusion & Outlook
The Transformer is the first sequence transduction model based entirely on self-attention, replacing recurrent layers in encoder-decoder architectures. It achieves state-of-the-art EN→DE (28.4 BLEU) and EN→FR (41.8 BLEU) while training significantly faster (3.5 days on 8 P100s).
Beyond translation, it generalizes to English constituency parsing with minimal task-specific tuning (WSJ F1: 92.7 semi-supervised).
Future directions named by authors: extend to images, audio, video; local/restricted attention for long inputs; making generation less sequential. These were all realized within 2–4 years (ViT, Speech Transformer, Video Transformer, Longformer).
Beyond translation, it generalizes to English constituency parsing with minimal task-specific tuning (WSJ F1: 92.7 semi-supervised).
Future directions named by authors: extend to images, audio, video; local/restricted attention for long inputs; making generation less sequential. These were all realized within 2–4 years (ViT, Speech Transformer, Video Transformer, Longformer).
Original §7, p.10
In this work, we presented the Transformer, the first sequence transduction model based entirely on attention... We are excited about the future of attention-based models and plan to apply them to other tasks.
Question
Why is sequential computation fundamentally limiting?
Gap / Idea
Key novelty: attention as the sole mechanism, not a supplement
Method
Residual + LayerNorm: why it matters
Method
Causal masking in decoder: enforcing auto-regression
Question
Why N=6? Is this a principled choice or trial-and-error?
Claim
Scaling by 1/√d_k prevents softmax saturation
Question
Q/K/V roles are the author's intent — does training actually enforce them?
Gap / Idea
Dot-product vs. additive attention: where does scaling fail?
Method
Three applications of attention in the Transformer
Question
Do the 8 heads actually learn different representations?
Claim
Multi-head compute is equivalent to single-head — benefit is diversity, not efficiency
Question
Can sinusoidal PE actually encode relative position?
Gap / Idea
Absolute PE is a limitation → relative PE as successor
Gap / Idea
O(n²) attention: quadratic bottleneck for long sequences
Claim
Attention weights as interpretability — and its limits
Method
Warmup LR schedule: why linear ramp is essential
Claim
Label smoothing: hurts perplexity but improves BLEU
Result
EN→DE: +2 BLEU over all prior ensembles at 3× less compute
Result
EN→FR: 41.8 BLEU at 1/6 the cost of prior best single model
Question
BLEU as a proxy: what does it miss?
Gap / Idea
Impact: Transformer as the universal architecture
Ext. reference
Code: tensorflow/tensor2tensor → Hugging Face
Question
KV Cache: why can K and V be reused but not Q?
Question
d_model vs d_k/d_v: the difference between single-head and multi-head
Question
Multi-layer: what flows between layers is the hidden state, not V
Question
KV Cache = Causal masking (why) + caching (implementation) combined
Question
Prefill vs Decode: with n input tokens, can the intermediate tokens be skipped?
Memo
[TEST] PDF search feature test
Test node for the right-click → search-in-PDF feature. Right-clicking the original quote should jump to the Abstract (p.1) of 1706.03762v7.pdf and highlight the sentence below. Safe to delete once verified.
Original Abstract, p.1
The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder.