Attention Is All You Need
A knowledge skill wrapping Vaswani et al., Attention Is All You Need (NIPS 2017, arXiv:1706.03762).
It provides the Transformer architecture's core concepts, formulas, hyperparameters, and benchmark
results so WorkBuddy can explain, compare, or implement the model accurately without re-deriving
from scratch.
The original 15-page PDF is bundled at assets/attention-is-all-you-need.pdf. Detailed notes live
in references/paper-notes.md and the formulas in references/key-equations.md. Load those when a
task needs more depth than the summary below.
When to Use
Invoke this skill when the user:
- Asks what the Transformer is, or how self-attention / multi-head attention works.
- Mentions "Attention Is All You Need", the NIPS 2017 paper, or Vaswani et al.
- Wants to implement a Transformer block, attention layer, or positional encoding.
- Compares RNN/CNN vs attention, or asks about parallelization and long-range dependencies.
- Needs the exact equations, hyperparameters (base/big), or BLEU benchmark numbers.
- Applies the Transformer to translation, parsing, or asks how it generalizes.
Core Knowledge Summary
The big idea
The Transformer replaces recurrence (RNN/LSTM/GRU) and convolution entirely with attention.
It is an encoder-decoder stack built from multi-head self-attention and position-wise
feed-forward layers, with residual connections + LayerNorm around each sub-layer. No sequential
computation means far better parallelization and much shorter paths between distant positions,
which makes long-range dependencies easier to learn.
Architecture at a glance
- Encoder: N=6 identical layers. Each layer = (1) multi-head self-attention, (2) position-wise
FFN. Output of every sub-layer:LayerNorm(x + Sublayer(x)). All dimensionsd_model = 512. - Decoder: N=6 identical layers. Adds a third sub-layer that attends over the encoder output,
and masks self-attention so positionionly sees positions< i(preserves autoregression).
Output embeddings are offset by one position.
Key equations (see references/key-equations.md for full LaTeX)
Scaled dot-product attention:
Attention(Q,K,V) = softmax(QKᵀ / √d_k) V
Multi-head attention:
MultiHead(Q,K,V) = Concat(head_1,…,head_h) W^O, where
head_i = Attention(QWᵢ^Q, KWᵢ^K, VWᵢ^V)
Position-wise FFN: FFN(x) = max(0, xW₁+b₁)W₂+b₂ (d_ff = 2048)
Positional encoding (sinusoidal, added to embeddings):
PE(pos,2i) = sin(pos / 10000^(2i/d_model)),
PE(pos,2i+1) = cos(pos / 10000^(2i/d_model))
Sub-layer output: LayerNorm(x + Sublayer(x)). Embedding weights are shared between the two
embedding layers and the pre-softmax linear layer, and scaled by √d_model.
Three uses of attention in the model
- Encoder self-attention — Q/K/V all from the previous encoder layer (each position sees all).
- Decoder self-attention — Q/K/V from previous decoder layer, masked to forbid looking ahead.
- Encoder-decoder attention — Q from decoder, K/V from encoder output (each target position
attends all source positions).
Hyperparameters
| N | d_model | d_ff | heads | d_k=d_v | P_drop | Steps | Params | |
|---|---|---|---|---|---|---|---|---|
| base | 6 | 512 | 2048 | 8 | 64 | 0.1 | 100K | 65M |
| big | 6 | 1024 | 4096 | 16 | 64 | 0.3 | 300K | 213M |
Training: Adam (β₁=0.9, β₂=0.98, ε=1e-9) with warmup-then-inverse-sqrt learning rate
(warmup_steps=4000); residual dropout 0.1 (base); label smoothing ε_ls=0.1.
Headline results
- WMT 2014 EN-DE: 28.4 BLEU (big) — beats all prior models incl. ensembles by >2 BLEU.
- WMT 2014 EN-FR: 41.0 BLEU single model (41.8 in some configs) at < 1/4 the training cost.
- English constituency parsing (WSJ): F1 91.3 (WSJ only) / 92.7 (semi-supervised) — strong
generalization beyond translation.
Why self-attention wins
Self-attention has O(1) sequential operations and O(1) maximum path length vs O(n) for
recurrent layers, so it parallelizes better and learns long-range dependencies more easily (at
O(n²·d) per-layer complexity). See Table 1 in references/paper-notes.md.
Using This Skill
- For a quick explanation or a specific fact (equation, hyperparameter, BLEU score), answer from the
summary above. - For deep dives, implementations, or citations, load
references/paper-notes.mdand
references/key-equations.md. - To quote or show the original publication, reference
assets/attention-is-all-you-need.pdf.
Resources
assets/attention-is-all-you-need.pdf— the original 15-page paper (Google NIPS 2017).references/paper-notes.md— section-by-section extraction: architecture, equations, tables,
training regime, results, and appendix visualizations.references/key-equations.md— copy-paste LaTeX for all formulas and the complexity table.
