troupe-standup/wizard-work/sphinx/transformer-archive/archive.md

Archive note - Attention Is All You Need (arXiv:1706.03762v7)

• Artifact: transformer-archive
• Curator: sphinx (Archive lane)
• Built (UTC): 2026-10-20T20:47:46+00:00
• Source URL: https://arxiv.org/html/1706.03762v7
• Version: v7 (submitted 12 Jun 2017; last revised 2 Aug 2023)
• Source snapshot: SOURCE-text.md (36700 bytes, sha256 8e0403fcc8d11dad28ab64ffa9edc302279c2b818ff45f64f6754b256b983161)
• Figure: figure1.png (PNG 1520x2239, 159152 bytes, sha256 56d69ccc192b90c3b93e5e285ff892f74753f970103939b40291945e106e9f79)
• Retrieval: web_extract on the arXiv HTML rendering; figure via curl from the same origin.
• Every quote below was sliced verbatim out of SOURCE-text.md at build time by anchor pair, not typed by hand. verify.py re-derives all of it.

Section map (from the snapshot)

• Abstract

• 1 Introduction

• 2 Background

• 3 Model Architecture

• 3.1 Encoder and Decoder Stacks

• Encoder:

• Decoder:

• 3.2 Attention

• 3.2.1 Scaled Dot-Product Attention

• 3.2.2 Multi-Head Attention

• 3.2.3 Applications of Attention in our Model

• 3.3 Position-wise Feed-Forward Networks

• 3.4 Embeddings and Softmax

• 3.5 Positional Encoding

• 4 Why Self-Attention

• 5 Training

• 5.1 Training Data and Batching

• 5.2 Hardware and Schedule

• 5.3 Optimizer

• 5.4 Regularization

• Residual Dropout

• Label Smoothing

• 6 Results

• 6.1 Machine Translation

• 6.2 Model Variations

• 6.3 English Constituency Parsing

• 7 Conclusion

• Acknowledgements

• References

• Attention Visualizations

Claims (each quote is verbatim from the snapshot)

C1 - EN-DE headline result

Claim: The big Transformer sets a new state of the art on WMT 2014 EN-DE at 28.4 BLEU, beating all prior models including ensembles by >2 BLEU. QUOTE| Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task, improving over the existing best results, including ensembles, by over 2 BLEU.

C2 - first pure-self-attention transduction model

Claim: The authors claim the Transformer is the first transduction model to rely entirely on self-attention, with no sequence-aligned RNNs or convolution. QUOTE| the Transformer is the first transduction model relying entirely on self-attention to compute representations of its input and output without using sequence-aligned RNNs or convolution.

C3 - encoder stack depth

Claim: The encoder is a stack of N=6 identical layers, each with a multi-head self-attention sub-layer and a position-wise feed-forward sub-layer. QUOTE| The encoder is composed of a stack of N=6 identical layers.

C4 - multi-head width

Claim: Multi-head attention uses h=8 heads with d_k=d_v=d_model/h=64 per head. QUOTE| In this work we employ h=8 parallel attention layers, or heads. For each of these we use d_k=d_v=d_(model)/h=64.

C5 - training hardware

Claim: Training ran on a single machine with 8 NVIDIA P100 GPUs. QUOTE| We trained our models on one machine with 8 NVIDIA P100 GPUs.

C6 - optimizer

Claim: Optimization used Adam with beta_1=0.9, beta_2=0.98, epsilon=1e-9. QUOTE| We used the Adam optimizer [20] with β_1=0.9, β_2=0.98 and ϵ=10^(−9).

C7 - label smoothing

Claim: Label smoothing of epsilon_ls=0.1 was applied during training, trading perplexity for accuracy/BLEU. QUOTE| During training, we employed label smoothing of value ϵ_(ls)=0.1 [36]. This hurts perplexity, as the model learns to be more unsure, but improves accuracy and BLEU score.

C8 - training data

Claim: WMT 2014 EN-DE training used ~4.5M sentence pairs with a shared ~37k BPE vocabulary. QUOTE| We trained on the standard WMT 2014 English-German dataset consisting of about 4.5 million sentence pairs.

C9 - self-attention vs recurrent complexity

Claim: A self-attention layer links all positions in O(1) sequential operations; a recurrent layer needs O(n). QUOTE| a self-attention layer connects all positions with a constant number of sequentially executed operations, whereas a recurrent layer requires O(n) sequential operations.

C10 - constituency parsing generalization

Claim: On English constituency parsing the Transformer beats BerkeleyParser even when trained only on the 40K-sentence WSJ set. QUOTE| the Transformer outperforms the BerkeleyParser [29] even when training only on the WSJ training set of 40K sentences.

C11 - conclusion

Claim: The paper's conclusion: the Transformer replaces recurrent encoder-decoder layers with multi-headed self-attention. QUOTE| we presented the Transformer, the first sequence transduction model based entirely on attention, replacing the recurrent layers most commonly used in encoder-decoder architectures with multi-headed self-attention.

C12 - future work / other modalities

Claim: Stated future work: extend the Transformer to input/output modalities other than text. QUOTE| We plan to extend the Transformer to problems involving input and output modalities other than text

Figure 1 - multimodal record

• Archived file: figure1.png
• Reference in the paper text: ![Refer to caption](1706.03762v7/Figures/ModalNet-21.png)
• Vision pass (measured, produced by a vision model over the pixels - not typed): Tall vertical flow chart on white, split into a left Encoder half and a right Decoder half. Encoder bottom-to-top: 'Inputs' -> 'Input Embedding' (pink) -> a plus-sign circle fed by a yin-yang swirl labelled 'Positional Encoding' -> an 'Nx' stack of 'Multi-Head Attention' (peach) / 'Add & Norm' (green, residual loop) / 'Feed Forward' (blue) / 'Add & Norm' (green). Decoder bottom-to-top: 'Outputs (shifted right)' -> 'Output Embedding' -> positional-encoding add -> an 'Nx' stack of 'Masked Multi-Head Attention' / 'Add & Norm' / 'Multi-Head Attention' (cross-attention, fed from the encoder top) / 'Add & Norm' / 'Feed Forward' / 'Add & Norm', then 'Linear' (lavender) -> 'Softmax' -> 'Output Probabilities'. Arrow types: thick upward main flow, residual loops curving into each Add & Norm, three short parallel Q/K/V arrows into each attention block, a horizontal encoder-to-decoder connector, and plus-sign add nodes. The image itself carries no caption text.
• Figure labels that also occur verbatim in the paper text (3): Positional Encoding, Multi-Head Attention, Softmax
• Figure-only labels, not present in the paper text (10): Inputs, Input Embedding, Nx, Add & Norm, Feed Forward, Outputs (shifted right), Output Embedding, Masked Multi-Head Attention, Linear, Output Probabilities

Provenance and integrity

• manifest.json carries sha256 + byte size for both payloads, the UTC build time, the section map, the quote list, and the figure-label cross-check.
• verify.py independently re-hashes the payloads, re-derives every quote as a substring of the snapshot, re-derives the figure-label split, and checks the PNG magic bytes and IHDR dimensions. Exit code 0 iff all of it holds.