Table 3: Variations on the Transformer architecture. Unlisted values are identical to those of the base
model. All metrics are on the English-to-German translation development set, newstest2013. Listed
perplexities are per-wordpiece, according to our byte-pair encoding, and should not be compared to
per-word perplexities.

base

N

dmodel

dff

h

dk

dv

Pdrop

ϵls

6

512

2048

8
1
4
16
32

64
512
128
32
16
16
32

64
512
128
32
16

0.1

0.1

32
128

32
128

(A)

(B)

train
steps
100K

2
4
8
(C)

256
1024
1024
4096

0.0
0.2

(D)
(E)
big

6

0.0
0.2
positional embedding instead of sinusoids
1024 4096 16
0.3

300K

PPL
(dev)
4.92
5.29
5.00
4.91
5.01
5.16
5.01
6.11
5.19
4.88
5.75
4.66
5.12
4.75
5.77
4.95
4.67
5.47
4.92
4.33

BLEU
(dev)
25.8
24.9
25.5
25.8
25.4
25.1
25.4
23.7
25.3
25.5
24.5
26.0
25.4
26.2
24.6
25.5
25.3
25.7
25.7
26.4

params
×106
65

58
60
36
50
80
28
168
53
90

213

development set, newstest2013. We used beam search as described in the previous section, but no
checkpoint averaging. We present these results in Table 3.
In Table 3 rows (A), we vary the number of attention heads and the attention key and value dimensions,
keeping the amount of computation constant, as described in Section 3.2.2. While single-head
attention is 0.9 BLEU worse than the best setting, quality also drops off with too many heads.
In Table 3 rows (B), we observe that reducing the attention key size dk hurts model quality. This
suggests that determining compatibility is not easy and that a more sophisticated compatibility
function than dot product may be beneficial. We further observe in rows (C) and (D) that, as expected,
bigger models are better, and dropout is very helpful in avoiding over-fitting. In row (E) we replace our
sinusoidal positional encoding with learned positional embeddings [9], and observe nearly identical
results to the base model.
6.3

English Constituency Parsing

To evaluate if the Transformer can generalize to other tasks we performed experiments on English
constituency parsing. This task presents specific challenges: the output is subject to strong structural
constraints and is significantly longer than the input. Furthermore, RNN sequence-to-sequence
models have not been able to attain state-of-the-art results in small-data regimes [37].
We trained a 4-layer transformer with dmodel = 1024 on the Wall Street Journal (WSJ) portion of the
Penn Treebank [25], about 40K training sentences. We also trained it in a semi-supervised setting,
using the larger high-confidence and BerkleyParser corpora from with approximately 17M sentences
[37]. We used a vocabulary of 16K tokens for the WSJ only setting and a vocabulary of 32K tokens
for the semi-supervised setting.
We performed only a small number of experiments to select the dropout, both attention and residual
(section 5.4), learning rates and beam size on the Section 22 development set, all other parameters
remained unchanged from the English-to-German base translation model. During inference, we
9

