<pad>

<pad>

<pad>

<pad>

<pad>

<pad>

<pad>
<pad>

<EOS>

<pad>
<pad>
<pad>

.
.

<EOS>
<pad>

voting

process
more
difficult
process
more
difficult

registration
or

or
voting

registration

making
the
making
the

laws

since
2009

new

laws
since
2009

have
passed
new
have
passed

American
governments

of
of

American
governments

spirit

that
a
majority
that
a
majority

in
this
in

this
spirit

It
is

It
is

Input-Input
Layer5
Attention Visualizations

Figure 3: An example of the attention mechanism following long-distance dependencies in the
encoder self-attention in layer 5 of 6. Many of the attention heads attend to a distant dependency of
the verb ‘making’, completing the phrase ‘making...more difficult’. Attentions here shown only for
the word ‘making’. Different colors represent different heads. Best viewed in color.

13

