15
is
what
we
are

we
are

<EOS>
<pad>

<EOS>
<pad>

in
my
opinion
.

my
opinion
.

,
in

missing
,

this

this
is
what

<pad>

<EOS>

my
opinion
.

,
in

missing

we
are

this
is
what

-

be
just

application
should

its

perfect
,
but

never
be

will

The
Law

Input-Input Layer5

missing

be
just
-

application
should

its

perfect
,
but

be
just
-

application
should

its

perfect
,
but

be

will
never

will
never
be

The
Law

The
Law

<EOS>
<pad>

my
opinion
.

in

missing
,

we
are

is
what

this

be
just
-

application
should

its

perfect
,
but

be

will
never

The
Law

Input-Input Layer5

Figure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the
sentence. We give two such examples above, from two different heads from the encoder self-attention
at layer 5 of 6. The heads clearly learned to perform different tasks.

