                                                                                                                 The      The                     The           The
                                                                                                                 Law      Law                     Law           Law
                                                                                                                  will    will                       will       will
                                                                                                                never     never               never             never
                                                                                                                   be     be                     be             be
                                                                                                              perfect     perfect          perfect              perfect
                                                                                                                    ,     ,                      ,              ,
                                                                                                                  but     but                  but              but
                                                                                                                   its    its                          its      its
                                                                                                           application    application   application             application
                                                                                                               should     should            should              should




                                                                                                                                           Input-Input Layer5
                                                                                                                                                                              Input-Input Layer5




                                                                                                                   be     be                         be         be
                                                                                                                  just    just                      just        just




15
                                                                                                                     -    -                                -    -
                                                                                                                  this    this                   this           this
                                                                                                                    is    is                       is           is
                                                                                                                 what     what                  what            what
                                                                                                                  we      we                        we          we




     at layer 5 of 6. The heads clearly learned to perform different tasks.
                                                                                                                  are     are                       are         are
                                                                                                              missing     missing          missing              missing
                                                                                                                      ,   ,                               ,     ,
                                                                                                                    in    in                            in      in
                                                                                                                  my      my                   my               my
                                                                                                              opinion     opinion          opinion              opinion
                                                                                                                     .    .                       .             .
                                                                                                              <EOS>       <EOS>            <EOS>                <EOS>
                                                                                                               <pad>      <pad>             <pad>               <pad>




     sentence. We give two such examples above, from two different heads from the encoder self-attention
     Figure 5: Many of the attention heads exhibit behaviour that seems related to the structure of the
