                                                                                                                    The      The                     The            The
                                                                                                                    Law      Law                     Law            Law
                                                                                                                     will    will                       will        will
                                                                                                                   never     never               never              never
                                                                                                                      be     be                     be              be
                                                                                                                 perfect     perfect          perfect               perfect
                                                                                                                       ,     ,                      ,               ,
                                                                                                                     but     but                  but               but
                                                                                                                      its    its                          its       its
                                                                                                              application    application   application              application
                                                                                                                  should     should            should               should
                                                                                                                                                                                  Input-Input Layer5




                                                                                                                                               Input-Input Layer5
                                                                                                                      be     be                         be          be
                                                                                                                     just    just                      just         just




14
                                                                                                                        -    -                                -     -
                                                                                                                     this    this                   this            this




     and 6. Note that the attentions are very sharp for this word.
                                                                                                                       is    is                       is            is
                                                                                                                    what     what                  what             what
                                                                                                                     we      we                        we           we
                                                                                                                     are     are                       are          are
                                                                                                                 missing     missing          missing               missing
                                                                                                                         ,   ,                               ,      ,
                                                                                                                       in    in                            in       in
                                                                                                                     my      my                   my                my
                                                                                                                 opinion     opinion          opinion               opinion
                                                                                                                        .    .                       .              .
                                                                                                                 <EOS>       <EOS>            <EOS>                 <EOS>
                                                                                                                  <pad>      <pad>             <pad>                <pad>




     Full attentions for head 5. Bottom: Isolated attentions from just the word ‘its’ for attention heads 5
     Figure 4: Two attention heads, also in layer 5 of 6, apparently involved in anaphora resolution. Top:
