attention pre-softmax scaling vs final perplexity over 56 16k-token documents

(s=8)

x6

Oo5

ul

12

13

14

15

Wt |
