Needle-in-a-Haystack sentence retrieval test, comparison between our method v.s. baselines v.s. GPT-

LMSys LongChat v1.5 7B 32K, acc 18.0

Together Al LLaMA-2 7B 32K, acc 27.9

LLaMA-2 7B LongLora 100K, acc 70.0

0%

50%

no}

® 100%

mee

YaRN Mistral 7B 128K, acc 57.4

Ours LLaMA 7B, post-trained on 80K, acc 88.0

OpenAl GPT-4 128K, acc 87.1

Q 0%

a

50%

100%

uae]

32K

64K

96K

128K

1K

32K

64K

96K

128K

1K

32K

64K

96K

128K
