(a) Llama2-4k and Chunkllama

(b) Together-32k and ChunkTogether

T

1F-

----- 7-3

1-

----- 3-2

=

0.5

0.5

a

Llama-4k

—e—

- ><

Together-32k

0

7

ChunkLlama-4k

0

= 2= ChunkTogether-32k

|

|

|

2k

4k

8k

16k

32k

64k

128k

192k

2k

Ak

8k

16k

32k

64k

128k

192k

Context Window

Context Window
