rform equally well, if not better, across these tasks. This evidence indicates the absence of negative effects from context expansion, highlighting the models’ capability to adapt to different task requirements without losing efficiency in shorter contexts.
Table 1: Performance evaluation across language tasks, comparing Llama-2 7B (4K context window) and context-expanded variants of LWM-Text (32K to 1M). The results demonstrate that increasing context length does not significantly degrade performance on tasks with shorter contexts.
LWM-Text
Task / Metric
Llama-2 7B
32k
128k
256k
512k
1M
arc_challenge/acc
0.40
0.43
0.45
0.44
0.44
0.43
arc_challenge/acc_norm
0.43
0.47
0.47
0.46
0.46
0.46
hellaswag/acc
0.57
0.57
0.57
0.57
0.56
0.57
hellaswag/acc_norm
0.77
0.76
0.76
0.76
0.75
0.75
mmlu
0.39
0.4
0.41
0.41
0.36
0.35
openbookqa/acc
0.32
0.33
0.31
0.32
0.33
0.30
openbookqa/acc_norm
0.44
0.44
0.44
0.43
0.41
0.41
3.3.2 Retrieval Task: Single Information
We evaluate on the popular Needle In A Haystack task (gkamradt, 2023) – more specifically an version (ArizeAI, 2023) that finds and retrieves random numbers assigned to randomized cities from the context. Figure 2 shows that we can scale to far larger contexts compared to the current best available LLMs.
Figure 11 in Appendix shows nearly perfect retrieval accuracy over the entire context of our 1M context model.
Appendix C shows more single needle retrieval results for our other shorter context length models.
3.3.3 Retrieval Task: Multiple Information
We additionally examine the performance of our model on more complex variant of the needle retrieval task by mixing in multiple needles, as well as trying to retrieve a specific subset of them. Figure 5 shows multi-needle retrieval results under different settings. Our model generalizes well when retrieving a single needle from multiple needles in context, with slight degradation when asked to retrieve more than one needle. Table 2 shows multi-needle comparisons, where our model is able to perform competitively or better than GPT-4 at retrieving one needle, or slightly lower performance when retrieving more than one needle. Furthermore, our model is also able to perform well and extend to longer context lengths of up to 1M tokens and far outperforms any recent shorter context baselines applies to longer sequence lengths through positional extrapolation techniques.. However, we note that we see degradation in accuracy while increasing the difficulty of the needle retrieval task, suggesting that there is still more room to improve on the 1M context utilization of our model. We believe that our released model will provide a foundation for future work on developing longer context models, as well as encourage more challenging benchmarks that contain difficult long-range tasks that require higher levels of synthesis, rather than pure fact retrieval.
Table 2: Multi Needle in a Haystack. * denotes models after the completion of this paper.
Context Length
Model
N=2,R=2N=2,R=2
N=4,R=1N=4,R=1
N=4,R=2N=4,R=2
32K
Gemini Pro (02/23)
0.34
0.44
0.6
GPT-4-1106
0.97
0.95
0.9
Llama-3.1-8B-Instruct*
0.87
0.95
0.93
Qwen2.5-7B-Instruct*
1.0
1.0
0.97
Mistral-7B-Instruct-v0.3*
0.98
0.85
0.83
LWM-Text-1M (Ours)
0.84
0.97
0.84
128K
Gemini Pro (02/23)
-
-
-
GPT-4-1106
0.92
0.8
0.82
Llama-3.1-8B-Instruct*
0.98
0.91
0.87
Qwen2.5-7B-Instruct*
0.98
0.80
0.90
Mistral-7B-Instruct-v0.3*
0.85
0.75
0.68
LWM-Text-1M (Ours)
0.83
0.98
0.83
1M
Gemini Pro (02/23)
-
-
-
GPT-4-1106
-
-
-
Llama-3.1-8B-Instruct*
0.27
0.32
0.18
Qwen2.5-7B-Instruct*
0.0
0.0
0.0
Mistral-7B-Instruct-v0.3*
0.05
0.13
0.10
LWM-Text-1M (Ours)
0.67
0.84
0.69
Figure 5: Multiple needles retrieval task with LWM-1M. NN is the number of facts in the context, and RR is the number of given facts model is asked to retrieve.
3.3.4 Evaluation on LOFT
Table 3: Evaluations on some benchmarks in the LOFT dataset.
Setting: 512K Context
LWM (512K)
GPT-4o (128K)
Claude 3 Opus (200K)
Quora
0.38\bf 0.38
0.230.23
0.370.37
NQ
0