328 lines
14 KiB
JSON
328 lines
14 KiB
JSON
{
|
||
"schema": "sphinx/archive-citations/1",
|
||
"corpus": "manifest.json",
|
||
"source_dir": "sources/",
|
||
"citation_count": 29,
|
||
"citations": [
|
||
{
|
||
"id": "LITM-1",
|
||
"slug": "lost-in-the-middle",
|
||
"arxiv_id": "2307.03172",
|
||
"title": "Lost in the Middle: How Language Models Use Long Contexts",
|
||
"section": "Abstract",
|
||
"line": 60,
|
||
"char_offset": 2434,
|
||
"words": 40,
|
||
"quote": "we observe that performance is often highest when relevant information occurs at the beginning or end of the input context, and significantly degrades when models must access relevant information in the middle of long contexts, even for explicitly long-context models."
|
||
},
|
||
{
|
||
"id": "LITM-2",
|
||
"slug": "lost-in-the-middle",
|
||
"arxiv_id": "2307.03172",
|
||
"title": "Lost in the Middle: How Language Models Use Long Contexts",
|
||
"section": "1 Introduction",
|
||
"line": 78,
|
||
"char_offset": 6934,
|
||
"words": 41,
|
||
"quote": "language model performance is highest when relevant information occurs at the very beginning (primacy bias) or end of its input context (recency bias), and performance significantly degrades when models must access and use information in the middle of their input context"
|
||
},
|
||
{
|
||
"id": "LITM-3",
|
||
"slug": "lost-in-the-middle",
|
||
"arxiv_id": "2307.03172",
|
||
"title": "Lost in the Middle: How Language Models Use Long Contexts",
|
||
"section": "1 Introduction",
|
||
"line": 79,
|
||
"char_offset": 7227,
|
||
"words": 34,
|
||
"quote": "when relevant information is placed in the middle of its input context, GPT-3.5-Turbo’s performance on the multi-document question task is lower than its performance when predicting without any documents (i.e., the closed-book setting; 56.1%)"
|
||
},
|
||
{
|
||
"id": "LITM-4",
|
||
"slug": "lost-in-the-middle",
|
||
"arxiv_id": "2307.03172",
|
||
"title": "Lost in the Middle: How Language Models Use Long Contexts",
|
||
"section": "1 Introduction",
|
||
"line": 92,
|
||
"char_offset": 10361,
|
||
"words": 22,
|
||
"quote": "we find that model performance saturates long before retriever recall saturates, indicating that current models fail to effectively use additional retrieved documents"
|
||
},
|
||
{
|
||
"id": "LITM-5",
|
||
"slug": "lost-in-the-middle",
|
||
"arxiv_id": "2307.03172",
|
||
"title": "Lost in the Middle: How Language Models Use Long Contexts",
|
||
"section": "1 Introduction",
|
||
"line": 93,
|
||
"char_offset": 10840,
|
||
"words": 36,
|
||
"quote": "to claim that a language model can robustly use information within long input contexts, it is necessary to show that its performance is minimally affected by the position of the relevant information in the input context"
|
||
},
|
||
{
|
||
"id": "RULER-1",
|
||
"slug": "ruler",
|
||
"arxiv_id": "2404.06654",
|
||
"title": "Ruler: What’s the Real Context Size of Your Long-Context Language Models?",
|
||
"section": "1 Introduction",
|
||
"line": 150,
|
||
"char_offset": 6418,
|
||
"words": 45,
|
||
"quote": "While all models claim context size of 32k tokens or greater, our results indicate that only half of them can effectively handle sequence length of 32K by exceeding a qualitative threshold. Moreover, almost all models fall below the threshold before reaching the claimed context lengths."
|
||
},
|
||
{
|
||
"id": "RULER-2",
|
||
"slug": "ruler",
|
||
"arxiv_id": "2404.06654",
|
||
"title": "Ruler: What’s the Real Context Size of Your Long-Context Language Models?",
|
||
"section": "Abstract",
|
||
"line": 70,
|
||
"char_offset": 1790,
|
||
"words": 47,
|
||
"quote": "The needle-in-a-haystack (NIAH) test, which examines the ability to retrieve a piece of information (the “needle”) from long distractor texts (the “haystack”), has been widely adopted to evaluate long-context language models (LMs). However, this simple retrieval-based test is indicative of only a superficial form of long-context understanding."
|
||
},
|
||
{
|
||
"id": "RULER-3",
|
||
"slug": "ruler",
|
||
"arxiv_id": "2404.06654",
|
||
"title": "Ruler: What’s the Real Context Size of Your Long-Context Language Models?",
|
||
"section": "4 Experiments & Results",
|
||
"line": 540,
|
||
"char_offset": 26133,
|
||
"words": 45,
|
||
"quote": "while they achieve nearly perfect performance on the passkey retrieval and the vanilla NIAH task (shown in Appendix E), all of them exhibit large degradation in RULER as sequence length increases and they fail to maintain performance above the Llama2-7B baseline at their claimed length."
|
||
},
|
||
{
|
||
"id": "RULER-4",
|
||
"slug": "ruler",
|
||
"arxiv_id": "2404.06654",
|
||
"title": "Ruler: What’s the Real Context Size of Your Long-Context Language Models?",
|
||
"section": "4 Experiments & Results",
|
||
"line": 306,
|
||
"char_offset": 22598,
|
||
"words": 15,
|
||
"quote": "We use the performance of Llama2-7b model at the 4K context length as the threshold."
|
||
},
|
||
{
|
||
"id": "RULER-5",
|
||
"slug": "ruler",
|
||
"arxiv_id": "2404.06654",
|
||
"title": "Ruler: What’s the Real Context Size of Your Long-Context Language Models?",
|
||
"section": "1 Introduction",
|
||
"line": 152,
|
||
"char_offset": 7657,
|
||
"words": 24,
|
||
"quote": "training on longer sequences does not always lead to better performance on Ruler, and that larger model sizes positively correlate with better long-context capabilities."
|
||
},
|
||
{
|
||
"id": "DCA-1",
|
||
"slug": "dca-training-free",
|
||
"arxiv_id": "2402.17463",
|
||
"title": "Training-Free Long-Context Scaling of Large Language Models",
|
||
"section": "Abstract",
|
||
"line": 67,
|
||
"char_offset": 1567,
|
||
"words": 33,
|
||
"quote": "Given the expensive overhead of finetuning large-scale models with longer sequences, we propose Dual Chunk Attention (DCA), which enables Llama2 70B to support context windows of more than 100k tokens without continual training."
|
||
},
|
||
{
|
||
"id": "DCA-2",
|
||
"slug": "dca-training-free",
|
||
"arxiv_id": "2402.17463",
|
||
"title": "Training-Free Long-Context Scaling of Large Language Models",
|
||
"section": "Abstract",
|
||
"line": 68,
|
||
"char_offset": 2316,
|
||
"words": 18,
|
||
"quote": "our training-free 70B model attains 94% of the performance of gpt-3.5-16k, indicating it is a viable open-source alternative."
|
||
},
|
||
{
|
||
"id": "DCA-3",
|
||
"slug": "dca-training-free",
|
||
"arxiv_id": "2402.17463",
|
||
"title": "Training-Free Long-Context Scaling of Large Language Models",
|
||
"section": "1 Introduction",
|
||
"line": 80,
|
||
"char_offset": 5241,
|
||
"words": 29,
|
||
"quote": "in a training-free setting, we find that these approaches usually lead to a notable increase in PPL especially in input lengths that are more than twice the training length"
|
||
},
|
||
{
|
||
"id": "DCA-4",
|
||
"slug": "dca-training-free",
|
||
"arxiv_id": "2402.17463",
|
||
"title": "Training-Free Long-Context Scaling of Large Language Models",
|
||
"section": "1 Introduction",
|
||
"line": 89,
|
||
"char_offset": 7944,
|
||
"words": 35,
|
||
"quote": "It first shows that LLMs with a 4k context window can be expanded to more than 32k without training, maintaining a negligible increase in PPL, whereas previous methods typically falter at context lengths beyond 8k."
|
||
},
|
||
{
|
||
"id": "DCA-5",
|
||
"slug": "dca-training-free",
|
||
"arxiv_id": "2402.17463",
|
||
"title": "Training-Free Long-Context Scaling of Large Language Models",
|
||
"section": "1 Introduction",
|
||
"line": 92,
|
||
"char_offset": 8516,
|
||
"words": 28,
|
||
"quote": "existing long-context LLMs, which have already supported a 32k context window, can further extrapolate to a 192k context length while maintaining high passkey retrieval accuracy and low perplexity."
|
||
},
|
||
{
|
||
"id": "YARN-1",
|
||
"slug": "yarn",
|
||
"arxiv_id": "2309.00071",
|
||
"title": "YaRN: Efficient Context Window Extension of Large Language Models",
|
||
"section": "Abstract",
|
||
"line": 60,
|
||
"char_offset": 1538,
|
||
"words": 30,
|
||
"quote": "Rotary Position Embeddings (RoPE) have been shown to effectively encode positional information in transformer-based language models.\nHowever, these models fail to generalize past the sequence length they were trained on."
|
||
},
|
||
{
|
||
"id": "YARN-2",
|
||
"slug": "yarn",
|
||
"arxiv_id": "2309.00071",
|
||
"title": "YaRN: Efficient Context Window Extension of Large Language Models",
|
||
"section": "Abstract",
|
||
"line": 62,
|
||
"char_offset": 1759,
|
||
"words": 31,
|
||
"quote": "We present YaRN (Yet another RoPE extensioN method), a compute-efficient method to extend the context window of such models, requiring 10x less tokens and 2.5x less training steps than previous methods."
|
||
},
|
||
{
|
||
"id": "YARN-3",
|
||
"slug": "yarn",
|
||
"arxiv_id": "2309.00071",
|
||
"title": "YaRN: Efficient Context Window Extension of Large Language Models",
|
||
"section": "4.1 Training",
|
||
"line": 208,
|
||
"char_offset": 25374,
|
||
"words": 20,
|
||
"quote": "As shown in Figure 2, YaRN converges faster compared to other interpolation techniques during training and consistently has lower loss."
|
||
},
|
||
{
|
||
"id": "YARN-4",
|
||
"slug": "yarn",
|
||
"arxiv_id": "2309.00071",
|
||
"title": "YaRN: Efficient Context Window Extension of Large Language Models",
|
||
"section": "B.2 Training Efficiency of YaRN",
|
||
"line": 1008,
|
||
"char_offset": 51640,
|
||
"words": 20,
|
||
"quote": "Even if YaRN was only fine-tuned for 400 steps compared to PI’s 1000 steps, we obtain similar results to PI."
|
||
},
|
||
{
|
||
"id": "SURVEY-1",
|
||
"slug": "lclm-survey",
|
||
"arxiv_id": "2503.17407",
|
||
"title": "A Comprehensive Survey on Long Context Language Modeling",
|
||
"section": "7.1.1 The False Promise of Support Context Length",
|
||
"line": 1884,
|
||
"char_offset": 175760,
|
||
"words": 30,
|
||
"quote": "Despite this impressive scaling of context extension, there exists a significant gap between the context length these models claim to support and the actual context length they can process effectively"
|
||
},
|
||
{
|
||
"id": "SURVEY-2",
|
||
"slug": "lclm-survey",
|
||
"arxiv_id": "2503.17407",
|
||
"title": "A Comprehensive Survey on Long Context Language Modeling",
|
||
"section": "7.1.1 The False Promise of Support Context Length",
|
||
"line": 1884,
|
||
"char_offset": 176732,
|
||
"words": 23,
|
||
"quote": "The findings reveal a consistent pattern: For most models across both categories, the effective context length rarely exceeds half of the claimed length."
|
||
},
|
||
{
|
||
"id": "SURVEY-3",
|
||
"slug": "lclm-survey",
|
||
"arxiv_id": "2503.17407",
|
||
"title": "A Comprehensive Survey on Long Context Language Modeling",
|
||
"section": "7.1.1 The False Promise of Support Context Length",
|
||
"line": 1884,
|
||
"char_offset": 176159,
|
||
"words": 26,
|
||
"quote": "performance remains robust when information is located at the beginning or end of the input, but deteriorates substantially when critical information is positioned in the middle"
|
||
},
|
||
{
|
||
"id": "SURVEY-4",
|
||
"slug": "lclm-survey",
|
||
"arxiv_id": "2503.17407",
|
||
"title": "A Comprehensive Survey on Long Context Language Modeling",
|
||
"section": "7.2.1 Positional Embedding",
|
||
"line": 2014,
|
||
"char_offset": 186649,
|
||
"words": 15,
|
||
"quote": "They point out that scaling down rotation bases can only achieve superficial long context capabilities."
|
||
},
|
||
{
|
||
"id": "SURVEY-5",
|
||
"slug": "lclm-survey",
|
||
"arxiv_id": "2503.17407",
|
||
"title": "A Comprehensive Survey on Long Context Language Modeling",
|
||
"section": "8.6 Application in Multimodal Tasks",
|
||
"line": 2072,
|
||
"char_offset": 203402,
|
||
"words": 25,
|
||
"quote": "Multimodal long context benchmarks can primarily be divided into two types: those that use multiple images and those that use long videos as the context."
|
||
},
|
||
{
|
||
"id": "SURVEY-6",
|
||
"slug": "lclm-survey",
|
||
"arxiv_id": "2503.17407",
|
||
"title": "A Comprehensive Survey on Long Context Language Modeling",
|
||
"section": "8.6 Application in Multimodal Tasks",
|
||
"line": 2072,
|
||
"char_offset": 204516,
|
||
"words": 27,
|
||
"quote": "Compared to multi-image long context data, understanding long videos is more challenging due to their larger context length and the inclusion of more redundant and noisy information."
|
||
},
|
||
{
|
||
"id": "DATAENG-1",
|
||
"slug": "data-eng-128k",
|
||
"arxiv_id": "2402.10171",
|
||
"title": "Data Engineering for Scaling Language Models to 128K Context",
|
||
"section": "Abstract",
|
||
"line": 37,
|
||
"char_offset": 1265,
|
||
"words": 25,
|
||
"quote": "for quantity, we show that 500 million to 5 billion tokens are enough to enable the model to retrieve information anywhere within the 128K context"
|
||
},
|
||
{
|
||
"id": "DATAENG-2",
|
||
"slug": "data-eng-128k",
|
||
"arxiv_id": "2402.10171",
|
||
"title": "Data Engineering for Scaling Language Models to 128K Context",
|
||
"section": "Abstract",
|
||
"line": 39,
|
||
"char_offset": 1523,
|
||
"words": 26,
|
||
"quote": "naïvely upsampling longer data on certain domains like books, a common practice of existing work, gives suboptimal performance, and that a balanced domain mixture is important."
|
||
},
|
||
{
|
||
"id": "DATAENG-3",
|
||
"slug": "data-eng-128k",
|
||
"arxiv_id": "2402.10171",
|
||
"title": "Data Engineering for Scaling Language Models to 128K Context",
|
||
"section": "1 Introduction",
|
||
"line": 72,
|
||
"char_offset": 6040,
|
||
"words": 33,
|
||
"quote": "continual pretraining on a small amount of long-context data, in our case, 1-5B tokens, can “unlock” a 7B model’s capability of precise retrieval over much longer context lengths than seen in original pretraining."
|
||
},
|
||
{
|
||
"id": "DATAENG-4",
|
||
"slug": "data-eng-128k",
|
||
"arxiv_id": "2402.10171",
|
||
"title": "Data Engineering for Scaling Language Models to 128K Context",
|
||
"section": "5.1 Overall Performance",
|
||
"line": 267,
|
||
"char_offset": 24011,
|
||
"words": 20,
|
||
"quote": "the model’s retrieval performance saturates at about 5B tokens, and further scaling to 10B tokens does not improve length generalization."
|
||
}
|
||
]
|
||
}
|