A Comprehensive Survey on Long Context
Language Modeling

Absolute Position Embedding provides each token with information about its absolute position in the sequence. This approach can be categorized into three types: (1) directly specifying the positional embedding (PE) for each position through a functional approach; (2) setting the PE for each position as trainable vectors that are updated during training; and (3) omitting positional embeddings entirely and relying on unidirectional attention in decoder-only architectures to learn positions implicitly. Below are the details:

• •

  Functional Positional Embedding [518]. The most representative example of functional position embedding is Sinusoidal Positional Embedding, which was proposed by the vanilla Transformer [518] and was widely used in variant models based on Transformers. It encodes positional information through periodic sine and cosine functions. In Transformer, it is usually added to word embeddings to inject absolute positional information into the network.

• •

  Learned Positional Embedding [143]. It was firstly proposed by Meta to introduce positional information into the convolutional sequence-to-sequence learning. The embedding representations of different positions are treated as learnable parameters and trained along with the network. This is widely used in models such as BERT [101], GPT [37], and OPT [656]. There are also some variants [525] that represents the position embedding in the complex plane and combines it with the continuous word embedding which is also represented in complex vectors.

• •

  No Position Embedding (NoPE) [74, 241]. NoPE proposes to incorporate no explicit position encoding into Transformer. It founds that the model still learns the permutation order of tokens. NoPE is also effective in language modeling and several probing tasks. [74] prove that NoPE in causal language models can incorporate implicit absolute positional information into the models. Note that NoPE also conveys relative positional information [241].

Instead of focusing on absolute positional information, Relative Position Embedding [453] captures the relative distances between tokens, as it assumes that relative positional information is more crucial for language understanding. For LCLMs, commonly used Relative Position Embeddings can be categorized into following three types:

• •

  T5-Style [425]. T5 initially maps the relative distance $(i-j)$ between tokens at positions $i$ and $j$ to a scalar bias value $b=f(i-j)$, where $f$ represents a lookup table. Subsequently, the relative bias $b$ (learned during the training process) is incorporated into the dot product of the query and key within the self-attention mechanism. The lookup table assigns the same parameter to distances exceeding a certain threshold, thereby facilitating generalization to unobserved distances. Base on T5’s position embedding, FIRE [287] proposes to map the positional representations as a learnable function. To solve the length generalization issue when the test length varies, FIRE proposes progressive interpolation by normalizing the distance by query position index. In this way, FIRE can map any test length into the training range.

• •

  ALiBi [410] and its Variants. ALiBi, which is adopted in BLOOM [440] and Baichuan [603], is similar to T5’s position encoding. However, it differs in that it subtracts a scalar bias from the attention score. This bias increases linearly in proportion to the distance between the query and key tokens. As a result, this actually generates a preference for tokens that are closer in position, which is known as a recency bias. Alibi can be formulated as the following:

  $$\mathbf{q}_{i}\mathbf{k}_{j}^{T}=(\mathbf{x}_{i}\mathbf{W}_{q})(\mathbf{x}_{j}\mathbf{W}_{k})^{T}+m(j-i)$$

  (1)

  where $m$ is a head-specific scalar hyper-parameter. $i$ and $j$ mean the position indices of $\mathbf{q}$ and $\mathbf{k}$. $\mathbf{W}_{q}$ and $\mathbf{W}_{k}$ represent the projection layer in the attention. Building on top of ALiBi, Kerple [75] introduces two trainable parameters for better length extrapolation, while Sandwich [76] simplifies the form of sinusoidal position embedding by only considering the cross term of position embeddings to alleviate the overfitting issue of sinusoidal position embedding. BAM [28] frames attention as a Bayesian mechanism and adopts a Generalized Gaussian Distribution as prior, achieving improved length extrapolation capability.

• •

  Rotary Position Embedding (RoPE) [476] and its variants. RoPE rotates the query and key representations by an angle that is proportional to their absolute positions prior to dot product attention. Owing to this rotation, the attention dot product will rely solely on the relative distance between tokens, thereby effectively transforming it into a relative positional encoding. To elucidate, given a hidden vector $\mathbf{h}=[h_{0},h_{1},...,h_{d-1}]$, where $d$ is the hidden dimension, and a position index $m$, RoPE operates as follows:

  $$f(\mathbf{h},m)=\begin{pmatrix}h_{0}\\ h_{1}\\ h_{2}\\ h_{3}\\ \vdots\\ h_{d-2}\\ h_{d-1}\end{pmatrix}\otimes\begin{pmatrix}\cos{m\theta_{0}}\\ \cos{m\theta_{0}}\\ \cos{m\theta_{1}}\\ \cos{m\theta_{1}}\\ \vdots\\ \cos{m\theta_{d/2-1}}\\ \cos{m\theta_{d/2-1}}\end{pmatrix}+\begin{pmatrix}-h_{1}\\ h_{0}\\ -h_{3}\\ h_{2}\\ \vdots\\ -h_{d-1}\\ h_{d-2}\end{pmatrix}\otimes\begin{pmatrix}\sin{m\theta_{0}}\\ \sin{m\theta_{0}}\\ \sin{m\theta_{1}}\\ \sin{m\theta_{1}}\\ \vdots\\ \sin{m\theta_{d/2-1}}\\ \sin{m\theta_{d/2-1}}\end{pmatrix}$$

  (2)

  where $\theta_{j}=10000^{-2j/d},j\in\{0,1,...,d/2-1\}$. Given a query $\mathbf{q}$ at position $m$ and a key $\mathbf{k}$ at position $n$, attention score $a(\mathbf{q},\mathbf{k})$ is defined as:

  	$\displaystyle a(\mathbf{q},\mathbf{k})$	$\displaystyle=<f(\mathbf{q},m),f(\mathbf{k},n)>$
  		$\displaystyle=\sum_{j=0}^{d/2-1}[(q_{2j}k_{2j}+q_{2j+1}k_{2j+1})\cos{(m-n)\theta_{j}}+(q_{2j}k_{2j+1}-q_{2j+1}k_{2j})\sin{(m-n)\theta_{j}}]$
  		$\displaystyle:=g(\mathbf{q},\mathbf{k},(m-n)\boldsymbol{\theta})$		(3)

  where g(·) is an abstract mapping function exclusively dependent on $\mathbf{q},\mathbf{k}$ and $(m-n)\boldsymbol{\theta}$. Benefiting from this good nature, RoPE becomes the most pervasive position embedding strategy in the era of LLMs, including LLaMA [515], Qwen [604], etc. Sun et al. [487] ascribes that the weak extrapolation ability of RoPE is related to the significant oscillation in their attention expectations. To address this problem, they propose XPOS that incorporates a balancing term to penalize the oscillation of unstable dimensions while maintaining the distribution of stable dimensions. Also building upon RoPE’s foundation, HoPE [67] enhances length extrapolation by replacing specific components with position-independent ones while retaining only high-frequency signals.

More recently, there has been a growing interest in Content-Aware Position Embedding [154, 680], which argues that position measurement should take into account more semantically meaningful units such as words or sentences, as illustrated below:

• •

  CoPE [154]. CoPE considers the joint modeling of content and position information by leveraging the content of the tokens and their positions in the sequence. Specifically, CoPE first calculates context-dependent gate values, and then employs these values to determine token positions through a cumulative summation process. In this way, positions embeddings are equipped with contextualized information.

• •

  DAPE [680, 681]. It proposes to model the positional information dynamically with the attention. Specifically, DAPE determines the position bias by not only the position indices but also the semantics information. In this way, DAPE overcomes the inflexibility and achieves relatively optimal performance for each individual instance by dynamically adjusting on each specific input data.

To address the quadratic complexity of attention in Transformer-based models and the challenge posed by the extremely large KV cache in long context scenarios, numerous prior works have explored sparse attention mechanisms, which reduce both computational and memory overhead, enabling faster processing and improved scalability for real-world applications. Sparse attention can be broadly categorized into training-based and training-free approaches, depending on whether it is applied during training or inference.

For training-based methods, the sparsity is mainly reflected from two perspectives, head dimension (e.g. group query attention) and context window dimension (e.g. sliding window attention). The most common function in head dimension sparse attention is group query attention(GQA) [5]. Grouped-query attention organizes query heads into $G$ distinct groups, where each group utilizes a shared key head and value head. When $G=1$, GQA has only one group, one key and one value head, which is comparable to Multi-Query Attention (MQA). When $G=H$ (the number of groups $G$ is equal to the number of heads $H$), the MQA is analogous to standard Multi-Head Attention (MHA). Weighted Grouped Query Attention [79] adds novel learnable parameters for every key and value head in the attention blocks to achieve aggregation among these key and value heads. Mixture of Attention (MoA) [127] automatically tailors distinct sparse attention configurations for different heads and layers, where the MoA constructs and navigates a search space of various attention patterns and their scaling rules relative to input sequence lengths.

As the context length grows, sparsity in the context window dimension can effectively reduce the extra computation caused by increasing the length and capturing valid information [343, 542, 272]. Beltagy et al. [25] introduce the Longformer with an attention mechanism that scales linearly with sequence length to process documents of thousands of tokens. Zebra [468] groups local-global attention layers into blocks during the training and inference phases. Xiao et al. [589] have found that keeping the KV of initial tokens will largely recover the performance of window attention, where these initial tokens lack meaningful semantics, but attract substantial attention scores. Gao et al. [137] adopts a fully learning-based approach to adaptively identify attention sparsity in LLMs and leverage the learned sparsity for efficient inference. Recently, following the principles of Mixture of Experts (MoE), the MoBA [338] has been proposed, where the trainable block sparse attention and parameter-less gating mechanism are introduced.

However, most previous methods require training a model from scratch, limiting their usability as direct plugins for ready-to-use large language models. Some training-free sparse attention approaches are as follows:

• •

  Static Strategy. A variety of methods have been proposed to enhance efficiency while maintaining performance during inference. Window attention [25, 78, 215] maintains a fixed-size sliding window on the KV states of most recent tokens, ensuring efficiency but suffering from performance degradation once the initial tokens are evicted. StreamingLLM [589] mitigates this issue by identifying “attention sinks" where the initial tokens often receive disproportionately high attention scores, allowing LLMs trained on finite attention windows to handle infinite-length sequences without fine-tuning. LM-Infinite [169] designs a $\Lambda$-shaped attention mask and a ceiling on attention distances and enables LLMs to generalize to extreme sequence lengths without any parameter updates.

• •

  Dynamic Strategy. Unlike previous methods that rely on static, fixed-window strategies, later methods employ dynamic strategy to select and retain the most important tokens. H₂O [669] retains only the “Heavy Hitters" tokens in the KV cache using accumulated attention scores. Similarly, Scissorhands [333] prioritizes important tokens based on the “Persistence of Importance Hypothesis". CORM [90] is a KV cache eviction policy to dynamically retains important key-value pairs for inference without model fine-tuning. SnapKV [296] improves efficiency by utilizing attention scores to identify and cluster significant tokens. FastGen [140] recognizes several fundamental attention patterns and introduces an adaptive KV cache eviction policy. MInference [218] accelerates the prefilling stage using dynamic sparse attention with spatial aggregation patterns, allowing seamless integration into existing LLMs. Quest [495] is a query-aware KV cache selection algorithm which tracks the minimum and maximum keys and values in KV cache pages.

• •

  Layer-Level Optimization. Most of the aforementioned methods adopt the same strategy across layers to reduce the KV cache, ignoring the varying importance of information processing at different layers. PyramidKV [40] and PyramidInfer [607] have discovered that LLMs aggregate information through a pyramid-shaped information funnel and dynamically adjust the KV cache size across different layers, allocating more KV cache to lower layers and less to higher layers. LazyLLM [126] apply layer-wise token pruning in each generation step. DynamicKV [693] dynamically optimizes token retention by adjusting the number of tokens retained at each layer to better suit specific tasks. LightTransfer [664] minimizes inter-layer KV cache redundancies by selectively removing cache from identified lazy layers. TidalDecode [612] accelerates LLM decoding through position persistent sparse attention.

• •

  Head-Level Optimization. Previous works have not focused on the differences in attention patterns across different heads. RazorAttention [494] and DuoAttention [587] significantly improves the long context efficiency by dividing the attention heads into Retrieval Heads (which require full KV cache) and Non-retrieval Heads (which only need a fixed-size KV cache), which substantially reduces memory consumption and maintains accuracy while increasing both decoding and prefilling speeds. AdaKV [125] adaptively allocates the KV cache budget across attention heads to optimize cache eviction strategies. HeadKV [129] incorporates Retrieval Heads identification to introduce a new adaptive budget allocation strategy, which globally allocates budgets individually for each attention head. LONGHEADS [342] splits the sequence context into chunks and each attention head focus on parts of chunks. When generating a particular token, LONGHEADS selects the $k$ blocks that are most relevant to the current token based on its query vector and chunk representation.

• •

  Other Methods. Despite these advancements, many approaches suffer from permanent information loss due to aggressive token pruning. Loki [465] is a PCA-based sparse attention mechanism by leveraging the low-dimensionality of key vectors in the attention block, yielding inference speedup and maintaining efficacy.

Various hierarchical mechanisms have been proposed to introduce a structured hierarchy into self-attention, which leverages high-level global information and low-level local attention for multi-scaled contextual receptive fields. HAN [618] pioneers the use of a two-level attention mechanism. It first applies self-attention to word features to obtain a sentence representation, then employs self-attention on sentence-level features to generate document-level features. Hi-Transformer [565] models documents in a hierarchical way, which learns both the sentence representation and the document representation. ERNIE-SPARSE[330] leverages a Hierarchical Sparse Transformer to sequentially unify local and global information. HOMER [470] processes extensive text by first dividing the input into smaller chunks and subsequently merging them in stages as they pass through the transformer’s layers.

Apart from the relatively high computational complexity, the self-attention mechanism integrates global and local information in the encoding of each token, rendering it less suitable for tasks such as long-text understanding. Based on this insight, some works strive to combine Transformer architecture with recurrent structures, enhancing the capture of long-term dependencies and improving the ability to understand long contexts [577, 629, 449].

In the earlier attempts, Transformer-XL [91] extends the context window beyond fixed lengths by employing segment-level recurrence and relative positional encoding. Memformer [570] utilizes an external dynamic memory to encode and retrieve past information, attaining linear time complexity and constant memory space complexity when processing long sequences. Compressive Transformer [423] maps past hidden activations (memories) to a smaller set of compressed representations (compressed memories) for long-range sequence learning. Block-Recurrent Transformer [204] applies a transformer layer in a recurrent manner along a sequence. During training, the recurrent cell operates on blocks of tokens rather than individual tokens and exploits parallel computation within a block to efficiently utilize accelerator hardware, exhibiting linear complexity with respect to the sequence length. Unlike existing recurrent-based models that use special global tokens to store representations and place them at the beginning of the input sequence, RMT[38] incorporates a memory mechanism through special memory tokens added to the input sequence, enabling the model to store and process both local and global information efficiently. SRformer[335] divides the input sequence into segments to compute segmented attention and adds recurrent attention to aggregate global information over segments. Infinite Attention[375] incorporates a compressive memory into the attention mechanism and incorporates both masked local attention and long-term linear attention mechanisms within a single Transformer block. ARMT[431] is based on transformer self-attention for local context and segment-level recurrence for storing task-specific information distributed over a long context.

By caching the previously computed key/value vectors, the decoders can reuse them for the current generation step. The key-value (KV) cache avoids encoding the history again for each token. Although this can significantly improve the inference efficiency, it imposes expensive memory overhead as the sequence length grows. To alleviate the KV cache memory consumption, some efficiency-driven modifications have been proposed, including reducing KV-heads, such as Grouped-Query Attention (GQA), Multi-Query Attention (MQA) and Multi-head Latent Attention (MLA) mechanism [310]. Note that the GQA and MQA has been discussed in Sparse Attention. For MLA, instead of directly reducing KV-heads, which is primarily considered a trade-off between performance and memory consumption, the MLA compresses the key and value into a latent vector to reduce the caching consumption and decompresses the key and value head for each query head when generating.

SSM is proposed based on classical Kalman filtering Kim and Bang [244], Gu et al. [156]. It describes the state representation of a sequence at each time and predicts its next state based on the input. The SSM consists of two main parts - state and observation. The state equation expresses the trend of the future state, while the observation equation predicts the current output based on the current state and input.

$x(t)\in\mathbb{R}^{n}$ represents the state vector, $u(t)\in\mathbb{R}^{d_{i}}$ represents the input signal and $y(t)\in\mathbb{R}^{d_{o}}$ represents the output signal. $\mathbf{A}\in\mathbb{R}^{n\times n}$, $\mathbf{B}\in\mathbb{R}^{n\times d_{i}}$, $\mathbf{C}\in\mathbb{R}^{d_{o}\times n}$ and $\mathbf{D}\in\mathbb{R}^{d_{o}\times d_{i}}$ is the state matrix, input matrix, output matrix, and feed-forward matrix, which are parameters learned by gradient descent. because the term $\mathbf{D}u(t)$ can be viewed as a skip connection and is easy to compute. We can hypothesis $\mathbf{D}$ is a zero matrix. The rewritten formula is as follows:

The classical SSM model is used to process continuous signals, while discrete inputs are common in the field of NLP [160]. Linear State Space Layer (LSSL) introduces continuous SSM to obtain two discretized representations including the Recurrent Representation and the Convolutional Representation. Mamba [157] leveraged the zero-order hold (ZOH) to discretize SSM. The Recurrent Representation of discretization is as follows:

$\overline{\mathbf{A}}=\exp(\Delta\mathbf{A})$, $\overline{\mathbf{B}}=(\Delta\mathbf{A})^{-1}(\exp(\Delta\mathbf{A})-\mathbf{I})\cdot\Delta\mathbf{B}$, $\Delta$ is step size that represents the resolution of the input. if we use $x$ instead of $u$ and $h$ instead of $x$, We can see that it has a similar formula as recurrent neural networks (RNNs),

Similar to RNN, the recurrent representation can be performed for efficient inference, but it cannot be used for parallel training. Therefore, LSSL proposes the convolutional representation for training.

Recently, the Structured State Space Sequence model (S4) [159] is proposed based on SSM to modeling the long sequences, which can be computed more efficiently than previous methods while retaining their theoretical advantages.

Prior work found that the SSM shows poor performance as it is a Linear Time-Invariant (LTI) system that is deeply connected to recurrence and convolutions. In other words, ($\Delta$, $\mathbf{A}$, $\mathbf{B}$, $\mathbf{C}$, $\overline{\mathbf{A}}$, $\overline{\mathbf{B}}$) in Equation 7 are fixed for all time-steps. However, LTI has fundamental limitations in modeling certain types of data. It loses the ability to efficiently select data in an input-dependent manner. Gu and Dao [158] propose Mamba to design a simple selection mechanism by parameterizing the SSM parameters. The details are as follows:

where $s_{B}$, $s_{C}$ and $s_{\Delta}$ are the Linear layer and $\tau_{\Delta}=softplus$. Meanwhile, Mamba introduces the hardware-aware algorithm that computes the model recurrently with a parallel scan instead of convolution. Notably, it avoids IO access between different levels of the GPU memory hierarchy without modeling the expanded state. Finally, Mamba has 5x higher throughput than the Transformer and has linear scaling over sequence length, which is friendly with long context capabilities.

Many methods have been proposed to optimize Mamba [65, 634, 471]. For example, ReMamba [634] employs selective compression and adaptation methods to compress and retain essential information, minimizing data degradation and reducing state space updates to alleviate information loss. Similar to ReMamba, TAIPAN [380] combines Mamba-2 with selective attention layers for enhancing performance and ensuring computational efficiency, where the selective attention layer is used to filter out irrelevant information. DeciMamba [26] introduces a context-extension technique tailored for Mamba, which leverages a concealed filtering mechanism, and allows the trained model to effectively extrapolate without further training.

Transformer has a significant advantage in long context tasks, and the current model has been extended to 128k or higher. But the time complexity and space complexity of the Transformer are both $O(n^{2})$. As a result, when $n$ is significantly large, the computational burden of the Transformer model becomes challenging. Recently, considerable efforts have been made to lessen the computational demands of self-attention in Transformer [455, 417, 170, 14, 593]. In this section, we will discuss mainstream works on linear attention as follows:

First, Katharopoulos et al. [240] propose the linear transformer model by using a kernel-based formulation [421] of self-attention and the associative property of matrix products to calculate the self-attention weights, which greatly reduces memory usage and scales linearly to the context length. Choromanski et al. [82] introduce the Performers, which utilize the fast attention via positive orthogonal random features and implement the kernelizable attention mechanisms beyond softmax.

RetNet [486] includes three computation paradigms, i.e., parallel, recurrent, and chunk-wise recurrent, and the chunk-wise recurrent paradigm achieves linear complexity for sequence modeling. which achieves training parallelism, good performance, and low inference cost simultaneously. Munkhdalai et al. [376] propose the Infini-attention and include a compressive memory into the vanilla attention mechanism, which uses masked local attention and long-term linear attention modules in the Transformer block. There are similar methods, such as Lightning Attention-2 [415] employs a tiling strategy to distinctly manage intra-block and inter-block elements during linear attention computation, where the MiniMax-01 [367] model uses this attention module. It adopts standard attention mechanisms for intra-block processing while implementing linear attention kernel techniques for inter-block operations. Transformer-VQ [308] is a transformer decoder that performs dense self-attention computations in linear time relative to sequence length. This efficiency is achieved by integrating vector-quantized keys, localized positional biases, and a compressive cache designed for efficient attention processing.

Unlike previous attempts with Recurrent Transformers, RWKV [400] introduces an enhanced linear attention mechanism, merging the computational efficiency of RNNs with the parallelism and expressive power of Transformers. This enables highly parallelized training and efficient inference, with its linear time complexity making it particularly suitable for long-sequence tasks. The acronym "RWKV" represents four key elements:

• •

  R (Receptance): A gating vector that integrates historical information.

• •

  W (Weight): A trainable decay factor applied across positions.

• •

  K (Key): Functions similarly to the key vector in standard attention mechanisms.

• •

  V (Value): Operates like the value vector in traditional attention systems.

RWKV-4 is the first publicly released version, following experimental iterations (RWKV-1, RWKV-2, and RWKV-3). The RWKV-4 model consists of multiple residual blocks, each featuring time-mixing and channel-mixing components. Building on RWKV-4, RWKV-5 (Eagle) and RWKV-6 (Finch) [401] introduce further innovations. Eagle enhances expressive power by replacing vector-valued states with multi-head matrix-valued states and refining the learning decay strategy. It also reconfigures receptive states and adds gating mechanisms for improved performance. RWKV-6 (Finch) further boosts expressiveness and adaptability by integrating data-driven functions, such as parameterized linear interpolation, into the time-mixing and token-shifting modules. It also introduces low-rank adaptation functions, making weight matrices trainable and enabling context-sensitive optimization of decay vectors. These advancements maintain RNN-like inference efficiency while significantly enhancing the model’s capabilities.

Another emerging approach to modeling long context is Test-Time Training (TTT) [485], which dynamically stores context information in the model’s adaptable weights (referred to as fast weights [442]). The core idea is to frame the update of these adaptable weights as online gradient descent, as exemplified by systems like Titans [24], Atlas [23], and TTT Done Right [659]. When the update logic for adaptable weights is linear, TTT becomes equivalent to linear attention. Therefore, TTT can be viewed as a more expressive framework than linear attention, capable of capturing more complex dependencies through its nonlinear update mechanisms.

The evolution of hybrid architecture in large language models represents a significant advancement in balancing performance and efficiency, particularly for handling long contexts. The layer-wise approach, combining linear complexity attention with standard attention modules, has emerged as the predominant strategy. Jamba [304] pioneered this hybrid architecture at scale, effectively combining Transformer and Mamba layers with a MoE module, discovering that a 7:1 ratio (Mamba to Transformer layers) provided optimal balance. Jamba 1.5 [505] scaled this approach beyond 100B parameters, maintaining the same 7:1 ratio while requiring only 9GB for its KV cache with a 256K context length—compared to 80-88GB for similarly sized transformer models. Google’s RecurrentGemma [33] employed the Griffin architecture combining linear recurrences with local attention, using a mixture of mechanisms within each layer rather than interleaving different layer types. Meanwhile, Microsoft’s Samba [429] combined Mamba with SWA rather than full attention, maintaining linear complexity while providing the benefits of attention for local contexts. Comprehensive experimentation demonstrated that this hybrid approach outperformed both pure Mamba and pure attention models of similar size across standard benchmarks. Zyphra’s Zamba [150] innovated by incorporating a shared attention mechanism—utilizing a global shared attention block that appeared every few Mamba blocks but shared parameters across all instances, reducing memory requirements while maintaining strong performance. Zamba2 [149] expanded on these innovations in multiple size levels, switching from Mamba to Mamba-2 and using two alternating shared attention blocks with non-shared low-rank adapters. MiniMax-01 [367] became the first open-source hybrid model to achieve true state-of-the-art performance across comprehensive benchmarks, using a configuration where one transformer block with softmax attention followed every seven Transformer blocks with lightning attention [416]. It matched the performance of leading commercial models while supporting context lengths up to 4 million tokens.

The sliding window approach represents another variant of hybrid attention. Google’s Gemma [504] employed a more balanced 1:1 ratio of full attention to SWA layers, emphasizing balanced performance across tasks rather than maximizing context length. Cohere’s Command R-7B [87] implemented a hybrid approach with SWA, enabling efficient processing of longer contexts while maintaining strong benchmark performance. Research from Character.ai [46] found that approximately 6:1 (linear complexity to full attention) yielded optimal results—aligning with findings from other hybrid models. This convergence on ratios of approximately 6:1 or 7:1 across multiple independent research efforts indicates a sweet spot in the design space, suggesting that hybrid architectures represent not just a compromise but potentially the optimal approach for next-generation language models efficiently handling diverse contexts.

The approaches discussed above primarily rely on standard full attention to capture global information. Recently, there have also been attempts [123] to compress global information into fixed-size representations, which preserve competitive performance while significantly lowering the compute and memory costs of the global part.

The prefilling-decoding hybrid approach optimizes language model operation by applying different attention mechanisms based on the operational phase—using linear complexity attention during prefilling and implementing hybrid architectures during decoding. YOCO [488] pioneered this paradigm with a decoder-decoder structure that dramatically reduces memory requirements. The self-decoder processes input context using linear-complexity attention and produces a single global key-value cache, which is then reused by all layers of the cross-decoder. This design reduces KV cache memory requirements by approximately a factor equal to the number of model layers, which enables early exit during prefilling and maintains performance comparable to traditional Transformers. GoldFinch [153] represents another innovation in this category, stacking a GOLD transformer on top of an enhanced Finch (RWKV-6) architecture. Its key contribution is the “TokenCat” mechanism, which generates a highly compressed key cache in linear time and space. This approach yields remarkable efficiency gains, with cache size savings 756-2550 times smaller than traditional transformer caches. Both approaches demonstrate that prefilling-decoding hybridization offers a powerful alternative to pure architectures, enabling dramatic efficiency improvements for long context inference without sacrificing model quality.

The head-wise hybridization approach combines linear and attention-based mechanisms in parallel within the same layer, allowing different attention heads to process identical inputs using different mechanisms simultaneously. Hymba [106] pioneered this approach with a hybrid-head architecture that integrates transformer attention mechanisms with state space models within each layer. Attention heads provide high-resolution recall capabilities while SSM heads enable efficient context summarization. This design can be interpreted as mimicking human cognition: attention heads function like snapshot memories storing detailed recollections, while SSM heads act as memories that retain core information while forgetting details. Samba [429] combined the Mamba block and sliding window attention block together with an MLP layer. It combines local and global attention mechanisms. Local attention maintains sensitivity to near-neighbor information, while global attention allows the model to capture dependencies over long distances.

SelectiveContext [293] enhances LLM inference efficiency by calculating token self-information with a small language model, grouping tokens into lexical units, and removing redundant content. AdaComp [655] adaptively selects relevant documents by considering both query complexity and retrieval quality. CPC [309] introduces a sentence-level compression technique leveraging a context-aware sentence encoder to rank sentences by their the embedding similarity to the query and removing less relevant sentences. TCRA-LLM [318] proposes a token compression scheme with two complementary methods: summarization compression which uses a T5-based model to reduce token size, and semantic compression which eliminates words with lower semantic impact.

Besides, reinforcement learning (RL) has been applied to improve the efficiency and effectiveness of prompt compression methods. DynaICL [689] enhances in-context learning efficiency by using reinforcement learning to fine-tune a meta-controller that dynamically adjusts the number of few-shot examples based on input query difficulty, balancing efficiency and performance. PCRL [234] applies a discrete prompt compression method with reinforcement learning. TACO-RL [446] leverages task-specific reward signals to fine-tune a encoder-based compression model using on-policy RL, enabling effective compression while maintaining performance.

In addition, the LLMLingua series aims to build a specialized language for LLMs through prompt compression, which accelerate model inference, reduce costs and improve downstream performance. LLMLingua [219] leverages a smaller language model to calculate perplexity (PPL) and remove redundant tokens, featuring a budget controller, iterative prompt algorithm, and alignment techniques to achieve up to 20x prompt compression while preserving semantic integrity. LongLLMLingua [220] proposes a question-aware coarse-to-fine compression method, a document reordering mechanism, dynamic compression ratios and a subsequence recovery strategy to improve LLMs’ ability to perceive and prioritize key information. LLMLingua-2 [392] introduces a task-agnostic prompt compression method, trained via data distillation from GPT-4 with a BERT-level encoder, achieving 3x-6x speed improvements over LLMLingua while preserving crucial information.

An alternative approach involves rewriting prompts instead of selecting them. Nano-Capsulator [84] encapsulate lengthy prompts into shorter ones while adhering to specific generation length constraints, maintaining performance through an explicit semantic-preserving objective with reward scoring. CompAct [627] captures pivotal information from extensive documents by dynamically retaining essential contexts and incorporating information. FAVICOMP [233] is a decoding-time evidence compression approach that produces a refined evidence set more familiar to the target model, enhancing RAG performance while seamlessly integrating parametric knowledge.

LLM-fixed methods focus on compressing prompts without updating the original LLMs’ parameters, offering an efficient solution for managing input context length. Instead, they involve training additional modules that transform discrete tokens into continuous embeddings, facilitating efficient prompt compression. Contrastive conditioning [562] focuses on learning compact soft prompts to simulate the original natural language prompt by minimizing the Kullback-Leibler (KL) divergence. However, contrastive conditioning incurs substantial computational overhead, as it requires retraining from scratch for each new incoming prompt, making it impractical for broad application. ICAE [141] generates compact and informative memory slots to represent the original context, enabling LLM to encode more information within the same context length. This improves the model’s capability to handle long contexts and reduces computation and memory overheads during inference. 500xCompressor [301] builds upon ICAE but utilizes KV values for compressed tokens instead of embeddings,achieving remarkable compression ratios. xRAG [72] uses a frozen embedding model as the encoder and a trainable adapter between the encoder and decoder LLM. Through modality fusion, it integrates document embeddings into the LLM’s representation space, eliminating the need for textual representations and achieving high compression rates. UniICL [132] compresses demonstrations into compressed features, which are then converted into compressed virtual tokens via a learnable projection layer. These virtual tokens replace original demonstrations to shorten input length and help select potential demonstrations. The queries and virtual tokens are then fed into the frozen LLM for response generation.

In summary, LLM-fixed methods offer diverse strategies for prompt compression by optimizing input representations while preserving the frozen parameters of the original LLM. As an upper-bound reference, Kuratov et al. [255] demonstrate near-lossless $\sim 1500\times$ compression by optimizing a small set of soft ‘memory’ vectors for a frozen LLM, clarifying the achievable limits of soft-prompt compression and pointing to directions for future encoders compression ratios. In general, these methods significantly enhance computational and memory efficiency, enabling more effective utilization of LLMs for a broad range of tasks.

Gist token-based methods aim to compress prompts by condensing the context into a small set of special tokens, referred to as “gist tokens" [372]. These tokens replace the original context, enabling efficient caching and reuse while reducing computational costs. However, such methods typically require modifying the parameters of the LLM. Gist [372] introduces a method to train models for prompt compression without additional cost over standard instruction finetuning, which inserts gist tokens after the prompt and modifies Transformer attention masks to prevent tokens after the gist tokens from attending to those before them, allowing the model to learn both prompt compression and instruction following simultaneously. AutoCompressors [73] process long documents by recursively generating gist tokens which are passed as soft prompts to all subsequent segments and result in more compact soft prompts. Activation Beacon [653] serves as a plug-in for Transformer-based LLMs that enables effective, efficient, and flexible compression of long contexts, featuring a progressive compression workflow that distills the context into a small set of activations.

In summary, gist token-based methods provide a powerful framework for reducing input length by converting context into compact, reusable tokens. These approaches offer significant efficiency gains, though they typically involve updating the LLM’s parameters to enable effective compression and integration.

Training LCLMs inherently requires larger token batch sizes and significantly varying data lengths. Given constraints in memory, network bandwidth, and PCIe bandwidth, reading and transferring these large data volumes from memory to the GPU substantially slows down batch construction. Because I/O performance typically lags behind modern GPU compute performance, I/O becomes a major bottleneck in the training process. Strategic approaches, such as increasing the number of I/O threads and utilizing pinned memory, are commonly employed to mitigate this issue. However, the optimal hyperparameters for these strategic I/O optimizations require case-by-case tuning based on model size, context window length, and hardware configuration; otherwise, CPU cores or memory may become new bottlenecks.

Data packing concatenates samples into longer sequences. To maximize effective data utilization in batches, packing requires the appropriate combinatorial arrangement and essential truncation [473, 340]. Packing would alter the data reading order and the actual training distribution, potentially affecting the training performance. Attention masks can differentiate packed samples [254], ensuring isolation when necessary. However, constructing non-causal attention masks introduces fragmented operations, and non-causal attention negatively impacts training efficiency. Some methods [15, 250] employ data sampling to ensure that sample lengths follow a predefined maximum. There exist other solutions which dynamically adjust context lengths. For example, Hydraulis [278] avoids the usage of fixed context windows, but uses dynamic programming to address data sampling imbalance and data packing imbalance. Data Decomposition [408] curates data to different windows using multiple bucket sizes. Building upon these established data organization strategies, many recent pre-trained models have adopted a progressive length extension approach. This method has demonstrated significant performance improvements on commonly used long text benchmarks [675, 58, 136, 604].

In the LLM training process, data is read from disk/network to host cache, then transferred to GPU via PCIe. In distributed training, data retrieval and distribution by CPUs are limited by PCIe bandwidth. Pre-fetching with near-end data workers [672] or caching [104] can effectively overlap I/O and computation, even hiding and eliminating the I/O latency. Upon the observation that SSD throughput and RDMA bandwidth were underutilized, 3FS [98] introduces a distributed, random-access approach, which enhances both performance and usability by leveraging the available resources more effectively.

GPU memory is a significant bottleneck in long context LLM training, with parameters, gradients, optimizer states, and especially activations (proportional to sequence length) consuming substantial resources. Reducing numerical precision [162] through low-precision or mixed-precision computation is a natural solution. Mixed-precision training [361] is a common memory optimization technique. Typically, floating-point operations use FP32 (single-precision), allocating 8 exponent bits, and 23 mantissa bits. In the memory-constrained scenarios, sacrificing some precision reduces storage requirements, e.g., FP16 requires half the storage. However, FP16’s maximum integer value of 65,536 can lead to the NaN exception or Inf exception owing to the numerical overflow. BF16, another half-precision format (requiring Ampere architecture or later), increases exponent bits, mitigating overflow at the cost of some precision. FP16/BF16 are now standard for LLM training, with FP32 reserved [529] for precision-sensitive operations (RoPE, LayerNorm, Softmax) and communication reductions (where BF16 should be avoided).

Recent quantized training methods incorporate quantized fine-tuning, using 8-bit floats (FP8) or integers (INT8), and preserve accuracy in the inference phase. FP8 are currently usable in TransformerEngine and other derivative acceleration libraries [671] on Hopper GPUs, primarily for less precision-sensitive matrix multiplications in Transformer layers. It applies E4M3 (4-bit exponent, 3-bit mantissa) for forward and E5M2 for backward passes, potentially offering a 2x performance boost over BF16. INT8 integers are more widely applicable, forming the basis of most current quantization methods. Weight quantization is the most intuitive approach, mapping values to 256 discrete levels within a statistically or pre-defined range for each tensor. Quantization significantly reduces memory footprint, and with appropriate hardware and software support, dequantization overhead can be fused and minimized, significantly boosting performance. However, reduced precision can irreversibly impact model performance. Some studies observe that the error can be properly compensated at critical locations in the training process [404, 632, 199]. Optimizer states are more sensitive to precision, requiring careful training with dynamic range scaling to align their distribution with the FP8 representation, reducing both memory footprint and quantization error [579, 100]. Apart from the weights and optimizer state quantization, activation quantization is also attracting significant research. Intuitively, quantizing activations leads to substantial information loss. Fortunately, latest research discovers that this loss primarily attributes to the activation outliers, where suppressing these outliers during training can enable reliable activation quantization [285, 306, 586].

The quadratic time and memory complexity of the Transformer’s multi-head attention mechanism presents significant computational and memory challenges, particularly with long input sequences. FlashAttention [93] addresses this by leveraging the high bandwidth, but limited capacity, of GPU shared memory (e.g., 19TB/s bandwidth with a 20MB limit) for block-wise processing. It employs a log-sum-exp trick for optimized softmax computation and caches key values and cumulative sums to minimize memory access. FlashAttention demonstrates substantial performance gains in both standard and long context training scenarios. Alongside these optimizations for standard attention, variants like sparse FlashAttention [390] have gained traction. FlashAttention-v2 [94] refines the computation order of its predecessor, reducing non-matrix multiplications and optimizing warp scheduling within thread blocks to minimize shared memory access and maximize GPU parallelism. Furthermore, leveraging the Transformer Engine introduced with the Hopper GPU architecture, FlashAttention-v3 [444] incorporates FP8 acceleration and three key techniques: asynchronous data loading and computation via a producer-consumer model, overlapping softmax and GEMM computations, and block-wise quantization. This block-wise optimization paradigm facilitates further algorithmic and engineering co-optimization, as exemplified by NSA’s [635] hybrid approach combining token compression, selection, and a sliding window, and MoBA’s [339] block-wise attention mechanism, achieving a favorable balance between full and sparse attention using a MoE-like strategy. Similar solutions have already been verified on MLA [280].

Distributed computing with sharding effectively mitigates the memory pressure imposed by long context windows. Standard self-attention has a computational complexity of O(n²), where n is the sequence length. Ring attention [312] reduces this to O(n) by restricting each token’s attention to a fixed number of surrounding tokens, significantly decreasing computational and memory costs, and enabling the processing of longer sequences. Hybrid approaches combining local and global attention mechanisms further improve performance by maintaining long-range dependency modeling while reducing computational overhead [311, 35]. More direct methods employ various parallelization strategies: Sequence Parallelism [249] distributes model layers across devices, reducing individual device load but requiring substantial inter-device communication, and currently only shards Dropout and LayerNorm activations, leaving room for optimization. Context Parallelism ⁵⁵ 5 https://docs.nvidia.com/megatron-core/developer-guide/latest/api-guide/context_parallel.html divides the context window into segments processed in parallel, subsequently aggregating their representations. This effectively reduces memory requirements and increases training speed. Ulysses Parallelism [209], integrated into the DeepSpeed framework, combines the advantages of both by sharding both model layers and the context window with an interleaved strategy, minimizing communication overhead while maximizing parallel efficiency, demonstrating significant advantages for extremely long context windows.

In long context inference, the KV cache can grow extremely large, leading to memory fragmentation and inefficient usage. Virtual memory management techniques address these challenges by optimizing how KV cache memory is allocated and accessed, reducing waste and improving performance for long sequences. PagedAttention [259] utilizes virtual memory to place the KV cache of tokens at the same position across different layers and heads within the same memory page. vTensor [597] introduces a virtual memory abstraction that decouples computation from defragmentation. KV-Compress [428] extends PageAttention to support token-based KV compression.

When dealing with long context, especially in batched inference or scenarios with shared prefixes (like in conversational AI), efficient scheduling of KV cache and computations becomes critical. Scheduling strategies tackle this by organizing data structures to facilitate deduplication and sharing, maximizing memory utilization and computational efficiency for long inputs. ChunkAttention [623] and MemServe [191] organize data structures to enable efficient cache deduplication and sharing of common prefixes, thereby improving both memory utilization and computational efficiency. SGLang [683] use RadixAttention for sharing common prefixes between batches.

Language modeling is the most fundamental capability for long context comprehension. Typically, a low perplexity is an indicator that the model have good understanding of the given input document. In the realm of long context modeling, it is not only required that LCLMs have an overall low perplexity, but also that this perplexity decreases as the context window grows, indicating that the models can make more accurate predictions of target tokens by fully utilizing the rich context information. In practice, there are two common ways to examine the efficacy of LCLMs through the lens of perplexity. The first way is to inspect cumulative average negative log-likelihood (NLL) as token position in long documents increases, as exemplified by Gemini-1.5 [503]. A decreasing trend in this cumulative NLL indicates that model can make better predictions of future tokens as the context window grows. An alternative way is to inspect PPLs calculated by the sliding window approach [410, 60, 698] under various window sizes. Specifically, the sliding window approach have a pre-defined window size $w$. The model first predicts the probability of the beginning $w$ tokens, then gradually slides the window forward to process the remaining tokens and calculate overall PPL. In this case, a decreasing PPL as the window size increases indicates that the model can effectively utilize long context information.

Retrieval requires LCLMs to identify and extract target information from one or multiple locations within long contexts. This capability underlies most long context tasks, as they typically require implicit retrieval of relevant content before subsequent processing.

• •

  Categories The retrieval capability comprises two distinct levels based on their complexity: Explicit Retrieval and Semantic Retrieval. Explicit retrieval refers to string matching based on given queries, where models must locate and extract matching content from the source text. Semantic retrieval, which is typically more challenging, requires models to extract semantically relevant content based on the semantic meaning of queries.

• •

  Synthetic Tasks Needle Retrieval (or Needle-in-a-Haystack, NIAH) serves as a prototypical synthetic task to evaluate retrieval capabilities. This task inserts one or more needles into a long sequence and queries the model to retrieve the corresponding needles [371, 236, 190, 284, 594, 697, 277, 315, 17, 430]. The specific instantiation of needles and documents takes on various forms. The needles can be n-digit numbers, specific sentences, UUIDs, dictionaries, functions, passages, etc. while documents can be repeated noisy sentences, meaningful content extracted from essays, code repositories, or other sources. In line with capability categorization, the needle retrieval task includes explicit and semantic variants. In Explicit (or Literal) Needle Retrieval, models simply need to perform exact matching of needles, such as locating the completion of a partial sentence within a long text [371, 236, 190, 284, 277]. Semantic Needle Retrieval, however, requires models to identify content based on semantic correspondence [370], such as retrieving paragraphs based on their abstracts [17, 697] or functions based on their behavioral descriptions [315]. Notably, while these needle retrieval tasks primarily assess retrieval capabilities, they can be highly challenging and thus serve as crucial evaluation tasks for long context modeling abilities.

Aggregation refers to the model’s ability to integrate information from multiple locations or even globally across long contexts. Building upon retrieval capabilities, this competency encompasses two key aspects: first, it requires the model to process the text piece-by-piece across multiple locations or even the entire context; second, it demands the model to recognize meaningful connections between these pieces and synthesize them into coherent higher-level representations.

• •

  Categories Based on the nature of aggregation targets, Aggregation can be further divided into two types: Statistical Aggregation and Semantic Aggregation. Statistical Aggregation requires models to perform quantitative analysis over long contexts, while Semantic Aggregation focuses on combining and synthesizing semantic information from different parts of the context.

• •

  Synthetic Tasks A variety of synthetic statistical tasks have been proposed to evaluate Statistical Aggregation capabilities [469, 190, 663]. These tasks include tracing variable states, extracting frequent patterns, computing descriptive statistics (e.g., maximum, median, mode), and performing other numerical operations across extended sequences. For Semantic Aggregation, SummHay [260] requires LCLMs to process the synthesized Haystack of documents and generate a summary. Additionally, some synthetic tasks specifically focus on solely evaluating the model’s piece-by-piece processing capability across long context, such as stacked news labeling [418] and stacked typo detection [418].

Long context reasoning refers to LCLMs’ ability to perform logical inference over information distributed across long contexts. While both reasoning and aggregation involve identifying and processing multiple pieces of information, reasoning further emphasizes the logical deduction and inference process rather than mere information collection and summarization.

• •

  Categories In the realm of long context comprehension, Reasoning can be further categorized into Parallel Reasoning and Iterative Reasoning. Parallel Reasoning involves gathering all relevant information first before conducting the reasoning process, while Iterative Reasoning requires a step-by-step approach, where each reasoning step informs the next information gathering target, forming a continuous cycle until reaching the final conclusion.

• •

  Synthetic Tasks In this aspect, multi-needle reasoning serves as a prototypical synthetic task that emphasizes both parallel reasoning and iterative reasoning capabilities. It requires models to reason across multiple needles distributed throughout a long document [284, 256]. These needles are typically logically related facts extracted from existing reasoning benchmarks. For example, the BABILong benchmark [256] utilizes samples from the bAbI dataset [558] as needles, while NeedleBench [284] opts for the R4C dataset [206].

Real-world adaptation represents the highest level in the capability hierarchy for long context comprehension, where models must effectively integrate and apply their fundamental capabilities (language modeling) and core capabilities (retrieval, aggregation, and reasoning) to address practical challenges. Unlike synthetic tasks that evaluate specific capabilities in controlled settings, real-world tasks present complex scenarios that often require multiple capabilities working in concert. These tasks not only test models’ individual capabilities but also their ability to appropriately combine and deploy these capabilities based on task requirements. Introductions to the most representative real-world tasks involving long context comprehension are provided below. For each task, we use stars to denote which of the three core capabilities - retrieval $\blacksquare$, aggregation $\blacksquare$, and reasoning $\blacksquare$ - the task tends to rely more heavily on.

• •

  Question Answering [$\blacksquare$ / $\blacksquare$ / $\blacksquare$] The Question Answering task requires models to provide accurate answers based on real-world long contexts and queries. These questions and contexts span diverse domains, including literature (NarrativeQA [246], QuALITY [393], LStQA [418], NoCha [239]), academic papers (Qasper [95]), encyclopedias (WikiQA [615], HotpotQA [617], 2WikiMultihopQA [184], DuReader [177], AltQA [391]), conversation history (LCvMem [418]), financial reports (DocFinQA [427]), structured data (Table QA [661]), hybrid scenarios (MultiFieldQA [17]), etc. Notably, due to the diversity of questions and domains, different QA tasks may emphasize different aspects of long context capabilities - some requiring primarily information retrieval, others focusing on information aggregation across multiple parts, and yet others demanding complex reasoning over the aggregated information.

• •

  Summarization [$\blacksquare$] Summarizing key information from long sequences of text has long been a crucial research area in NLP. Throughout the development of this field, researchers have created diverse summarization datasets spanning nearly all sorts of relevant domains, including novels ([663], LStSum [418], SQuALITY [524]), government reports (GovReport [196]), meeting scripts (QMSum [684], VCSUM [567]), news articles (MultiNews [118]), patents (BigPatent [452]), screenplays (SummScreen [59]), legal documents (MultiLexSum [454]), and more. Notably, long context summarization also stands as one of the primary application scenarios for long context models.

• •

  Document Retrieval and Reranking [$\blacksquare$ / $\blacksquare$] Document Retrieval and Reranking are crucial components of modern Information Retrieval (IR) systems, responsible for retrieving relevant documents from a pool of candidates and reordering them based on their relevance to the query, respectively. The advent of LCLMs has enabled a new paradigm where models directly process all candidate documents and produce their rankings in a generative manner [484, 349, 536]. Conversely, retrieval and reranking performance can serve as indicators of these models’ long context capabilities, with the former emphasizing the ability to retrieve relevant information and the later focusing on reasoning across different portions of long context [264, 625]. For example, the HELMET benchmark includes a Reranking test where candidate documents are sampled from the MSMARCO [381] retrieval dataset, and LCLMs are challenged with generating top-10 document IDs ranked by relevance. Beyond relevance-based reranking, the task encompasses broader forms of sequence organization, such as timeline ordering [540, 279, 651] and paragraph order reconstruction [107].

• •

  Retrieval-Augmented Generation [$\blacksquare$] Retrieval-augmented generation enhances the accuracy and reliability of generative models by providing them with a corpus of facts retrieved from external sources. In long context scenarios, where the input corpus can extend up to 1M tokens [264], relevant information becomes more dispersed and sparse, challenging LCLMs’ ability to aggregate information across extended contexts. Existing evaluation approaches [264, 625] primarily employ open-domain QA to assess LCLMs’ performance in retrieval-augmented generation. These evaluations typically process open-domain QA datasets such as Natural Questions [258], TriviaQA [232], and HotpotQA [617] by concatenating their passages into a large corpus. The model input consists of this corpus along with queries targeting specific passages, requiring LCLMs to locate relevant information within the extended corpus to generate appropriate answers.

• •

  In-Context Learning [$\blacksquare$] With extended context window size, In-Context Learning (ICL) expands the scale of demonstration examples from dozens, as seen in traditional ICL frameworks, to hundreds or even thousands. This scenario challenges LCLMs to generate precise predictions based on this substantially larger set of examples[290, 599, 264, 27]. Recent empirical studies have revealed that existing LCLMs exhibit several limitations in this setting: they demonstrate significant performance degradation beyond certain context lengths, show susceptibility to example ordering effects, display recency bias by favoring predictions aligned with labels presented near the sequence terminus, etc. [290, 599] These findings indicate that there remains a substantial journey ahead for LCLMs to fully unleash the potential of ICL.

• •

  Code Relevant Real-World Tasks [$\blacksquare$] Processing repository-level code represents a particularly compelling application scenario that necessitates LCLMs. Among various specific tasks, code completion has garnered significant attention due to its comprehensive reflection of a model’s capability to process entire repositories [30, 323, 642, 225]. Additionally, several tasks address other potential challenges in repository-level code processing, including CI builds repair, commit message generation, bug localization, module summarization, execution simulation, code translation, etc. [30, 663, 550]. Despite significant advances, the intrinsic complexity of repository-level code still poses substantial challenges for current LCLMs in addressing real-world code processing requirements, as demonstrated by their performance limitations on SWE-Bench [225].

From a high-level perspective, an effective long context comprehension benchmark should satisfy three key requirements: coverage of sample lengths matching models’ context windows, evaluation of fundamental long context modeling capabilities, and assessment of downstream task performance. Given that current LCLMs typically feature context windows exceeding 128k tokens, benchmarks primarily focusing on documents under 32k tokens may inadequately assess the capabilities of mainstream LCLMs. For fundamental capabilities, including retrieval, aggregation and reasoning as discussed in Section 6.1.1, the main challenges lie in the dispersion of key information and the difficulty of extraction [152], with more dispersed and semantically oriented (rather than literal) information posing greater challenges [370]. Although synthetic tasks are particularly well suited for evaluating these aspects, research indicates that excellence in synthetic tasks alone does not guarantee downstream competence. Therefore, comprehensive benchmarks must incorporate specific downstream long context tasks, particularly question answering, summarization, RAG, and in-context learning, which serve as a relatively complete proxy for real-world applications. A final notable trend in evaluation methodology is the increasing formulation of tasks such as multiple choice question answering, allowing direct performance assessment through accuracy measurements.

In long-form generation, QA mainly refers to long-form QA. ELI5 [119] is the first large-scale benchmark for long-form QA, where the task is to generate multi-sentence explanations in response to open-ended questions sourced from Reddit. Similarly, MS-NLG is a subset of MS MARCO [381] that focuses on natural language generation. In addition to general long-form QA, ASQA [475] and QASA [267], which focus on ambiguous questions and scientific questions, respectively. In the realm of LLMs, ExpertQA [355] and ProxyQA [490] are high-quality long-form QA benchmarks with questions crafted by experts across various fields, along with expert-verified golden answers. Retrieval Augmented Generation (RAG) is an important application of LLMs, CLAPNQ [432] and Long2RAG [412] are benchmarks designed to evaluate LLM-based RAG systems. Besides, generating long-form and well-supported responses is particularly essential in high-stakes domains such as medicine [189, 213]. Although LLMs have demonstrated strong performance in many tasks, they often contain factual errors when generating long responses [556]. Min et al. [365] propose a novel framework for factual evaluation, comprising three steps: decomposing the response into atomic facts, annotating each fact as supported, unsupported, or irrelevant, and computing the proportion of supported facts. In addition to concerns related to factuality, long-form responses are also susceptible to biases. To evaluate fairness in long-form generation, Jeung et al. [214] propose an evaluation framework that covers 14 topics and 10 demographic axes.

Summarization is a classic task in long context modeling. As input documents become increasingly numerous and lengthy, the length of summaries has also grown significantly. Compared to early summarization benchmarks where summaries were around 50 words long [378], recent summarization benchmarks feature much longer summaries. As a result, summarization not only evaluates a model’s ability to comprehend and condense long input documents but also tests its capability for generating long-form summaries. Multi-News [118] is the first large-scale multi-document news summarization benchmark, collected from newser.com. AQUAMUSE [252] proposes a scalable method to automatically mine summarizations from Natural Questions [258] and Common Crawl. LCFO [88] is a human-annotated benchmark for evaluating long context summarization and summary expansion, consisting of 252 long documents from diverse domains.

Apart from using questions as inputs for LLMs, instructions and other forms of prompts can also serve as inputs. Such tasks prioritize the evaluation of a model’s ability to follow instructions, encompassing both straightforward instruction adherence and more complex, constrained instruction-following. To evaluate the ability of LLMs in structured problem-solving, DoLoMiTes [354] consists of 519 domain-specific long-form methodical tasks with 1,857 examples collected from experts across 25 fields. Similarly, LongGenBench-SUTD⁶⁶ 6 In fact, there are two benchmarks named LongGenBench [576, 326]. To distinguish them, we add the suffix of the respective primary institution. [576] and LongProc [624] are designed to evaluate the capability of LLMs to generate high-quality long-form text while adhering to complex instructions. Besides, LongLaMP [253] is a benchmark for personalized generation, covering four tasks: personalized email completion, abstract generation, review writing, and topic writing. Recently, increasing attention has been given to aligning LLMs with long-form output instructions, such as creative writing and story generation. To enhance instruction tuning for long-form generation, LongForm-C [248] gathers instruction-following examples via reverse instructions on C4 and English Wikipedia. Bai et al. [19] propose a long instruction-following dataset LongWriter-6K aimed at extending the output length of existing models to over 10,000 words. Similarly, Suri [405] is a benchmark consisting of 20,000 long-form human-written texts, each paired with backtranslated instructions with multiple constraints. Quan et al. [419] propose Self-Lengthen, an iterative training framework that utilizes LLMs’ intrinsic knowledge without relying on auxiliary data. To evaluate its effectiveness, they introduce LongGen, a benchmark for evaluating LLMs’ long-form generation capabilities in Chinese and English across various tasks, with length-constrained user instructions and corresponding responses.

Some benchmarks include multiple task types to enable more comprehensive evaluation. Facts Grounding [211] evaluates LLMs’ ability to generate factually accurate long-form text based on a given context and user requests, encompassing tasks QA, summarization, and document rewriting. HelloBench [420] is a comprehensive benchmark for evaluating LLM’s long-form generation capabilities, consisting of 647 samples across 38 subcategories and 5 tasks (QA, summarization, chat, completion, generation), sourced from real-world scenarios and publicly available datasets.

Automatic metrics are evaluation methods that evaluate model-generated responses without the need for additional human involvement or the use of LLMs. Most traditional metrics are classified as automatic metrics. Existing automatic metrics in the field of long-form generation can be categorized into four types:

• •

  Semantic Aspect: This type of metrics evaluates the semantic aspect of the response. Metrics like ROUGE [305] and BLEU [394] are the most commonly used and were initially developed for summarization tasks and machine translation tasks. ROUGE and BLEU measure the semantic similarity between the response and the reference answer, providing an indication of the generation quality of LLMs. They are adopted as evaluation metrics in 13 out of 28 benchmarks. Similarly, METEOR [21] is also used to measure the semantic similarity and is applied in LongForm-C [248] and LongLamp [253]. Besides, BERTScore [657] is a metric that evaluates the semantic similarity of two texts using a finetuned BERT model [213].

• •

  Repetitive Aspect: These metrics evaluate the repetitiveness or fluency of the responses. For instance, Perplexity (PPL) [303] calculates the overall probability of the response. Lower perplexity indicates higher fluency and suggests the sentence is more likely to be generated by an LLM. Metrics like Repetition [450] and Distinct [281] are used to measure repetitiveness. Repetition-n calculates the percentage of n-grams that appear at least twice in the text, while Distinct-n quantifies text diversity by computing the number of unique n-grams in the output. These metrics are used in the LOT-OutGen [161] and LCFO [88].

• •

  Accuracy-Based: Accuracy is more commonly used in short-form generation scenarios like multiple-choice QA. In long-form generation, LongGenBench-HKUST [326] combines multiple questions into a single input, requiring the model to generate responses for all of them. Each response is paired with a corresponding standard answer (either a specific option or a number), making it well-suited for accuracy-based evaluation. Wu et al. [576] propose the CR and STIC metrics, both designed to measure the accuracy of responses in specific aspects. In addition, QASA [267] uses the standard accuracy as the evaluation metric.

• •

  Task-Specific: Due to the varying focus of different benchmarks, task-specific metrics are often used. For example, ExpertQA [355] uses QAFactEval [117] to evaluate the factual consistency of the responses, while ASQA [475] uses Disambiguation Metrics to evaluate whether the response resolves ambiguities. Retrieval-oriented metrics, such as nDCG [432] and KPR [412], are used in CLAPNQ and Long2RAG to evaluate the effectiveness of the LLM-based RAG systems.

Although automatic metrics are convenient, recent studies have shown that the correlation between automatic metrics and human evaluation is quite low [251, 596]. To achieve more accurate evaluations, the best approach is to establish standardized evaluation criteria and have human evaluators evaluate the responses of LLMs [433]. ELI5 [119], ASQA [475], Long2RAG [412], AQUAMUSE [252], Multi-News [118], LongForm-C [248], LongBench-Write [19], LOT-OutGen [161], and HelloBench [420] have all conducted experiments involving human evaluation or provided human evaluation criteria.

The main drawback of human evaluation lies in its time-consuming, labor-intensive, and inefficient nature. Recently, there has been a growing trend of using strong-performing LLMs to replace humans for fine-grained evaluations [682]. This strategy is also known as LLM-as-a-Judge. In ProxyQA [490], LLMs are used to evaluate whether the responses to proxy questions are correct. In LFMedQA [213], GPT-4o and Claude-3.5 are used for pairwise comparisons. In HelloBench [420], predefined checklists are set for each sub-task, and the LLM determines whether each checklist is satisfied. The overall score is calculated using human-correlated weights.

The data source lays the foundation for the quality of a benchmark. In Section 6.2.2, we categorize the data sources for long-form generation into five types: web-sourced data, user-sourced data, synthetic data, PADs, and CrowdSrc. Web-sourced data is advantageous in terms of scale and accessibility, but it often suffers from low quality. Similarly, while user-sourced data is directly obtained from users, its quality can also vary significantly. Synthetic data has the advantage of making evaluation easier, as it allows for the use of automatic metrics with high evaluation accuracy. However, such data often lacks alignment with real-world scenarios. Previous available datasets pose risks such as data leakage. While CrowdSrc provides high-quality data, it may still deviate from practical use cases. It is clear that each data source has its own limitations. So, what type of data can be considered good for long-form generation? We argue that user-sourced data combined with detailed post-processing is a promising direction for future research. Specifically, the optimization goal of LLMs is to serve users, which makes understanding the real issues users encounter critical. As such, obtaining first-hand interaction data directly from users is ideal. However, user-sourced data often contains low-quality samples, making data filtering a top priority. We believe that human involvement in the data post-processing phase is essential to achieve better outcomes. As long as data quality is ensured, having slightly fewer evaluation samples is unlikely to introduce significant evaluation bias.

RoPE [476] explicitly encodes the relative position between tokens into attention. Recall that for a model with even hidden size $d$, the RoPE embedding [477] for absolute position $n$ would be:

The $\beta$ in Equation 10 is the rotation base. The trigonometric functions could be associated with hypothetical “wave lengths” $\lambda_{2i}=\left\{2\pi\beta^{\frac{2i}{d}}\right\}_{i=0}^{\frac{d}{2}-1}$, which are indicative of the model’s context capacity. For a given $\beta$, the wave lengths range from $2\pi$ to $2\pi\beta$ [402]. A typical choice for $\beta$ is 10000 [476], yielding max wave length $\sim 63K$. The hypothetical “frequency” associated with a wave length is $f_{2i}=\frac{2\pi}{\lambda_{2i}}$. Therefore, the larger the base, the larger the wavelength and the lower the frequency is.