496 lines
35 KiB
XML
496 lines
35 KiB
XML
<?xml version='1.0' encoding='UTF-8'?>
|
|
<feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:arxiv="http://arxiv.org/schemas/atom" xmlns="http://www.w3.org/2005/Atom">
|
|
<id>https://arxiv.org/api/mxbHxwasnZN4YmADZaqgiXpYXpk</id>
|
|
<title>arXiv Query: search_query=all:long OR all:context&id_list=&start=0&max_results=15</title>
|
|
<updated>2026-10-27T16:53:12Z</updated>
|
|
<link href="https://arxiv.org/api/query?search_query=all:long+OR+all:context&start=0&max_results=15&id_list=" type="application/atom+xml"/>
|
|
<opensearch:itemsPerPage>15</opensearch:itemsPerPage>
|
|
<opensearch:totalResults>355402</opensearch:totalResults>
|
|
<opensearch:startIndex>0</opensearch:startIndex>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2602.02159v1</id>
|
|
<title>Focus-dLLM: Accelerating Long-Context Diffusion LLM Inference via Confidence-Guided Context Focusing</title>
|
|
<updated>2026-02-02T14:36:10Z</updated>
|
|
<link href="https://arxiv.org/abs/2602.02159v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2602.02159v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Diffusion Large Language Models (dLLMs) deliver strong long-context processing capability in a non-autoregressive decoding paradigm. However, the considerable computational cost of bidirectional full attention limits the inference efficiency. Although sparse attention is promising, existing methods remain ineffective. This stems from the need to estimate attention importance for tokens yet to be decoded, while the unmasked token positions are unknown during diffusion. In this paper, we present Focus-dLLM, a novel training-free attention sparsification framework tailored for accurate and efficient long-context dLLM inference. Based on the finding that token confidence strongly correlates across adjacent steps, we first design a past confidence-guided indicator to predict unmasked regions. Built upon this, we propose a sink-aware pruning strategy to accurately estimate and remove redundant attention computation, while preserving highly influential attention sinks. To further reduce overhead, this strategy reuses identified sink locations across layers, leveraging the observed cross-layer consistency. Experimental results show that our method offers more than $29\times$ lossless speedup under $32K$ context length. The code is publicly available at: https://github.com/Longxmas/Focus-dLLM</summary>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-02-02T14:36:10Z</published>
|
|
<arxiv:primary_category term="cs.CL"/>
|
|
<author>
|
|
<name>Lingkun Long</name>
|
|
</author>
|
|
<author>
|
|
<name>Yushi Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Shihao Bai</name>
|
|
</author>
|
|
<author>
|
|
<name>Ruihao Gong</name>
|
|
</author>
|
|
<author>
|
|
<name>Jun Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Ao Zhou</name>
|
|
</author>
|
|
<author>
|
|
<name>Jianlei Yang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.00749v1</id>
|
|
<title>ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents</title>
|
|
<updated>2026-09-01T05:27:38Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.00749v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.00749v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-horizon large language model (LLM) agents require context assembly: the runtime must decide what to include in each prompt, in what order, and when to compact history under a hard context-window budget and a byte-sensitive prompt cache. In production agentic systems, this logic is scattered across prompt builders, ad hoc compaction routines, cache-break workarounds, and per-provider shims. We argue that context assembly is structurally isomorphic to query execution in a relational database: both execute under a hard budget, exploit a tiered cache, and leverage statistics. We adopt this discipline in ContextPipe: a five-phase pipeline (Plan Bind Optimize Execute Feedback) backed by a structured data-source catalog, a deterministic cache-aware optimizer, and an EXPLAIN ANALYZE trace. We show that context in ContextPipe is auditable, replayable, and failure-isolated. A preliminary evaluation using the SWE-bench Pro Qutebrowser subset shows that, compared with the append-only context construction policy, ContextPipe reduces total token volume by 31%, LLM calls by 23%, and response time by 9%, at the cost of a lower KV cache-hit ratio.</summary>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.DB" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-01T05:27:38Z</published>
|
|
<arxiv:primary_category term="cs.AI"/>
|
|
<author>
|
|
<name>Peng Xu</name>
|
|
</author>
|
|
<author>
|
|
<name>Zuyu Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yuze Sun</name>
|
|
</author>
|
|
<author>
|
|
<name>Feng Tian</name>
|
|
</author>
|
|
<author>
|
|
<name>Long Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Chen Zhang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.31382v1</id>
|
|
<title>Highlight-Then-Summarize: Learning to Compress Evidence for Long-Context Understanding</title>
|
|
<updated>2026-09-25T15:16:35Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.31382v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.31382v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-context understanding requires large language models (LLMs) to reason over lengthy documents, conversations, and code, yet task-relevant evidence is often sparse and scattered amid substantial irrelevant and redundant content. We propose Highlight-Then-Summarize (H2S), a compress-then-reason paradigm that first identifies source-grounded, question-relevant evidence and then integrates it into a compact, question-conditioned summary before producing the final answer. To train this behavior, we construct H2S-Dataset, comprising 6,647 examples from 11 benchmark families with an average context length of 43.9K tokens, and introduce H2S-RL, which provides process-level rewards for evidence selection and summary construction in addition to final-answer correctness. We evaluate on H2S-Bench, a seven-task long-context suite. Under a shared 128K input and 4K output budget, H2S-14B achieves an average score of 32.60, outperforming Qwen3.8-27B by 10.17 points and obtaining the strongest overall result among the evaluated open-source models. H2S-14B also achieves the highest Evidence-Summary Quality score and retains 97.1% of its 16K-budget performance with only a 4K output budget. These results show that explicitly selecting and integrating evidence improves long-context reasoning while enabling more compact generation.</summary>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-25T15:16:35Z</published>
|
|
<arxiv:comment>23 pages, 13 figures. Zhaoyuan Xia and Qinghongbing Xie contributed equally. Corresponding authors: Dai Dai, Tong Mo, and Long Zeng. Code and data are available at https://github.com/X-Luffy/Highlight-Then-Summarize</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CL"/>
|
|
<author>
|
|
<name>Zhaoyuan Xia</name>
|
|
<arxiv:affiliation>Peking University</arxiv:affiliation>
|
|
<arxiv:affiliation>Baidu Inc</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Qinghongbing Xie</name>
|
|
<arxiv:affiliation>Tsinghua University</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Yung Xiang Hue</name>
|
|
<arxiv:affiliation>Tsinghua University</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Jianguang Jiang</name>
|
|
<arxiv:affiliation>Baidu Inc</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Gaofeng Lu</name>
|
|
<arxiv:affiliation>Baidu Inc</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Zhenyu Jiao</name>
|
|
<arxiv:affiliation>Baidu Inc</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Xing Yuan</name>
|
|
<arxiv:affiliation>Baidu Inc</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Dai Dai</name>
|
|
<arxiv:affiliation>Baidu Inc</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Tong Mo</name>
|
|
<arxiv:affiliation>Peking University</arxiv:affiliation>
|
|
</author>
|
|
<author>
|
|
<name>Long Zeng</name>
|
|
<arxiv:affiliation>Tsinghua University</arxiv:affiliation>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.06924v1</id>
|
|
<title>A$^2$RD: Agentic Autoregressive Diffusion for Long Video Consistency</title>
|
|
<updated>2026-05-07T20:35:46Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.06924v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.06924v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Synthesizing consistent and coherent long video remains a fundamental challenge. Existing methods suffer from semantic drift and narrative collapse over long horizons. We present A$^2$RD, an Agentic Auto-Regressive Diffusion architecture that decouples creative synthesis from consistency enforcement. A$^2$RD formulates long video synthesis as a closed-loop process that synthesizes and self-improves video segment-by-segment through a Retrieve--Synthesize--Refine--Update cycle. It comprises three core components: (i) Multimodal Video Memory that tracks video progression across modalities; (ii) Adaptive Segment Generation that switches among generation modes for natural progression and visual consistency; and (iii) Hierarchical Test-Time Self-Improvement that self-improves each segment at frame and video levels to prevent error propagation. We further introduce LVBench-C, a challenging benchmark with non-linear entity and environment transitions to stress-test long-horizon consistency. Across public and LVBench-C benchmarks spanning one- to ten-minute videos, A$^2$RD outperforms state-of-the-art baselines by up to 30% in consistency and 20% in narrative coherence. Human evaluations corroborate these gains while also highlighting notable improvements in motion and transition smoothness.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-07T20:35:46Z</published>
|
|
<arxiv:comment>Project page: http://dxlong2000.github.io/AARD</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Do Xuan Long</name>
|
|
</author>
|
|
<author>
|
|
<name>Yale Song</name>
|
|
</author>
|
|
<author>
|
|
<name>Min-Yen Kan</name>
|
|
</author>
|
|
<author>
|
|
<name>Tomas Pfister</name>
|
|
</author>
|
|
<author>
|
|
<name>Long T. Le</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/cmp-lg/9701001v1</id>
|
|
<title>Exploiting Context to Identify Lexical Atoms -- A Statistical View of Linguistic Context</title>
|
|
<updated>1997-01-02T16:35:24Z</updated>
|
|
<link href="https://arxiv.org/abs/cmp-lg/9701001v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/cmp-lg/9701001v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary> Interpretation of natural language is inherently context-sensitive. Most words in natural language are ambiguous and their meanings are heavily dependent on the linguistic context in which they are used. The study of lexical semantics can not be separated from the notion of context. This paper takes a contextual approach to lexical semantics and studies the linguistic context of lexical atoms, or "sticky" phrases such as "hot dog". Since such lexical atoms may occur frequently in unrestricted natural language text, recognizing them is crucial for understanding naturally-occurring text. The paper proposes several heuristic approaches to exploiting the linguistic context to identify lexical atoms from arbitrary natural language text.</summary>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>1997-01-02T16:35:24Z</published>
|
|
<arxiv:comment>10 pages, postscript file, to appear in Proceedings of International and Interdisciplinary Conference on Modelling and Using Context (CONTEXT-97)</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CL"/>
|
|
<arxiv:journal_ref>Proceedings of the International and Interdisciplinary Conference on Modelling and Using Context (CONTEXT-97), Rio de Janeiro, Brzil, Feb. 4-6, 1997. 119-129.</arxiv:journal_ref>
|
|
<author>
|
|
<name>Chengxiang Zhai</name>
|
|
<arxiv:affiliation>Carnegie Mellon University</arxiv:affiliation>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2503.10589v1</id>
|
|
<title>Long Context Tuning for Video Generation</title>
|
|
<updated>2025-03-13T17:40:07Z</updated>
|
|
<link href="https://arxiv.org/abs/2503.10589v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2503.10589v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Recent advances in video generation can produce realistic, minute-long single-shot videos with scalable diffusion transformers. However, real-world narrative videos require multi-shot scenes with visual and dynamic consistency across shots. In this work, we introduce Long Context Tuning (LCT), a training paradigm that expands the context window of pre-trained single-shot video diffusion models to learn scene-level consistency directly from data. Our method expands full attention mechanisms from individual shots to encompass all shots within a scene, incorporating interleaved 3D position embedding and an asynchronous noise strategy, enabling both joint and auto-regressive shot generation without additional parameters. Models with bidirectional attention after LCT can further be fine-tuned with context-causal attention, facilitating auto-regressive generation with efficient KV-cache. Experiments demonstrate single-shot models after LCT can produce coherent multi-shot scenes and exhibit emerging capabilities, including compositional generation and interactive shot extension, paving the way for more practical visual content creation. See https://guoyww.github.io/projects/long-context-video/ for more details.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-03-13T17:40:07Z</published>
|
|
<arxiv:comment>Project Page: https://guoyww.github.io/projects/long-context-video/</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Yuwei Guo</name>
|
|
</author>
|
|
<author>
|
|
<name>Ceyuan Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Ziyan Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhibei Ma</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhijie Lin</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhenheng Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Dahua Lin</name>
|
|
</author>
|
|
<author>
|
|
<name>Lu Jiang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/1006.3049v2</id>
|
|
<title>Long paths and cycles in subgraphs of the cube</title>
|
|
<updated>2015-03-20T10:51:01Z</updated>
|
|
<link href="https://arxiv.org/abs/1006.3049v2" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/1006.3049v2" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Let $Q_n$ denote the graph of the $n$-dimensional cube with vertex set $\{0,1\}^n$ in which two vertices are adjacent if they differ in exactly one coordinate. Suppose $G$ is a subgraph of $Q_n$ with average degree at least $d$. How long a path can we guarantee to find in $G$?
|
|
Our aim in this paper is to show that $G$ must contain an exponentially long path. In fact, we show that if $G$ has minimum degree at least $d$ then $G$ must contain a path of length $2^d-1$. Note that this bound is tight, as shown by a $d$-dimensional subcube of $Q_n$. We also obtain the slightly stronger result that $G$ must contain a cycle of length at least $2^d$ and prove analogous results for other `product-type' graphs.</summary>
|
|
<category term="math.CO" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2010-06-15T18:40:15Z</published>
|
|
<arxiv:comment>27 pages, 6 figures</arxiv:comment>
|
|
<arxiv:primary_category term="math.CO"/>
|
|
<author>
|
|
<name>Eoin Long</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2508.06447v2</id>
|
|
<title>SlimInfer: Accelerating Long-Context LLM Inference via Dynamic Token Pruning</title>
|
|
<updated>2025-11-24T06:26:25Z</updated>
|
|
<link href="https://arxiv.org/abs/2508.06447v2" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2508.06447v2" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-context inference for Large Language Models (LLMs) is heavily limited by high computational demands. While several existing methods optimize attention computation, they still process the full set of hidden states at each layer, limiting overall efficiency. In this work, we propose SlimInfer, an innovative framework that aims to accelerate inference by directly pruning less critical prompt tokens during the forward pass. Our key insight is an information diffusion phenomenon: As information from critical tokens propagates through layers, it becomes distributed across the entire sequence. This diffusion process suggests that LLMs can maintain their semantic integrity when excessive tokens, even including these critical ones, are pruned in hidden states. Motivated by this, SlimInfer introduces a dynamic fine-grained pruning mechanism that accurately removes redundant tokens of hidden state at intermediate layers. This layer-wise pruning naturally enables an asynchronous KV cache manager that prefetches required token blocks without complex predictors, reducing both memory usage and I/O costs. Extensive experiments show that SlimInfer can achieve up to $\mathbf{2.53\times}$ time-to-first-token (TTFT) speedup and $\mathbf{1.88\times}$ end-to-end latency reduction for LLaMA3.1-8B-Instruct on a single RTX 4090, without sacrificing performance on LongBench. Our code is available at https://github.com/Longxmas/SlimInfer.</summary>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-08-08T16:42:38Z</published>
|
|
<arxiv:primary_category term="cs.CL"/>
|
|
<author>
|
|
<name>Lingkun Long</name>
|
|
</author>
|
|
<author>
|
|
<name>Rubing Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yushi Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Desheng Hui</name>
|
|
</author>
|
|
<author>
|
|
<name>Ao Zhou</name>
|
|
</author>
|
|
<author>
|
|
<name>Jianlei Yang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/1301.2195v1</id>
|
|
<title>Long geodesics in subgraphs of the cube</title>
|
|
<updated>2013-01-10T17:35:50Z</updated>
|
|
<link href="https://arxiv.org/abs/1301.2195v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/1301.2195v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>A path in the hypercube $Q_n$ is said to be a geodesic if no two of its edges are in the same direction. Let $G$ be a subgraph of $Q_n$ with average degree $d$. How long a geodesic must $G$ contain? We show that $G$ must contain a geodesic of length $d$. This result, which is best possible, strengthens a theorem of Feder and Subi. It is also related to the `antipodal colourings' conjecture of Norine.</summary>
|
|
<category term="math.CO" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2013-01-10T17:35:50Z</published>
|
|
<arxiv:comment>8 pages</arxiv:comment>
|
|
<arxiv:primary_category term="math.CO"/>
|
|
<author>
|
|
<name>Imre Leader</name>
|
|
</author>
|
|
<author>
|
|
<name>Eoin Long</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2505.09561v2</id>
|
|
<title>Learning Long-Context Diffusion Policies via Past-Token Prediction</title>
|
|
<updated>2025-05-19T20:37:41Z</updated>
|
|
<link href="https://arxiv.org/abs/2505.09561v2" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2505.09561v2" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Reasoning over long sequences of observations and actions is essential for many robotic tasks. Yet, learning effective long-context policies from demonstrations remains challenging. As context length increases, training becomes increasingly expensive due to rising memory demands, and policy performance often degrades as a result of spurious correlations. Recent methods typically sidestep these issues by truncating context length, discarding historical information that may be critical for subsequent decisions. In this paper, we propose an alternative approach that explicitly regularizes the retention of past information. We first revisit the copycat problem in imitation learning and identify an opposite challenge in recent diffusion policies: rather than over-relying on prior actions, they often fail to capture essential dependencies between past and future actions. To address this, we introduce Past-Token Prediction (PTP), an auxiliary task in which the policy learns to predict past action tokens alongside future ones. This regularization significantly improves temporal modeling in the policy head, with minimal reliance on visual representations. Building on this observation, we further introduce a multistage training strategy: pre-train the visual encoder with short contexts, and fine-tune the policy head using cached long-context embeddings. This strategy preserves the benefits of PTP while greatly reducing memory and computational overhead. Finally, we extend PTP into a self-verification mechanism at test time, enabling the policy to score and select candidates consistent with past actions during inference. Experiments across four real-world and six simulated tasks demonstrate that our proposed method improves the performance of long-context diffusion policies by 3x and accelerates policy training by more than 10x.</summary>
|
|
<category term="cs.RO" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-05-14T17:00:47Z</published>
|
|
<arxiv:comment>Videos are available at https://long-context-dp.github.io</arxiv:comment>
|
|
<arxiv:primary_category term="cs.RO"/>
|
|
<author>
|
|
<name>Marcel Torne</name>
|
|
</author>
|
|
<author>
|
|
<name>Andy Tang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yuejiang Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Chelsea Finn</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2507.06415v3</id>
|
|
<title>PERK: Long-Context Reasoning as Test-Time Learning</title>
|
|
<updated>2026-08-31T15:26:53Z</updated>
|
|
<link href="https://arxiv.org/abs/2507.06415v3" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2507.06415v3" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-context reasoning requires accurately identifying relevant information in extensive, noisy input contexts. In this work, we propose PERK (Parameter Efficient Reasoning over Knowledge), a scalable approach for learning to encode long contexts using gradient updates at test time. Specifically, PERK employs two nested optimization loops in a meta-training phase. The inner loop rapidly encodes contexts into a low-rank adapter (LoRA) that serves as a parameter-efficient memory module for the base model. Concurrently, the outer loop learns to use the updated adapter to accurately recall and reason over relevant information from the encoded long context. Our evaluations on several long-context reasoning tasks show that PERK significantly outperforms the standard long-context finetuning, achieving average absolute performance gains of up to 20% for Qwen-2.5 (0.5B & 7B) on synthetic and real-world long-context reasoning. PERK also maintains its advantages across model scales and families. Compared to specialized long-context LLMs, PERK matches or surpasses their performance. Finally, our analyses show PERK is more robust to reasoning complexity, length extrapolation, and the positions of relevant information in contexts. https://perk-long-context.web.app</summary>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-07-08T21:38:45Z</published>
|
|
<arxiv:comment>ICLR 2026, 10 pages, 7 figures, test-time learning, long context, meta-learning</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CL"/>
|
|
<arxiv:journal_ref>The Fourteenth International Conference on Learning Representations, 2026</arxiv:journal_ref>
|
|
<author>
|
|
<name>Zeming Chen</name>
|
|
</author>
|
|
<author>
|
|
<name>Angelika Romanou</name>
|
|
</author>
|
|
<author>
|
|
<name>Gail Weiss</name>
|
|
</author>
|
|
<author>
|
|
<name>Antoine Bosselut</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2502.17129v2</id>
|
|
<title>Thus Spake Long-Context Large Language Model</title>
|
|
<updated>2025-11-11T10:41:41Z</updated>
|
|
<link href="https://arxiv.org/abs/2502.17129v2" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2502.17129v2" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long context is an important topic in Natural Language Processing (NLP), running through the development of NLP architectures, and offers immense opportunities for Large Language Models (LLMs), giving LLMs the lifelong learning potential akin to humans. Unfortunately, the pursuit of a long context is accompanied by numerous obstacles. Nevertheless, long context remains a core competitive advantage for LLMs. In the past two years, the context length of LLMs has achieved a breakthrough extension to millions of tokens. Moreover, research on long-context LLMs has expanded beyond length extrapolation to a comprehensive focus on architecture, infrastructure, training, and evaluation technologies.
|
|
Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy between the journey of extending the context of LLM and the attempts of humans to transcend their mortality. In this survey, we will illustrate how LLM struggles between the tremendous need for a longer context and its equal need to accept the fact that it is ultimately finite. To achieve this, we give a global picture of the lifecycle of long-context LLMs from four perspectives: architecture, infrastructure, training, and evaluation, showcasing the full spectrum of long-context technologies. At the end of this survey, we will present 10 unanswered questions currently faced by long-context LLMs. We hope this survey can serve as a systematic introduction to research on long-context LLMs.
|
|
Video: https://www.bilibili.com/video/BV11h9AYoEYj.
|
|
Github: https://github.com/OpenMOSS/Thus-Spake-Long-Context-LLM.</summary>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-02-24T13:19:33Z</published>
|
|
<arxiv:comment>a global picture of the lifecycle of long-context LLMs from four perspectives: architecture, infrastructure, training, and evaluation</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CL"/>
|
|
<author>
|
|
<name>Xiaoran Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Ruixiao Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Mianqiu Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhigeng Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Yuerong Song</name>
|
|
</author>
|
|
<author>
|
|
<name>Qipeng Guo</name>
|
|
</author>
|
|
<author>
|
|
<name>Siyang He</name>
|
|
</author>
|
|
<author>
|
|
<name>Qiqi Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Linlin Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Qun Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Ziwei He</name>
|
|
</author>
|
|
<author>
|
|
<name>Yaqian Zhou</name>
|
|
</author>
|
|
<author>
|
|
<name>Xuanjing Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Xipeng Qiu</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2511.09138v1</id>
|
|
<title>Trusted Multi-view Learning for Long-tailed Classification</title>
|
|
<updated>2025-11-12T09:26:23Z</updated>
|
|
<link href="https://arxiv.org/abs/2511.09138v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2511.09138v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Class imbalance has been extensively studied in single-view scenarios; however, addressing this challenge in multi-view contexts remains an open problem, with even scarcer research focusing on trustworthy solutions. In this paper, we tackle a particularly challenging class imbalance problem in multi-view scenarios: long-tailed classification. We propose TMLC, a Trusted Multi-view Long-tailed Classification framework, which makes contributions on two critical aspects: opinion aggregation and pseudo-data generation. Specifically, inspired by Social Identity Theory, we design a group consensus opinion aggregation mechanism that guides decision making toward the direction favored by the majority of the group. In terms of pseudo-data generation, we introduce a novel distance metric to adapt SMOTE for multi-view scenarios and develop an uncertainty-guided data generation module that produces high-quality pseudo-data, effectively mitigating the adverse effects of class imbalance. Extensive experiments on long-tailed multi-view datasets demonstrate that our model is capable of achieving superior performance. The code is released at https://github.com/cncq-tang/TMLC.</summary>
|
|
<category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-11-12T09:26:23Z</published>
|
|
<arxiv:comment>This paper has been accepted to AAAI2026</arxiv:comment>
|
|
<arxiv:primary_category term="cs.LG"/>
|
|
<author>
|
|
<name>Chuanqing Tang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yifei Shi</name>
|
|
</author>
|
|
<author>
|
|
<name>Guanghao Lin</name>
|
|
</author>
|
|
<author>
|
|
<name>Lei Xing</name>
|
|
</author>
|
|
<author>
|
|
<name>Long Shi</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2606.12956v2</id>
|
|
<title>SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation</title>
|
|
<updated>2026-10-12T04:39:18Z</updated>
|
|
<link href="https://arxiv.org/abs/2606.12956v2" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2606.12956v2" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-horizon robot mobile manipulation requires continual reasoning about localization, environment changes, and task progress, all of which are challenging to infer from image observations alone. In this paper, we show that conditioning a mobile manipulation policy on a spatiotemporal feature map improves reasoning over long horizons. The map represents the environment and the articulated robot body as neural points in a shared latent space and is updated online from egocentric observations and proprioceptive state. We update the environment neural points using object-level rigid tracking and the robot neural points using forward kinematics. We use our spatiotemporal environment and robot feature (SERF) map as a state input to a vision-language-action (VLA) model by extracting map tokens from multiple reference frames and spatial scales, providing the policy with both local and global context. We demonstrate SERF on BEHAVIOR-1K, a benchmark for long-horizon mobile manipulation in household environments. Experiments show that the SERF VLA policy outperforms image-only baselines, reaches subgoals faster by following more direct trajectories, improves robustness to scene-configuration shifts, and recovers from object-drop failures.</summary>
|
|
<category term="cs.RO" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-06-11T06:29:49Z</published>
|
|
<arxiv:comment>CoRL 2026, project page: https://existentialrobotics.org/serf/</arxiv:comment>
|
|
<arxiv:primary_category term="cs.RO"/>
|
|
<author>
|
|
<name>Sunghwan Kim</name>
|
|
</author>
|
|
<author>
|
|
<name>Byeonghyun Pak</name>
|
|
</author>
|
|
<author>
|
|
<name>Kehan Long</name>
|
|
</author>
|
|
<author>
|
|
<name>Yulun Tian</name>
|
|
</author>
|
|
<author>
|
|
<name>Nikolay Atanasov</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2504.10326v1</id>
|
|
<title>AlayaDB: The Data Foundation for Efficient and Effective Long-context LLM Inference</title>
|
|
<updated>2025-04-14T15:34:26Z</updated>
|
|
<link href="https://arxiv.org/abs/2504.10326v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2504.10326v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>AlayaDB is a cutting-edge vector database system natively architected for efficient and effective long-context inference for Large Language Models (LLMs) at AlayaDB AI. Specifically, it decouples the KV cache and attention computation from the LLM inference systems, and encapsulates them into a novel vector database system. For the Model as a Service providers (MaaS), AlayaDB consumes fewer hardware resources and offers higher generation quality for various workloads with different kinds of Service Level Objectives (SLOs), when comparing with the existing alternative solutions (e.g., KV cache disaggregation, retrieval-based sparse attention). The crux of AlayaDB is that it abstracts the attention computation and cache management for LLM inference into a query processing procedure, and optimizes the performance via a native query optimizer. In this work, we demonstrate the effectiveness of AlayaDB via (i) three use cases from our industry partners, and (ii) extensive experimental results on LLM inference benchmarks.</summary>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.DB" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.IR" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2025-04-14T15:34:26Z</published>
|
|
<arxiv:comment>14 pages, 12 figures, conference</arxiv:comment>
|
|
<arxiv:primary_category term="cs.AI"/>
|
|
<author>
|
|
<name>Yangshen Deng</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhengxin You</name>
|
|
</author>
|
|
<author>
|
|
<name>Long Xiang</name>
|
|
</author>
|
|
<author>
|
|
<name>Qilong Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Peiqi Yuan</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhaoyang Hong</name>
|
|
</author>
|
|
<author>
|
|
<name>Yitao Zheng</name>
|
|
</author>
|
|
<author>
|
|
<name>Wanting Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Runzhong Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Haotian Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Kyriakos Mouratidis</name>
|
|
</author>
|
|
<author>
|
|
<name>Man Lung Yiu</name>
|
|
</author>
|
|
<author>
|
|
<name>Huan Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Qiaomu Shen</name>
|
|
</author>
|
|
<author>
|
|
<name>Rui Mao</name>
|
|
</author>
|
|
<author>
|
|
<name>Bo Tang</name>
|
|
</author>
|
|
</entry>
|
|
</feed>
|