135 lines
277 KiB
JSON
135 lines
277 KiB
JSON
{"abstract": "LLM agents combine models with execution harnesses, so agentic benchmark performance reflects their joint behavior. We argue that harnesses should therefore be evaluated through their interactions with underlying models. We develop a framework combining five complementary measures: (i) Valid Completion Rate (VCR), how reliably agents finish normally and deliver evaluable results, irrespective of task correctness; (ii) Model-Harness Fit, how well a harness supports task achievement through valid completion for a given model; (iii) Cross-Model Fit, a harness's overall suitability across evaluated models; (iv) Interaction Residuals, pairing-specific execution strengths and weaknesses hidden by component-level summaries; and (v) Token Load, the token demands of agent execution, including failed attempts. We evaluate four general-purpose harnesses (OpenCode, Codex CLI, Pi CLI, and DeepAgents) with four core models, plus two fixed native model-harness systems, on six benchmarks covering context management, tool use, long-context planning, multimodal interaction, and artifact handoff. Including native references and stress conditions, 111 evaluation settings yield 52,168 trajectories. Harness rankings vary across models and benchmarks; task performance and valid completion can favor different pairings, and similar task scores can accompany distinct execution profiles. OpenCode ranks first in cross-model performance, while DeepSeek V4 Flash Preview maintains consistently high Mean VCR across harnesses on the shared four-benchmark panel. These findings support evaluating model-harness interactions, training transferable harness-use capabilities in models, and harness designs adapted to the model and task.", "authors": ["Shao Zhang", "Hangfan Zhang", "Yuanyuan Shi", "Hao Li", "Min-Le Su", "Kangcong Li", "Jingping Wang", "Chunjiang Mu", "Chen Zhang", "Yang Chen", "Bo Zhang", "Shuyue Hu"], "base_id": "2610.32483", "categories": ["cs.AI", "cs.CL", "cs.SE"], "id": "2610.32483v1", "pdf_url": "https://arxiv.org/pdf/2610.32483v1", "primary_category": "cs.AI", "published": "2026-10-23T10:57:54Z", "query_hits": ["q1_longctx_multimodal"], "seq": 1, "title": "When Agents Fail: Model or Harness--Who's to Blame?", "updated": "2026-10-23T10:57:54Z"}
|
|
{"abstract": "Visual document understanding is essential for acquiring knowledge from visually rich documents and is attracting growing research interest. Existing surveys emphasize architectures or technical approaches, with limited attention to how task bottlenecks shape technical choices and research priorities. We present a task-centric survey that decomposes long-document visual question answering (VQA) into four components: document parsing, evidence organization and retrieval, generator organization and learning, and evaluation. For each, we develop a fine-grained taxonomy and trace motivations, technical responses, and cross-component connections. We focus on explicit retrieval while covering direct reading. Our temporal analysis traces expanding priorities from multimodal content understanding to sufficient, high-fidelity evidence acquisition and sustained evidence management. We outline future directions in evidence sufficiency, verifiable and correctable understanding, and budget-aware coordination, with an outlook on continual document understanding.", "authors": ["Yufei Tong", "Guanjie Cheng", "Tianle Wu", "Leilei Zhang", "Hang Zheng", "Peihan Wu", "Jianwei Yin", "Shuiguang Deng"], "base_id": "2610.31385", "categories": ["cs.CV", "cs.CL", "cs.IR"], "id": "2610.31385v1", "pdf_url": "https://arxiv.org/pdf/2610.31385v1", "primary_category": "cs.CV", "published": "2026-10-22T14:52:47Z", "query_hits": ["q4_long_document"], "seq": 2, "title": "A Task-Centric Survey of Long-Document Visual Question Answering: Taxonomy, Evolution, and Future Directions", "updated": "2026-10-22T14:52:47Z"}
|
|
{"abstract": "Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.", "authors": ["Ruotong Liao", "Meikel Kokowski", "Xiaoxiao Sun", "Yongqin Xian", "Laura Bravo S\u00e1nchez", "Shiye Su", "Federico Tombari", "Serena Yeung-Levy", "Volker Tresp"], "base_id": "2610.30089", "categories": ["cs.CV"], "id": "2610.30089v1", "pdf_url": "https://arxiv.org/pdf/2610.30089v1", "primary_category": "cs.CV", "published": "2026-10-21T17:37:58Z", "query_hits": ["q5_videolang_long"], "seq": 3, "title": "ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video", "updated": "2026-10-21T17:37:58Z"}
|
|
{"abstract": "Long-video understanding with multimodal LLMs is fundamentally constrained by the mismatch between video length and the model's limited visual context budget, making frame selection essential for practical reasoning. Existing query-aware methods typically rely on heuristic pipelines: they assign one-dimensional query-conditioned scores to frames and then apply sampling, ranking, or extraction rules to form the final subset. While effective in practice, such methods are generally not derived from an explicit optimization objective and struggle to capture the heterogeneous evidence required for complex reasoning. In this paper, we revisit long-video frame selection from a principled optimization perspective. We formulate it as a structured evidence-allocation problem and propose OTFS, an optimal transport framework that maps multiple evidence sources onto video frames. To reflect the asymmetry of this problem---important evidence should be preserved, whereas redundant frames need not absorb mass---we instantiate this view with a semi-unbalanced entropic optimal transport objective and an efficient solver, OTFS-Sinkhorn. Combined with an exact dynamic program for coverage-aware subset extraction, OTFS yields a practical two-stage training-free pipeline. We further show that, in the single-source case, our formulation induces a Gibbs-type distribution over frames, making Q-Frame's temperature-scaled distribution a special case of our framework while also providing a broader perspective for understanding related prior methods. Experiments on three long-video understanding benchmarks show that OTFS consistently outperforms strong training-free baselines. Codes are available at https://github.com/barry664/OTFS.", "authors": ["Ziwei Zhao", "Yuhang Liu"], "base_id": "2610.27692", "categories": ["cs.CV"], "id": "2610.27692v1", "pdf_url": "https://arxiv.org/pdf/2610.27692v1", "primary_category": "cs.CV", "published": "2026-10-20T06:12:30Z", "query_hits": ["q2_long_video"], "seq": 4, "title": "A Principled Optimal Transport Framework for Frame Selection in Long Video Understanding", "updated": "2026-10-20T06:12:30Z"}
|
|
{"abstract": "While VideoLLMs have made significant progress in understanding short video benchmarks, they often struggle to summarize long-form ones. This is mainly due to performance degradation caused by long-context prompts, the inherent difficulty of preserving temporal continuity and a lack of semantically rich concepts in event descriptions. To tackle these issues, this paper introduces the first recursive Video Language Summarization Model (RVLSM) designed for video processing and tailored to long-form video-to-text summarization. RVLSM recursively decomposes videos into semantically coherent segments and delegates chunk-level reasoning to VideoLLM sub-models and external tools. It employs (1) Semantic Temporal Decomposition (STD), which preserves temporal coherence and continuity through a scene-aware chunking strategy that dynamically partitions videos according to semantic boundaries, and (2) Adaptive Multimodal Tool Calling (AMTC), which ensures semantic richness of the summary via a routing mechanism that selectively invokes auxiliary modules such as action recognition, optical character recognition, automatic speech recognition, and object detection. Experiments on the VideoXum and MixTyVSum benchmarks show that RVLSM consistently improves over zero-shot VideoLLM baselines across multiple model settings and performance metrics, and is competitive with a state-of-the-art fine-tuned model for video-to-text summarization. Importantly, the named entities in the RVLSM's summaries align more closely with those detected in the ground-truth summaries than those detected by baseline and state-of-the-art methods, confirming the saliency of the event descriptions it generates.", "authors": ["Alessia Intini", "Lorenzo Vaiani", "Enrico Chen", "Luca Cagliero"], "base_id": "2610.26431", "categories": ["cs.CV"], "id": "2610.26431v1", "pdf_url": "https://arxiv.org/pdf/2610.26431v1", "primary_category": "cs.CV", "published": "2026-10-19T13:56:33Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv", "q5_videolang_long"], "seq": 5, "title": "Long-form Video-to-Text Summarization using Recursive Language Models", "updated": "2026-10-19T13:56:33Z"}
|
|
{"abstract": "Multimodal Large Language Models (MLLMs) face significant scalability challenges in long-video understanding due to quadratic attention complexity and context window constraints. Existing retrieval-based approaches typically rely on supervised learning from static keyframe annotations. However, these label priors often fail to align with the actual information required by downstream MLLMs for complex reasoning, creating a misalignment between the training target and inference needs. To bridge this gap, we propose a probabilistic framework that reformulates frame selection as latent-variable inference. We first initialize a lightweight policy using supervised learning to capture general visual saliency. Then, we treat a frozen MLLM as a task environment and use its answer likelihood as dense task evidence to refine the policy via Bayesian Group Relative Policy Optimization (GRPO). Within this formulation, the negative log-likelihood (NLL) over ground-truth answer tokens provides a principled dense reward, while a prior-matching KL term preserves semantic consistency with the elicited label prior. This process transitions the selection principle from static label priors to task evidence. Extensive experiments on Video-MME, MLVU, and LongVideoBench demonstrate that our selector achieves competitive accuracy with MLLM-based scorers while retaining the inference efficiency of lightweight methods. The selector is trained once with a frozen MLLM environment and transfers directly to multiple downstream MLLMs without retraining. Code is available at \\url{github.com/Miraclecsc/Frame_Selection_GRPO}.", "authors": ["Shuochen Chang", "Bingjie Gao", "Qingyang Liu", "Qianli Ma", "Yibo Miao", "Haonan Zhao", "Zhaohe Liao", "Xiaofeng Zhang", "Jiangtong Li", "Li Niu"], "base_id": "2610.25418", "categories": ["cs.CV"], "id": "2610.25418v1", "pdf_url": "https://arxiv.org/pdf/2610.25418v1", "primary_category": "cs.CV", "published": "2026-10-18T17:17:59Z", "query_hits": ["q2_long_video"], "seq": 6, "title": "From Label Priors to Task Evidence: Efficient Long-Video Frame Selection via Bayesian GRPO", "updated": "2026-10-18T17:17:59Z"}
|
|
{"abstract": "Human memories are organized in a way that connects events with their corresponding spatial contexts. An AI assistant should also be able to memorize and recall the nexus of events and spaces. Thus we introduce SENRI, a task that studies this space--event connection in long, continuous real-life egocentric video. It contains six task types in two directions: Event$\\rightarrow$Space, which recovers spatial knowledge from event cues, and Space$\\rightarrow$Event, which retrieves events from regions and trajectories. To evaluate this, we build SENRI-Bench, containing 500 manually annotated multiple-choice questions over 10 uninterrupted self-recorded videos with around 25.3 hours of content. Also, we evaluate three typical long video understanding paradigms: direct perception, evidence seeking, and persistent memory. The task remains challenging across all three paradigms, highlighting substantial room for improvement. Then we show two representative failure modes of existing memory methods and address these evidence gaps with SENRI-Agent, a controller that uses Target-Guided Visual Check (TGVC) to propose local visual check tasks for parallel verifiers and integrate the reports into a working memory to help with reasoning. Built upon existing memory systems, SENRI-Agent improves the performance of existing baselines without modifying their stored memories. We hope SENRI will provide valuable insights into how real-life egocentric understanding should be evaluated, reveal the gaps in existing approaches, and offer a possible direction for improvement.", "authors": ["Kehan Lan", "Kaining Ying", "Henghui Ding", "Kaipeng Zhang"], "base_id": "2610.25309", "categories": ["cs.CV"], "id": "2610.25309v1", "pdf_url": "https://arxiv.org/pdf/2610.25309v1", "primary_category": "cs.CV", "published": "2026-10-18T14:41:41Z", "query_hits": ["q2_long_video"], "seq": 7, "title": "SENRI: Understanding the Space-Event Nexus in Real-Life Videos", "updated": "2026-10-18T14:41:41Z"}
|
|
{"abstract": "Long-video understanding often requires answering many questions about a single video whose evidence is distributed across extended temporal contexts. Conventional video-language models repeatedly encode frames or retrieve visual evidence for every question. Parameterized video memory offers a write-once alternative, but existing methods rely primarily on synthesized text and fixed adaptation configurations, leaving open where video knowledge should be written and how visual evidence should shape the memory. We introduce Memory Internalization via Test-Time Training (MIT^3), a per-video adaptation framework that writes each long video into compact, video-specific adaptive parameters for video-free, text-only question answering. Under compact multi-scale supervision, causal tracing measures the causal memory utility of each model layer for video-grounded answering and selects video-specific adaptation locations. Visual memory distillation then trains the text-only memory to reproduce the representation change induced by the relevant video clip, preserving the internal effect of visual evidence beyond textual supervision. Across LVBench, LongVideoBench, and Video-MME, MIT^3 achieves data-efficient video-free question answering with substantially fewer synthesized samples than prior parameterized video-memory methods, demonstrating that effective video internalization depends on both where knowledge is written and how visual evidence is distilled into the model.", "authors": ["Haonan Zhao", "Jiaxuan Chen", "Qingyang Liu", "Shuochen Chang", "Siyu Sun", "Jiangtong Li", "Li Niu"], "base_id": "2610.26910", "categories": ["cs.CV"], "id": "2610.26910v1", "pdf_url": "https://arxiv.org/pdf/2610.26910v1", "primary_category": "cs.CV", "published": "2026-10-17T04:06:57Z", "query_hits": ["q2_long_video", "q5_videolang_long"], "seq": 8, "title": "Memory Internalization via Test-Time Training for Long-Video Understanding", "updated": "2026-10-17T04:06:57Z"}
|
|
{"abstract": "Reconstructing high-quality, renderable 3D scenes from long videos is essential for robotic perception and embodied intelligence. Many feed-forward Gaussian reconstruction methods rapidly predict 3D scenes from a small set of images, but extending them to longer sequences presents two bottlenecks: jointly processing all views increases GPU memory requirements, while accumulating pixel-aligned predictions redundantly represents the same surfaces and increases storage and rendering costs. We present LonGaussian, a streaming feed-forward framework that reconstructs compact, globally renderable Gaussian scenes from long image sequences with known intrinsics and unknown camera poses. Built on a pretrained streaming 3D reconstruction backbone, LonGaussian decodes long-context geometric features into dense Gaussians. Within each chunk, multi-frame point tracks link repeated observations of the same surface and guide a learned merger to consolidate the corresponding Gaussian predictions. Rendered-depth consistency filtering further suppresses Gaussians from other chunks that incorrectly occlude locally reconstructed surfaces. On DL3DV and RealEstate10K benchmarks, LonGaussian improves novel-view rendering quality while using substantially fewer Gaussians than the evaluated feed-forward baselines.", "authors": ["Zhoufeng Xie", "Guanjun Wu", "Shuai Zhang", "Xinggang Wang", "Wenyu Liu"], "base_id": "2610.23250", "categories": ["cs.CV"], "id": "2610.23250v1", "pdf_url": "https://arxiv.org/pdf/2610.23250v1", "primary_category": "cs.CV", "published": "2026-10-16T10:24:17Z", "query_hits": ["q3_longctx_cv"], "seq": 9, "title": "LonGaussian: Tracking Foundation Priors for Long-Context Feedforward Gaussian Reconstruction", "updated": "2026-10-16T10:24:17Z"}
|
|
{"abstract": "Multimodal long-document question answering requires models to seek sparse evidence across pages and modalities, understand heterogeneous document content, and reason over the collected evidence. Even advanced vision-language models (VLMs) can overlook sparse evidence and struggle with cross-modal reasoning in a single pass. Existing document agents further introduce multi-step interaction, but some still rely on predefined workflows or coarse-grained actions, limiting the flexibility and granularity of document reasoning. We introduce DocPilot, an agentic document framework that supports expressive orchestration and makes complex document solving learnable by small language models (SLMs). DocPilot orchestrates evidence seeking through page-level navigation and fine-grained evidence localization, and integrates local visual inspection, code-based numerical reasoning, and cross-modal reasoning for multimodal understanding. To transfer these capabilities to small models, we propose curriculum-based agentic training that progresses from single-ability learning to adaptive orchestration through supervision from data-grounded reasoning traces and executable interaction trajectories. Across MMLongBench-Doc, DocBench, FinRAGBench-V, and MADQA, DocPilot achieves strong gains with frontier VLM backbones, while DocPilot-4B delivers competitive performance against substantially larger models, demonstrating strong effectiveness and generalization across diverse long-document reasoning settings. The code is available at https://github.com/HKUSTDial/DocPilot.", "authors": ["Zhuowen Liang", "Xiaotian Lin", "Yuyu Luo", "Haixun Wang", "Nan Tang"], "base_id": "2610.21327", "categories": ["cs.AI", "cs.CL", "cs.CV"], "id": "2610.21327v1", "pdf_url": "https://arxiv.org/pdf/2610.21327v1", "primary_category": "cs.AI", "published": "2026-10-15T06:16:43Z", "query_hits": ["q4_long_document"], "seq": 10, "title": "DocPilot: Interactive Agent Orchestration and Training for Long Visual Document Understanding", "updated": "2026-10-15T06:16:43Z"}
|
|
{"abstract": "Spatial agents have access to reconstruction and segmentation tools, yet inconsistent predictions and fragmented object identities can prevent their outputs from becoming reliable 3D scene information. A simple oracle experiment exposes this perception bottleneck: replacing tool-predicted perception with ground-truth 3D bounding boxes raises an agent to nearly full score across five object-centric VSI-Bench tasks, while retaining the same reasoning tools. Motivated by this finding, we believe the devil of spatial reasoning lies in 3D itself, and introduce DeSpatial(Devil of Spatial), which explicitly decomposes spatial understanding into perception and reasoning. DeSpatial has a shared 3D perception tool to reconcile independent perception outputs into a unified, metric, gravity-aligned 3D scene. This perception tool has two core components: a) a joint segmentation--depth co-refinement module that consolidates masks of the same object and removes projection artifacts caused by inconsistent segmentation and depth within a single frame; b) a cross-view instance fusion module that consolidate voxel-overlapping instances but not allowed to merge same-frame instances. This fusion module additionally fuses spatially disjoint fragments caused by insufficient multi-view observations, by checking their 3D structure and source mask appearance. After receiving the complete and unified 3D scene information, the agent chooses reasoning or computational tools to solve the spatial problem. DeSpatial achieves leading performance on VSI-Bench, MindCube-Tiny, and ViewSpatial, with additional long-video results on VSI-SUPER, revealing the bottleneck of spatial understanding lies in a precise 3D perception itself.", "authors": ["Yifei Liu", "Xiaolong Zhou", "Ziyang Gong", "Yifan Zhou", "Yuanyuan Gao", "Yuning Gong", "Yuchen Yang", "Hongjie Zhang", "Xue Yang", "Xiao Sun", "Zhihang Zhong"], "base_id": "2610.20990", "categories": ["cs.CV"], "id": "2610.20990v1", "pdf_url": "https://arxiv.org/pdf/2610.20990v1", "primary_category": "cs.CV", "published": "2026-10-14T22:05:23Z", "query_hits": ["q2_long_video"], "seq": 11, "title": "DeSpatial: The devil of Agentic Spatial Reasoning Lies in 3D Itself", "updated": "2026-10-14T22:05:23Z"}
|
|
{"abstract": "Existing long-document QA benchmarks rarely combine large-scale human-authored questions with detailed visual evidence annotations, limiting diagnosis of evidence localization and interpretation failures. We present VELD-QA, a large-scale benchmark for Visual Evidence grounding and question answering in Long Documents. It contains 15,000 fully human-authored question-answer pairs across 5,859 real-world PDFs totaling 372,445 pages, balancing chart, table, and complex-layout tasks. Human annotations identify supporting pages and evidence elements and describe cited visual content, enabling separate retrieval and interpretation evaluation. We evaluate nine vision-language models under four evidence settings and seven retrieval-augmented and agentic systems. Our analysis identifies bottlenecks in evidence localization and interpretation. Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages; full-document access does not close the gap. Localization alone is insufficient: oracle accuracy peaks at 81.7%, highlighting persistent visual interpretation and reasoning challenges. Enriching inputs with our visual-element annotations narrows the performance gap by 17-24%, demonstrating value beyond evaluation and motivating data enrichment for multimodal document QA. Our code is available at https://github.com/oracle-samples/veld-qa.", "authors": ["Zhivar Sourati", "Marianne Menglin Liu", "Rongguang Wang", "Zheng Wang", "Yazhe Hu", "Lan Guo", "Yuying Wang", "Mengqing Guo", "Bhavesh Dnyaneshwar Gore", "Shubham Pathak", "Ramapriya Ranganath", "Sandeep Sharma", "Tao Sheng", "Sujith Ravi", "Dan Roth"], "base_id": "2610.20964", "categories": ["cs.CV", "cs.CL"], "id": "2610.20964v2", "pdf_url": "https://arxiv.org/pdf/2610.20964v2", "primary_category": "cs.CV", "published": "2026-10-14T21:36:28Z", "query_hits": ["q4_long_document"], "seq": 12, "title": "VELD-QA: A Large-Scale Dataset for Visual Evidence Grounding and Question Answering in Long Documents", "updated": "2026-10-16T19:11:33Z"}
|
|
{"abstract": "Vision-language models (VLMs) have been widely adopted for document parsing and optical character recognition (OCR), as they can accurately extract page-level elements. However, downstream applications such as retrieval-augmented generation (RAG) also require coherent document-level structure. Although these models indicate heading levels using different numbers of preceding ``\\#'' symbols, page-by-page prediction often lacks document-level consistency. In practice, input-resolution and context-length constraints make it infeasible for a VLM to jointly process all page images in documents containing tens or hundreds of pages. To address this limitation, we propose a VLM-based framework that incrementally reconstructs heading hierarchies with a \\emph{level-wise multimodal memory}. The memory retains a compact subset of all previously processed pages, selected by anchors. Our framework supports flexible anchor selection, and in this work we select the most recent node at each level, allowing the VLM to observe the visual, textual, and layout patterns of headings at every level. For comprehensive evaluation, we also introduce the Complex and Long Document Heading Hierarchy Reconstruction Benchmark (CLDoc-HHRBench), comprising 250 documents and 11,212 pages with expert-verified annotations. Experiments on CLDoc-HHRBench and three existing benchmarks demonstrate the effectiveness of our method.", "authors": ["Qian Chen", "Xianyin Zhang", "Lifan Guo", "Chi Zhang"], "base_id": "2610.18238", "categories": ["cs.CV", "cs.AI"], "id": "2610.18238v1", "pdf_url": "https://arxiv.org/pdf/2610.18238v1", "primary_category": "cs.CV", "published": "2026-10-13T12:06:28Z", "query_hits": ["q4_long_document"], "seq": 13, "title": "Heading Hierarchy Reconstruction for Complex and Long Documents with Level-Wise Multimodal Memory", "updated": "2026-10-13T12:06:28Z"}
|
|
{"abstract": "Long video understanding requires VideoLLMs to answer queries about videos spanning tens of minutes to hours, but such long videos produce far more visual tokens than current VideoLLMs can process. To address this issue, keyframe selectors are proposed to select the most query-relevant frames by computing the similarity between frame and query embeddings. Existing selectors either encode frames independently without temporal context or process chunks in isolation with ranking objectives that are difficult to optimize. To alleviate those limitations, we propose FrameScout with two core designs: (i) a streaming selector architecture that leverages a sliding-window KV cache and successor aggregation to produce temporally-contextualized frame embeddings; (ii) a frame-query contrastive objective that aligns frame and query embeddings in a shared space, directly training the selector to distinguish relevant frames from irrelevant ones. Extensive experiments on three long-video benchmarks and six base VideoLLMs demonstrate that FrameScout consistently achieves state-of-the-art performance. Code is available at https://github.com/Hubert-1102/FrameScout.", "authors": ["Haonan Hu", "Shuhao Chen", "Weisen Jiang", "Lizhao Gao", "James Kwok", "Yu Zhang"], "base_id": "2610.23975", "categories": ["cs.CV", "cs.LG"], "id": "2610.23975v1", "pdf_url": "https://arxiv.org/pdf/2610.23975v1", "primary_category": "cs.CV", "published": "2026-10-13T11:22:36Z", "query_hits": ["q2_long_video"], "seq": 14, "title": "FrameScout: Scouting Query-Relevant Frames for Long Video Understanding", "updated": "2026-10-13T11:22:36Z"}
|
|
{"abstract": "Long-context training of large language and multimodal models couples structured attention masks (document, prefix-LM, share-question, sliding-window, block-sparse) with multi-node context parallelism, making the mask a dominant bottleneck. FlashMask's column-wise sparse mask compresses them from $O(n^2)$ to $O(n)$ via four query-axis boundary vectors, covering broad structured patterns. Implemented for Ampere GPUs, it neither exploits FlashAttention-3's TMA/WGMMA warp-specialized pipeline on Hopper nor supports distribution. The column-wise sparse mask is indexed by key positions but stores query-axis interval boundaries. Under any multi-chunk context-parallel assignment, it reduces per-rank mask localization to a provably zero-communication \\emph{clip--shift--max} on the boundary vectors, compatible with the single-GPU kernel. FlashMask-3 introduces a mask-aware Hopper kernel with three-role warp specialization atop FlashAttention-3's TMA/WGMMA pipeline, plus a mask-aware distributed extension: zero-communication mask localization, a sparsity-aware load balancer (HCC-IPO), and a mask-aware compute-communication overlap with mask-driven communication pruning, whose forward all-gather uses topology-aware hierarchical routing. On a single H100 across 12 masks at 4K--128K context, it improves achieved TFLOPs over FlashMask, FlexAttention, and MagiAttention by 45--141%, 34--71%, and 2--44%. At context-parallel degrees 4--32 with 8K sequence per device, it attains 31--75% compute-communication overlap efficiency and improves throughput over Megatron-Core CP ring, Megatron-Core CP all-gather, and MagiAttention by $1.80\\times$--$3.71\\times$, $1.71\\times$--$2.92\\times$, and $1.09\\times$--$1.46\\times$. In end-to-end training of a 118B-A14B MoE at 128K on 128 H100 GPUs, it raises throughput by 7.3--11.0% over a Dual Chunk all-gather baseline, with attention forward+backward 24.5--44.9% faster.", "authors": ["Guoxia Wang", "Qianyue He", "Siming Wu", "Haoyang Xie", "Qiwen Bao", "Jinle Zeng", "Jiabin Yang", "Tian Wu", "Dianhai Yu"], "base_id": "2610.17901", "categories": ["cs.LG"], "id": "2610.17901v1", "pdf_url": "https://arxiv.org/pdf/2610.17901v1", "primary_category": "cs.LG", "published": "2026-10-13T07:55:51Z", "query_hits": ["q1_longctx_multimodal"], "seq": 15, "title": "FlashMask-3: Efficient and Expressive Mask-Aware Distributed Attention", "updated": "2026-10-13T07:55:51Z"}
|
|
{"abstract": "Reliable video reasoning requires models to produce correct answers and expose supporting visual evidence. Open-o3-Video enables verifiable reasoning by embedding timestamps and object-level bounding boxes into reasoning traces, but is restricted to frames from an initial sparse sampler. We introduce Open-o3-Video-v2, which formulates video reasoning as a unified agentic evidencegrounded process. The model identifies missing information, acquires additional observations, grounds the moments and objects supporting its reasoning, and writes this evidence into the trajectory before continuing or answering. To support this behavior, we construct STGR-v2-CoT135K, including 52K multi-turn agentic grounded reasoning trajectories, and STGR-v2-RL-13K for reinforcement learning. We further propose Clue-Seek GRPO (CS-GRPO), which tracks how tool calls change accumulated evidence, rewards clue coverage and step-wise gains, and penalizes excessive, repetitive, or uninformative retrieval. On V-STaR, Open-o3-Video-v2-4B achieves 40.4% mAM and 56.3% mLGM, outperforming its Qwen3-VL-4B backbone by 8.5% and 12.6%, respectively. It also delivers consistent gains across general video understanding, long-video reasoning, and temporal grounding benchmarks. These results show that active evidence acquisition and verifiable object-level grounding are complementary for reliable video reasoning. The code and models are released at https://msalab-pku.github.io/projects/Open-o3-Video-v2.", "authors": ["Jiahao Meng", "Zhongyuan Bao", "Yue Tan", "Haiquan Wen", "Renrui Zhang", "Anna Wang", "Jason Li", "Yanwei Fu", "Yunhai Tong", "Ming-Hsuan Yang"], "base_id": "2610.17656", "categories": ["cs.CV"], "id": "2610.17656v1", "pdf_url": "https://arxiv.org/pdf/2610.17656v1", "primary_category": "cs.CV", "published": "2026-10-13T03:36:33Z", "query_hits": ["q2_long_video"], "seq": 16, "title": "Open-o3-Video-v2: Seek, Ground, and Reason with Verifiable Spatio-Temporal Evidence", "updated": "2026-10-13T03:36:33Z"}
|
|
{"abstract": "Video language models represent a video as a long sequence of visual embeddings in a decoder's input. Every visual embedding is processed by every decoder layer during prefill, stored in the key value cache, and attended by every generated token, so a long video raises prefill, memory, and generation cost at once. We present WIRE (Weight Space Implicit Representation Encoding), which reduces this recurring cost by moving part of the video conditioned computation of a frozen video language model from its input activations into a temporary low rank update of its weights. WIRE keeps a short explicit visual context, so the update acts as a correction rather than a replacement of the video. WIRE first compresses the full resolution embeddings of each frame into a small set of carrier tokens using a question conditioned bidirectional selective scan, and physically contracts the decoder input so every decoder layer, cache entry, and decoding step operates on the shorter sequence. Hypernetworks at selected decoder layers read the compressed forward pass together with a readout of the full resolution features removed by contraction and generate LoRA updates to the attention and feedforward projections. The updates are merged after prefill, so autoregressive decoding runs the unmodified frozen backbone. On Qwen3-VL-8B, WIRE with 16 carrier tokens per frame contracts the visual stream by $3.5\\times$, reduces decoder prefill time by $3.35\\times$ at 256 frames and $4.14\\times$ at 1024 frames, shrinks the key value cache proportionally, and keeps per token decoding latency approximately constant. Across nine benchmark settings at 256 frames, it obtains the highest average accuracy among efficient video language models we evaluate under a common 256 frame budget.", "authors": ["Ameen Ali", "Tamim Zoabi", "Lior Wolf", "Chaim Baskin"], "base_id": "2610.17355", "categories": ["cs.CV"], "id": "2610.17355v1", "pdf_url": "https://arxiv.org/pdf/2610.17355v1", "primary_category": "cs.CV", "published": "2026-10-12T20:13:21Z", "query_hits": ["q5_videolang_long"], "seq": 17, "title": "WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs", "updated": "2026-10-12T20:13:21Z"}
|
|
{"abstract": "Can a vision-language model tell left from right? Modern VLMs solve visual puzzles and follow hour-long videos, yet when asked whether the camera moved left or right, most do no better than a coin flip. Adding camera-motion data or geometric representations has been tried, separately, and neither closes the gap. We argue that camera-motion understanding needs two conditions at once: the representation must couple off-screen camera motion with on-screen flow dynamics, and it must be interpreted by the language model rather than trained as a standalone pose or depth estimator. We propose CameraVLM, which pairs a motion-state representation with state-language alignment. Camera tokens capture how the camera moves off screen, flow tokens capture how the subject and background move on screen, and a two-stage training strategy keeps language in the loop so that both are learned together with camera-motion semantics. With only 9B parameters, CameraVLM outperforms all open-source and camera-motion-specific models by a wide margin across three benchmarks and more than 20 baselines, and matches or surpasses GPT-6 Astra as well as human viewers in a 48-participant study. Project page: https://3dagentworld.github.io/CameraVLM/", "authors": ["Chong Cheng", "Ran Zhang", "Xinyuan Li", "Alan Zhao", "Hao Wang", "Xi Wang"], "base_id": "2610.17242", "categories": ["cs.CV"], "id": "2610.17242v1", "pdf_url": "https://arxiv.org/pdf/2610.17242v1", "primary_category": "cs.CV", "published": "2026-10-12T18:09:24Z", "query_hits": ["q2_long_video"], "seq": 18, "title": "When Left Is Right? Camera-Motion Understanding beyond Screen", "updated": "2026-10-12T18:09:24Z"}
|
|
{"abstract": "Video anomaly understanding (VAU) aims to generate language-level interpretations of anomalous events in long videos, requiring visual evidence that supports explanation beyond anomaly detection alone. Existing VAU pipelines commonly rely on anomaly-centric or predefined temporal sampling strategies, which efficiently emphasize suspicious moments but overlook contextual cues, object interactions, and temporally distributed evidence needed for reasoning. Motivated by recent progress in learnable temporal selection for long-video question answering, we revisit VAU temporal sampling as a downstream-aligned evidence selection problem. We introduce the semantic-conditioned Temporal Sampling Q-Former that learns to select temporal visual evidence through the VAU generation objective. Since generic VAU prompts provide weak video-specific guidance, we condition the sampler on CLIP-embedded object tags automatically extracted from the input video. These tags are used only for temporal evidence selection and are not appended to the LLM generation prompt, separating sampler conditioning from prompt augmentation. The proposed sampler operates in visual-token space, computes soft temporal selection weights from frame-level representations, and merges duplicated selections before response generation. Experiments on HIVAU-70K show that our model achieves the best performance among the compared fine-tuned models, while ablations demonstrate that learnable temporal sampling improves over fixed anomaly-focused sampling and that CLIP-embedded object-tag conditioning further improves over generic-prompt conditioning. Our results suggest that temporal sampling in VAU should be aligned with explanation-oriented evidence utility rather than anomaly saliency alone.", "authors": ["Hangil Park", "Seungheon Kim", "Tae-Kyun Kim"], "base_id": "2610.16299", "categories": ["cs.CV"], "id": "2610.16299v1", "pdf_url": "https://arxiv.org/pdf/2610.16299v1", "primary_category": "cs.CV", "published": "2026-10-12T07:26:31Z", "query_hits": ["q2_long_video"], "seq": 19, "title": "Learning Temporal Evidence Selection for Video Anomaly Understanding", "updated": "2026-10-12T07:26:31Z"}
|
|
{"abstract": "Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point -- the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact, failing to resolve the fundamental bottleneck. We propose FramePack-U, a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed size visual representation at constant memory cost. FramePack-U requires no fine-tuning or architectural modification -- it plugs directly into off-the-shelf VLMs at inference time. Inspired by FramePack from the generation literature, we propose a content-aware compression strategy for visual understanding under a fixed token budget. Specifically, FramePack-U (1) scores each frame by its semantic similarity to the text prompt and ranks frames in a diversity-driven manner, and (2) packs the selected frames at varying resolutions into a fixed-length context window. This concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While FramePack-U can in theory pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that FramePack-U consistently outperforms full-resolution baselines and prior token reduction and keyframe selection methods across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, FramePack-U yields gains of +12.9\\%, +10.6\\%, and +9.7\\% over the official model on the three benchmarks, respectively.", "authors": ["Shuo Gao", "Chenhao Zheng", "Zhongzheng Ren"], "base_id": "2610.16118", "categories": ["cs.CV", "cs.LG", "cs.MM"], "id": "2610.16118v1", "pdf_url": "https://arxiv.org/pdf/2610.16118v1", "primary_category": "cs.CV", "published": "2026-10-12T03:33:06Z", "query_hits": ["q2_long_video"], "seq": 20, "title": "Elephant in the Fridge: Constant-Memory Frame Packing for Long Video Understanding", "updated": "2026-10-12T03:33:06Z"}
|
|
{"abstract": "Detailed video captioning has become an essential semantic interface for multi-modal understanding and generation. While existing methods mainly target short videos, extending them to long videos remains challenging, primarily due to limited context windows of current video understanding models. A naive solution is to adopt segment-wise captioning followed by concatenation, yet this paradigm often suffers from inconsistent subject references and fragmented long-range narratives. To mitigate these issues, we introduce PIGEONS, an incremental long-video captioning model built upon a Persistent Identity Gallery and an Evolving Overarching Narrative Summary. Before caption generation, PIGEONS scans the entire video to construct an identity gallery that captures key subjects along with their visual references, thereby providing a stable referential anchor. The model then performs clip-by-clip captioning conditioned on the persistent gallery and an evolving overarching narrative summary, which compactly maintains and updates essential long-range contextual information after each captioning step. To enable these capabilities, we develop a three-stage training strategy that integrates supervised fine-tuning and reinforcement learning. Extensive experiments demonstrate that PIGEONS markedly improves long-video captioning performance, achieving superior accuracy, completeness, and subject referential consistency over existing methods, while maintaining competitive performance on short-video captioning.", "authors": ["Xinlong Chen", "Weihong Lin", "Yuanxing Zhang", "Xinyu Liu", "Yue Ding", "Bozhou Li", "Bohan Zeng", "Yang Shi", "Qiang Liu", "Pengfei Wan", "Liang Wang", "Tieniu Tan"], "base_id": "2610.16066", "categories": ["cs.CV"], "id": "2610.16066v1", "pdf_url": "https://arxiv.org/pdf/2610.16066v1", "primary_category": "cs.CV", "published": "2026-10-12T02:11:38Z", "query_hits": ["q2_long_video"], "seq": 21, "title": "PIGEONS: Incremental Long-Video Captioning with Persistent Identity Gallery and Evolving Overarching Narrative Summary", "updated": "2026-10-12T02:11:38Z"}
|
|
{"abstract": "Language models are becoming infrastructure, and infrastructure built in the open, by public institutions, on documented data, is the alternative to dependence on a few providers; Apertus shows this is viable at scale. Apertus 1.5 extends the fully open Apertus family from text to multimodal reasoning. Continued pretraining of the Apertus 1.0 models, on 4T additional tokens for the 8B and 2T for the 70B, adds image and audio understanding and extends the context window fourfold to 262,144 tokens; multimodality enters through the tokenizer, the language-model head remains restricted to text, and the architecture is unchanged. Supervised finetuning, long-context finetuning, reinforcement learning with verifiable rewards, and preference alignment complete the released models. Thinking is a flag of the chat template rather than a separate model, and lifts mathematical reasoning by up to 20 points on the 8B model. The released models improve over their predecessors on nearly every benchmark; among fully open models of its class, the 8B leads on knowledge, instruction following, and cultural benchmarks, and its instruction following surpasses all compared models, including open-weight ones. On image and audio understanding, a first for the family, the models are competitive with fully open multimodal models, strongest on document understanding and long-form and lower-resource speech recognition, while specialized vision and audio models remain ahead on perception and transcription of English and other high-resource languages. As in Apertus 1.0, we train only on openly available data, respect crawler opt-outs retroactively, and exclude restrictively licensed material. We release the model weights, data preparation and training recipes, and evaluation suites under permissive licenses; openness extends to values, with the Apertus charter guiding alignment stages published alongside the models.", "authors": ["Project Apertus", " :", "Yixuan Xu", "Yassine Turki", "Vinko Sabol\u010dec", "Valentina Pyatkin", "Tommy Chu", "Sven Najem-Meyer", "Sean F. Carroll", "Robert Smith", "Raphael Kreft", "Oscar Alessandro Tazza", "Oleg Lavrovsky", "Nicola Irmiger", "Matteo Santelmo", "Marian Schneider", "Juan Garc\u00eda Giraldo", "Hanna Yukhymenko", "Eduard \u010eurech", "Davit Melikidze", "Dani Tamayo", "Bettina Messmer", "Aryan Ahadinia", "Anunay Yadav", "Allen Huang", "Ahmad Fraij", "Alessandro Bombarda", "Alex Padula", "Alexander H\u00e4gele", "Alexander Sternfeld", "Anna Hedstr\u00f6m", "Anna Kosovskaia", "Arthur Wuhrmann", "Arundhati Balasubramaniam", "Badr Al Mahouri", "Barna Pasztor", "Bartosz Szostakiewicz", "Bich Ngoc Doan", "Bryan Schmid", "Burak Sezen", "Camille Challier", "Cl\u00e9ment Charmillot", "Dana Shayakhmetova", "Daniel Perruchoud", "David Br\u00fcggemann", "David Isztl", "Elia Palme", "Emma Anastassova", "Fabian B\u00f6sch", "Faruk Zahiragi\u0107", "Fawzi Roberto Mohamed", "Fiona Panicali", "Florian Kolly", "Flynn Guerra", "Francesco Monti", "Frederike L\u00fcbeck", "Gennaro Oliva", "Haofeng Yang", "Haoliang Shang", "Helen El Khoury", "Henrique Mendon\u00e7a", "Jannis Vamvas", "J\u00e9rome Landolt", "Jessica Lam", "Jingchun Xiao", "Jonathan Coles", "Joshua Oyebanji", "Khanh Nguyen", "Konstantinos Pitas", "Lasse B. Strand", "Li Mi", "Lo\u00efc Deslarzes", "Lorenzo Baggi", "Lorenzo Noci", "Luca Baumann", "Luca Sartori", "Lukas Drescher", "Marco De Negri", "Marco Passerini", "Mark Klein", "Marko Stojanovic", "Martin Majsec", "Matin Ansaripour", "Mattea Busato", "Mauro Pellonara", "Max Neuwinger", "Michael Graber", "Michal Te\u0161nar", "Michelle Wastl", "Miguel Gila", "Mikolaj Boronski", "Morgane Magnin", "Moritz Reihs", "Mukhammadali Sayfiddinov", "Nathan Ranchin", "Nicolas Cuveillier", "Oscar Wohlfahrt", "Qianqian Zou", "Rada Kamysheva", "Rita Sevastjanova", "Roman Machacek", "Sebastian Keller", "Simon Matrenok", "Simon Pierre", "Stefano Schuppli", "Suei-Wen Chen", "Szabina Horv\u00e1th-Mikulas", "Taliesin Perez", "Thilo Spinner", "Thomas Kleine Buening", "Tim Robinson", "Tymoteusz Kwiecinski", "Vil\u00e9m Zouhar", "Vinzenz Thoma", "Xiaozhe Yao", "Yannick Metz", "Yuhei Fukuhara", "Yusif Askari", "Zixuan Chen", "Alexander Ilic", "Ana Klimovic", "Andreas Krause", "Andrei Kucharavy", "Elliott Ash", "Joost VandeVondele", "Martin Rajman", "Mennatallah El-Assady", "Thomas Schulthess", "Timo Kehrer", "Torsten Hoefler", "Antoine Bosselut", "Martin Jaggi", "Imanol Schlag"], "base_id": "2610.15542", "categories": ["cs.CL"], "id": "2610.15542v1", "pdf_url": "https://arxiv.org/pdf/2610.15542v1", "primary_category": "cs.CL", "published": "2026-10-11T11:34:43Z", "query_hits": ["q1_longctx_multimodal"], "seq": 22, "title": "Apertus 1.5: From Open Text Models to Open Multimodal Reasoners", "updated": "2026-10-11T11:34:43Z"}
|
|
{"abstract": "Media-bridged time series forecasting is expanding to encompass traditional \"multivariate\" and emerging \"multimodal\" (e.g., through textual assistance). Existing Time Series Forecasting (TSF) models still rely on paradigm-specific relation, fusion, and temporal modules, hindering a common forecasting backbone across numerical and pre-aligned narrative-flow settings. To explore this, we propose the Multimedia Identity-Aware Prism Network (MIDAPN), a unified spatiotemporal forecasting backbone based on media-general graph adaptation and automatic temporal learning: (1) Following media pre-alignment, our Multimedia Identity-Aware Graph (MIDAG) revisits identity through static essence, dynamic behavior, and latent commonality, inducing affinities that extend variable-specific dependencies across media. Contextual Identity Modulation (CIM) further refines discriminative aggregation. (2) We develop Spectral Prism Convolution (SPConv) to automatically perform hierarchical temporal analysis, balancing coarse trends and fine-grained details. Meanwhile, its Adaptive Search Guidance configures a scale-efficient architecture for temporal-dimension reconstruction. These decoupled yet synergistic components jointly address media identity disentanglement and temporal-scale mismatch. Comprehensive evaluations involving 16 SOTA TSF models across 13 \"multivariate\" and 12 \"multimodal\" datasets, alongside targeted long-context comparisons against 14 time series foundation models and fused pretrained language models, demonstrate MIDAPN's consistent superiority and broad shared backbone compatibility. The code is available at \\href{https://github.com/leijieruilq/MIDAPN/tree/main}{https://github.com/MIDAPN}.", "authors": ["Jierui Lei", "Wenjian Zhang", "Qingyi Yang", "Yuyang Hong", "Fangzheng Chen", "Zhengbo Zhang", "Haina Tang", "Shiming Xiang"], "base_id": "2610.11924", "categories": ["cs.CV", "cs.LG"], "id": "2610.11924v1", "pdf_url": "https://arxiv.org/pdf/2610.11924v1", "primary_category": "cs.CV", "published": "2026-10-08T13:20:49Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 23, "title": "Revisiting Identity and Spectra Dispersion in Media-Bridged Time Series Forecasting: Linking Multivariate Signals and Narrative Flows", "updated": "2026-10-08T13:20:49Z"}
|
|
{"abstract": "To adapt billion-parameter Vision Transformers for long-video understanding, recent methods freeze the backbone and train lightweight convolutional modules. While effective for parameter-efficient training, existing adapters do not reduce inference-time computation, leaving scalability with respect to video length largely unaddressed. Token selection can reduce attention cost by pruning redundant tokens, but it breaks the spatial grid structure required by convolutional adapters. This forces an expensive dense reconstruction, nullifying much of the potential speedup. We address this by introducing native sparse 2D convolutions, a primitive that allows these adapters, for the first time, to operate directly and efficiently on dynamically pruned token sets. We integrate this primitive into SV-TAD, an adapter framework for temporal action detection, reducing VideoMAEv2-L computation by up to 64% and achieving 2.2x faster inference, while maintaining state-of-the-art accuracy on THUMOS-14 and ActivityNet-1.3. When scaled to InternVideoNext-L, our approach surpasses the previous state of the art at roughly half its computational cost. Moreover, the sparse formulation naturally supports auxiliary task tokens, which improves fine-grained assembly detection on ATTACH.", "authors": ["Ricardo Pizarro", "Roberto Valle", "Jos\u00e9 M. Buenaposada", "Luis M. Bergasa", "Luis Baumela"], "base_id": "2610.11579", "categories": ["cs.CV"], "id": "2610.11579v1", "pdf_url": "https://arxiv.org/pdf/2610.11579v1", "primary_category": "cs.CV", "published": "2026-10-08T09:31:38Z", "query_hits": ["q2_long_video"], "seq": 24, "title": "SV-TAD: Native Sparse Convs for Efficient Temporal Action Detection", "updated": "2026-10-08T09:31:38Z"}
|
|
{"abstract": "Remote sensing visual question answering (RS-VQA) requires answers that truly depend on image evidence, yet current remote sensing multimodal large language models (RS MLLMs) are still evaluated mainly by accuracy, which cannot reveal whether a correct answer was produced without using the image. This setting introduces three challenges: demonstrations retrieved across different ground sample distances (GSD), sensors, and city textures may be poorly matched; real queries often involve long visual contexts from multi-scale tiles, multi-temporal frames, and retrieved knowledge, where evidence can be diluted and errors can propagate through multi-step reasoning; and prior-only answering remains invisible to standard metrics. We present AVA, a training-free framework that wraps a frozen RS MLLM with three modules. GDR performs GSD-bucketed, scene-graph-based demonstration retrieval and filters demonstrations whose answers are unsupported by their images. MACE decomposes each query into an evidence-typed sub-question DAG, routes sub-questions to suitable perception tools, and records verified provenance in a shared evidence ledger with bounded re-perception. CVA audits the final answer through counterfactual perturbations of cited evidence, including ablation and swap, and computes a visual-dependency score for confidence calibration and evidence-forcing re-query. On RSVQA-LR/HR, VRSBench, and LHRS-Bench, AVA improves average accuracy by up to 2.40 points over the strongest same-setting baselines, raises the visual-dependency score from 0.28 to 0.49, reduces the prior-only answer rate from 46.2 percent to 21.7 percent, and lowers the hallucination rate from 17.1 percent to 9.8 percent, with 2.6x the cost of a single backbone forward pass and no fine-tuning.", "authors": ["Jingwen Huang", "Yiming Fang", "Hanxiao Wu", "Boyuan He", "Keyi Luo"], "base_id": "2610.12517", "categories": ["cs.CV"], "id": "2610.12517v1", "pdf_url": "https://arxiv.org/pdf/2610.12517v1", "primary_category": "cs.CV", "published": "2026-10-07T19:50:09Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 25, "title": "TerraAgent: Demonstration-Grounded Multi-Agent Reasoning for Long-Context Remote Sensing Visual Question Answering", "updated": "2026-10-07T19:50:09Z"}
|
|
{"abstract": "Long video understanding increasingly relies on external memory to organize massive visual streams into compact representations. However, most memory-based methods dynamically adapt how information is retrieved for different questions, while largely fixing what is remembered. This mismatch makes missing details costly to recover, whereas stored information is valuable only when it can be reliably retrieved. To address this issue, we propose VideoEvolve, a novel self-evolving framework that jointly evolves memory and retrieval for long video understanding. Specifically, starting from a coarse low-frame-rate overview, VideoEvolve couples a Memory Evolver for selective memory augmentation with a Retrieval Evolver for adaptive retrieval over the evolving memory. We then co-evolve the two Evolvers through alternating agentic reinforcement learning (Agentic RL), updating one while freezing the other. To steer this alternating evolution, Bottleneck-Aware Evolution Feedback (BEF) identifies whether the current bottleneck lies in memory or retrieval and directs optimization toward the more limiting side. Furthermore, VideoEvolve introduces Capability-Aware Evolution Feedback (CEF) to alleviate downstream feedback from over-specializing memory to a fixed set of training questions, shifting training toward underdeveloped yet learnable video capabilities. By integrating Agentic RL with BEF and CEF, VideoEvolve transforms downstream reasoning experience into transferable capability updates, providing a concrete path from static long-video systems toward experience-driven, self-improving multimodal intelligence. Extensive experiments on multiple long video understanding benchmarks demonstrate the effectiveness of VideoEvolve.", "authors": ["Yongchao Xu", "Bowen Ye", "Jiefeng Gan", "Junkai Ma", "Wenzhao Li", "Sen Tao", "Yi Wei", "Jiawei Liu"], "base_id": "2610.10183", "categories": ["cs.CV", "cs.AI"], "id": "2610.10183v1", "pdf_url": "https://arxiv.org/pdf/2610.10183v1", "primary_category": "cs.CV", "published": "2026-10-07T14:51:23Z", "query_hits": ["q2_long_video"], "seq": 26, "title": "VideoEvolve: Co-Evolving Memory and Retrieval for Long Video Understanding", "updated": "2026-10-07T14:51:23Z"}
|
|
{"abstract": "Recent advances in Vision-Language Models (VLMs) have led to rapid progress in video understanding across a wide range of benchmark tasks. However, existing evaluations largely focus on short-term reasoning, failing to assess a critical capability: maintaining cumulative temporal consistency over extended time horizons. To close this evaluation gap, we introduce HeiCo-FOCUS, a clinically grounded dataset for evaluating long-context video understanding through the task of Foreign Object Contextual Understanding in Surgery. Built on a dataset of Heidelberg Colorectal surgeries, this task requires models to continuously track multiple objects as they are inserted, manipulated, occluded, and removed over procedures lasting up to hours. HeiCo-FOCUS comprises 30,000 visual question answering (VQA) pairs covering five core capabilities: object recognition, temporal grounding, aggregation, event and procedural understanding, and complex reasoning. The dataset was constructed through a rigorous multi-stage annotation pipeline involving large-scale crowd annotation and 39 surgical domain experts to ensure high quality and clinical relevance. To systematically probe model behavior, we introduce a multi-track evaluation framework that progressively increases temporal and contextual demands from single frames to full procedures. Experiments with ten frontier VLMs show that HeiCo-FOCUS tasks are far from solved: only around half of the models clearly outperform a text-only baseline. Across the video tracks, models perform best on event and procedural understanding (mean Accuracy: 56.5% across all models), while temporal grounding remains particularly challenging for all evaluated models (mean Accuracy: 19.7%). We therefore expect HeiCo-FOCUS to serve as a catalyst for the development of models capable of reliable, temporally consistent reasoning over hours-long videos.", "authors": ["Leon Mayer", "Lucas Luttner", "Patrick Godau", "Kai Fritzsche", "Annika Reinke", "Leonie Boland", "Jule Brandt", "Janne Heinecke", "Chloe K. Nobuhara", "Niklas Holzwarth", "Evangelia Christodoulou", "Marcel Knopp", "Dominik Michael", "Pascale Piermarco", "Saliq Neyaz", "Korhan Derin \u00d6zarslan", "Jakob Hennighausen", "Carlos Aumente-Maestro", "Tim R\u00e4dsch", "Dheeraj Baji", "Peter Maximilian Full", "Finn Aichholz", "Justus Veit Erpenbeck", "Linus Finn Schott", "Bastian Winkelhausen", "Claas de Boer", "Bianca G\u00fcttner", "Anneli Hummel", "Gregor Just", "Max Kirchner", "Chenyang Li", "Rozenn Raffaut", "Ariel Rodriguez", "Danush Kumar Venkatesh", "Kevin Wang", "Jinjing Xu", "Mona Sheikh Zeinoddin", "Salman Khan", "Thomas M. Pausch", "Stefanie Speidel", "Danail Stoyanov", "Daniel A. Hashimoto", "Fiona R. Kolbinger", "Thomas G. Weiser", "Lena Maier-Hein"], "base_id": "2610.10156", "categories": ["cs.CV"], "id": "2610.10156v1", "pdf_url": "https://arxiv.org/pdf/2610.10156v1", "primary_category": "cs.CV", "published": "2026-10-07T14:33:47Z", "query_hits": ["q2_long_video", "q3_longctx_cv"], "seq": 27, "title": "HeiCo-FOCUS: A Clinically Grounded Dataset for Long-Context Video Understanding", "updated": "2026-10-07T14:33:47Z"}
|
|
{"abstract": "Modern language model systems operate autonomously over increasingly long contexts containing untrusted text and images. Can an adversarial input continue to steer a model even after that input is removed from its context? We show that attacks can be trained to persist through the key/value (KV) cache of subsequent tokens, allowing adversarial influence to outlive direct access to its source.We consider the Visual Memory Injection (VMI; Schlarmann and Hein, 2026) attack setting, in which an adversarial image that stays in the context plants a hidden backdoor: the model behaves normally until a chosen trigger elicits an attacker-chosen response. We first demonstrate persistence in this setting with optimized soft prompts, which remain effective after we mask the prompt from attention. We then introduce Persistent Visual Memory Injection (P-VMI), which optimizes images to preserve this adversarial behaviour after they are masked from attention. These attacks persist over conversations substantially longer than those used during optimization. On Qwen3-VL-8B-Instruct, P-VMI achieves up to approximately $90\\%$ target success in its strongest configuration and remains effective under a stricter removal setting that exposes the image only on the first turn. A cache-swap ablation localizes the persistent influence to the KV cache. Finally, we show that these attacks can be trained to survive compaction that retains the KV cache of a summary generated by the same model, demonstrating that adversarial behaviour can persist in cached state without continued access to its source.", "authors": ["David Dobre", "Leo Schwinn", "Gauthier Gidel", "Spandana Gella", "Perouz Taslakian", "Pierre-Andr\u00e9 No\u00ebl"], "base_id": "2610.09027", "categories": ["cs.CR", "cs.AI", "cs.CV"], "id": "2610.09027v1", "pdf_url": "https://arxiv.org/pdf/2610.09027v1", "primary_category": "cs.CR", "published": "2026-10-06T19:25:08Z", "query_hits": ["q3_longctx_cv"], "seq": 28, "title": "Visual Memory Attacks Can Persist Through The KV Cache", "updated": "2026-10-06T19:25:08Z"}
|
|
{"abstract": "Video large language models (Vid-LLMs) excel at diverse video-language tasks by reasoning over selected frames. However, frame selection for long videos remains challenging, as it requires retrieving relevant frames distributed across segments from a large candidate pool given complex queries. This paper investigates dominant approaches to long-video frame selection from a task-decomposition perspective, identifying two key challenges: the Query Comprehension Gap in similarity-based methods and the Interpretation--Selection Gap in judgment-based methods. To address them, we propose RACER, a training-free reflective agentic framework that decomposes long-video frame selection into query interpretation driven by a lightweight Vid-LLM and evidence localization supported by an embedding model serving as a retrieval tool. Specifically, the Vid-LLM is responsible solely for reformulating the complex query into sub-queries that make implicit information requirements explicit, mitigating the Query Comprehension Gap. Meanwhile, the retrieval tool leverages these sub-queries to localize relevant evidence, relieving the Vid-LLM of direct frame selection and thus addressing the Interpretation--Selection Gap. Finally, the retrieved frames are fed back to the Vid-LLM for sub-query refinement, forming a reflection loop that iteratively improves query interpretation and frame selection. Experiments across multiple benchmarks show that RACER consistently improves long video understanding. Notably, RACER achieves effective frame selection even with limited-capability components, demonstrating that agentic integration enables these components to enhance more capable Vid-LLMs.", "authors": ["Yiyang Huang", "Yitian Zhang", "Yizhou Wang", "Jianglin Lu", "Qihua Dong", "Hailing Wang", "Huimin Zeng", "Mingyuan Zhang", "Yun Fu"], "base_id": "2610.08954", "categories": ["cs.CV", "cs.AI"], "id": "2610.08954v1", "pdf_url": "https://arxiv.org/pdf/2610.08954v1", "primary_category": "cs.CV", "published": "2026-10-06T18:21:13Z", "query_hits": ["q2_long_video", "q5_videolang_long"], "seq": 29, "title": "RACER: Reflective Agent Coupling Query Interpretation and Tool-Based Retrieval for Frame Selection in Long Video Understanding", "updated": "2026-10-06T18:21:13Z"}
|
|
{"abstract": "Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most of them operate at the frame or snippet level, which limits their ability to model how events evolve over time and relate to each other. In this paper, we propose Event Chain Retrieval-Augmented Generation (EC-RAG), a training-free framework that organizes video content into an explicit event chain before question answering. Instead of retrieving isolated frames or text segments, EC-RAG first partitions the video into semantically coherent segments, represents each segment using multi-modal signals, and then links them into a structured chain that preserves temporal order and captures inter-event relationships. Given a query, the system identifies relevant events within this chain and gathers supporting evidence from the associated modalities. Our approach offers several practical advantages: (i) event-level abstraction that better reflects how video content is naturally structured, enabling more reliable localization compared to frame-level retrieval; (ii) structured multi-modal fusion that aggregates speech, text, and visual cues at the event level, allowing complementary information to be more effectively utilized during reasoning; and (iii) plug-and-play compatibility with existing LVLM backbones, requiring no additional training or reliance on proprietary models. Experiments on Video-MME, MLVU, and LongVideoBench show that this event-centric design consistently outperforms frame-level retrieval baselines, highlighting the importance of modeling temporal structure for long-video understanding.", "authors": ["Yuhao Qin", "Junbo Wang", "Yuke Li", "Yining Zhu"], "base_id": "2610.08674", "categories": ["cs.CV"], "id": "2610.08674v1", "pdf_url": "https://arxiv.org/pdf/2610.08674v1", "primary_category": "cs.CV", "published": "2026-10-06T16:57:46Z", "query_hits": ["q2_long_video", "q5_videolang_long"], "seq": 30, "title": "EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding", "updated": "2026-10-06T16:57:46Z"}
|
|
{"abstract": "Vision-Language-Action (VLA) models have achieved impressive results in robotic manipulation and ground-mobile navigation, yet language-conditioned control of unmanned aerial vehicles (UAVs) in smart warehouses remains largely unexplored, hindered by the lack of benchmarks that jointly provide continuous low-level flight actions, fine-grained natural-language target descriptions, and realistic industrial environments. This paper introduces WareFly-VLA, a photorealistic UAV VLA framework and dataset for language-guided human search, localization, and tracking in warehouse environments. It contains 507 human-teleoperated flight episodes and 8,504 high-resolution RGB transitions collected in NVIDIA Isaac Sim, each paired with a human-written appearance description of the target worker and a synchronized four-degree-of-freedom control command. Two aerial tasks are covered: target approach and person following, under occlusion, long-range search, altitude variation, and clutter. A unified benchmark of four open-source VLA architectures (SmolVLA, GR00T N1.7, pi_0 and OpenVLA) is established under a leakage-free episode-level protocol at two control rates. The results show that language-conditioned aerial control in warehouses is far from solved: performance drops substantially under strict generalization settings, continuous action modeling consistently outperforms discrete action tokenization, only the forward channel is reliably learnable from a single frame, and current foundation-model interfaces transfer poorly from ground and humanoid embodiments to aerial platforms. The synchronized video, language, action, pose, and difficulty annotations further support world-model research. The dataset, baselines, and evaluation protocol are released to support language-grounded aerial autonomy in smart warehouses.", "authors": ["Thinh D. Le", "Son T. Nguyen", "Duong Q. Nguyen", "Dung D. Le", "Ngo Anh Vien", "H. Nguyen-Xuan"], "base_id": "2610.08526", "categories": ["cs.RO", "cs.CV"], "id": "2610.08526v1", "pdf_url": "https://arxiv.org/pdf/2610.08526v1", "primary_category": "cs.RO", "published": "2026-10-06T15:21:14Z", "query_hits": ["q5_videolang_long"], "seq": 31, "title": "WareFly-VLA: A Vision-Language-Action Framework for UAV Navigation and Human Tracking in Smart Warehouses", "updated": "2026-10-06T15:21:14Z"}
|
|
{"abstract": "Long-video understanding places substantial demands on memory, as answering questions often requires retrieving information distributed across extended temporal spans. Existing approaches broadly follow two paradigms: query-driven exploration, which is sensitive to localization errors, and query-independent memory construction, which may omit question-specific details. We introduce VideoTapestry, a training-free multi-agent framework that adapts a preconstructed hierarchical video memory through coarse-to-fine, query-driven refinement. The preconstructed memory organizes video content into three levels, capturing global narrative context, event-level temporal structure, and fine-grained relational evidence, respectively. To support coarse-to-fine localization and observation, we assign a specialized agent to each level, keeping retrieval and refinement within a scale-specific context. Guided by the query, these agents revisit relevant video regions and enrich layer-wise memories with targeted multimodal observations. Their refinements are assembled according to the original hierarchy into a composite query-adaptive memory, preserving global context in a compact form while retaining fine-grained evidence along query-relevant branches for final reasoning. Compared with direct GPT-5.5 inference, VideoTapestry achieves absolute accuracy gains of 17.2%, 14.9%, 9.8%, and 7.0% on LVBench, LongVideoBench (Long), Video-MME (Long), and EgoSchema, respectively, achieving the state-of-the-art results among all competitors.", "authors": ["Yucheng Liu", "Yufei Yin", "Mingxiao Feng", "Jiajun Deng", "Wengang Zhou", "Houqiang Li"], "base_id": "2610.06672", "categories": ["cs.CV", "cs.AI"], "id": "2610.06672v1", "pdf_url": "https://arxiv.org/pdf/2610.06672v1", "primary_category": "cs.CV", "published": "2026-10-05T16:44:25Z", "query_hits": ["q2_long_video"], "seq": 32, "title": "VideoTapestry: Query-Adaptive Memory Refinement for Multi-Agent Long-Video Understanding", "updated": "2026-10-05T16:44:25Z"}
|
|
{"abstract": "Despite their impressive performance on a wide range of video understanding tasks, current Vision Language Models (VLMs) are predominantly designed for offline scenarios and struggle to handle online streaming videos that demand low latency response. Several studies have explored memory and token compression strategies in an attempt to adapt offline VLMs for streaming video understanding tasks. However, through our probing experiment, we identify that most existing works tend to progressively lose long context information as length of input stream increases. To address this, we propose ReMem, a novel training-free adaptation technique that enables VLMs to process streaming videos of arbitrary lengths while improving their long context information retention capability. ReMem exploits memory from two perspectives, implemented as two core components. The Streaming Context Memory (SCM) continuously compresses historical context with query-independent attention. The Retrieved Vision Memory (RVM) then retrieves the most salient, query-relevant context from memory to augment the VLM's input. Comprehensive experiments demonstrate that the proposed ReMem achieves state-of-the-art (SOTA) performance across a variety of widely used benchmarks, spanning both streaming video and general long video understanding tasks.", "authors": ["Li Yiheng", "He Xu", "Wang Shaobo", "Shao Ling", "Lu Shijian"], "base_id": "2610.05940", "categories": ["cs.CV", "cs.AI"], "id": "2610.05940v1", "pdf_url": "https://arxiv.org/pdf/2610.05940v1", "primary_category": "cs.CV", "published": "2026-10-05T07:56:41Z", "query_hits": ["q2_long_video", "q3_longctx_cv"], "seq": 33, "title": "ReMem: Streaming Video Understanding With Long Context Retention", "updated": "2026-10-05T07:56:41Z"}
|
|
{"abstract": "The next generation of multimodal research agents must reason over long-lived research histories rather than short model completions. During a single task, an agent may repeatedly search the web, inspect visual evidence, revisit earlier hypotheses, and accumulate tens of thousands of tokens of multimodal context. Despite this trend, online RL for multimodal research agents remains largely confined to shorter contexts and interaction horizons. We push online RL training to 128k context and 75+ tool-interaction turns. To our knowledge, this is the first online multimodal deep-research RL study trained at 128k context, and the first trained with a 75 tool-turn horizon. Scaling to this regime exposes several practical limitations of conventional RL training. Early in training, weak policies make poor use of large interaction budgets, causing expensive rollouts with little reward improvement. Later, policy entropy can collapse before performance has saturated, prematurely ending useful learning. We introduce Long-MDR, a three-component training recipe designed specifically for this setting: On-Policy Distillation Warmup, Progressive Horizon Expansion, and Entropy-Triggered Rescue. Together, these techniques improve both the learning efficiency and stability of long-horizon RL, enabling continued gains in a regime where direct training is slow and costly. Code will be available at https://github.com/navileon/Long-MDR", "authors": ["On Tai Tang", "Huanyao Zhang", "Jiepeng Zhou", "Wentao Zhang"], "base_id": "2610.05195", "categories": ["cs.CV"], "id": "2610.05195v3", "pdf_url": "https://arxiv.org/pdf/2610.05195v3", "primary_category": "cs.CV", "published": "2026-10-04T13:08:13Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 34, "title": "Long-MDR: Long-Context Reinforcement Learning for Multimodal Deep-Research Agents", "updated": "2026-10-21T01:27:11Z"}
|
|
{"abstract": "Efficient long-video understanding with vision-language models (VLMs) is often framed as selecting informative frames or visual tokens at a fixed native resolution. We show that per-frame resolution can instead be traded for denser temporal coverage, while front-end decoding latency depends on the size of the candidate pool rather than the final token budget. An empirical study across multiple VLMs and long-video benchmarks yields three findings: dense low-resolution sampling outperforms sparse native-resolution sampling at matched token budgets; resolution-sensitive tasks benefit from selected high-resolution frames; and front-end decoding dominates wall time for hour-long videos. Motivated by these findings, we introduce LoHi, a training-free, single-pass framework that combines a dense low-resolution video stream with sparse high-resolution image frames through the VLM's native video and image pathways. LoHi-Anchor selects high-resolution frames using codec I-frame metadata, while LoHi-SemDiv uses query relevance and visual diversity over CLIP features. Across three long-video benchmarks, LoHi improves average accuracy by 10.6 percentage points over the native-resolution baseline at a matched token budget and by 5.2 percentage points over the strongest prior efficiency method. It also reduces front-end decoding latency by up to 7x on hour-long videos. Project page: https://sixundong.com/projects/lohi", "authors": ["Sixun Dong", "Wei Li", "Andong Deng", "Qi Qian", "Victor Zhu", "Zhengping Ji", "Chen Chen"], "base_id": "2610.04318", "categories": ["cs.CV"], "id": "2610.04318v1", "pdf_url": "https://arxiv.org/pdf/2610.04318v1", "primary_category": "cs.CV", "published": "2026-10-03T06:13:46Z", "query_hits": ["q2_long_video"], "seq": 35, "title": "Rethinking Long-Video Efficiency: A Joint Allocation Perspective on Frames, Pixels, and Front-End Latency", "updated": "2026-10-03T06:13:46Z"}
|
|
{"abstract": "Long-video generation and world models have shown strong potential for interactive entertainment and embodied simulation by predicting future observations conditioned on user actions and historical memory. However, as memory sequences grow longer and their structures become increasingly complex, managing long-range spatial context becomes increasingly challenging, calling for a more intelligent and systematic memory-management strategy. Building on the advancing spatial reasoning capabilities of multimodal large language models (MLLMs) and the broader vision of unified models, we propose Spatial Memory Intelligence (SMI), the first framework to systematically employ an understanding model for spatial-memory management in long-video world models. SMI introduces four coordinated atomic operations: spatial clustering, within-cluster sparsification, action-aware retrieval, and reliability-aware filtering. Extensive experiments across multiple baselines, benchmarks, and world-model backbones demonstrate the effectiveness and generalizability of SMI, achieving comprehensive improvements in memory sparsity, generation stability, and spatial consistency.", "authors": ["Ying Yang", "Guiyu Zhang", "Lianghua Huang", "Chang Nie", "Chenyang Si", "Haofan Wang", "Shaoshuai Shi", "Li Jiang"], "base_id": "2610.02521", "categories": ["cs.CV"], "id": "2610.02521v2", "pdf_url": "https://arxiv.org/pdf/2610.02521v2", "primary_category": "cs.CV", "published": "2026-10-01T21:50:57Z", "query_hits": ["q2_long_video"], "seq": 36, "title": "Spatial Memory Intelligence: Endowing World Models with Understanding-Driven Long-Term Memory", "updated": "2026-10-08T18:02:44Z"}
|
|
{"abstract": "We present OmniSeek, an agentic framework that transforms an Omni Large Language Model (Omni-LLM) into an active, multi-turn reasoning agent with native tool use. Rather than passively processing an entire audio-visual sequence in a single forward pass, OmniSeek makes evidence acquisition part of the reasoning process: it dynamically decides whether to look or listen, and over which temporal window, to retrieve sparse but critical evidence across different modalities within long contexts. Through an iterative multi-turn protocol, the retrieved raw audio or visual segments are appended back into the context to support subsequent reasoning. To cold-start this capability, we build a data engine that synthesizes OmniTraj-170K, a corpus of multi-hop Chain-of-Thought trajectories with interleaved audio and visual evidence. We first supervise the model on these trajectories to instill multi-turn tool-use behavior, and then further optimize the policy via a two-stage reinforcement learning with verifiable rewards. Moreover, we introduce an Audio-Visual Necessity objective that explicitly rewards successful trajectories whose reasoning depends on both modalities, discouraging single-modality shortcuts. Extensive experiments across a wide range of benchmarks demonstrate that OmniSeek learns adaptive cross-modal evidence seeking and consistently improves audio-visual reasoning performance.", "authors": ["Haibo Wang", "Jiteng Mu", "Jialu Li", "Jingru Yi", "Yuanjun Xiong", "Jianming Zhang", "Lifu Huang", "Mingze Xu"], "base_id": "2610.02181", "categories": ["cs.CV"], "id": "2610.02181v1", "pdf_url": "https://arxiv.org/pdf/2610.02181v1", "primary_category": "cs.CV", "published": "2026-10-01T17:58:16Z", "query_hits": ["q3_longctx_cv"], "seq": 37, "title": "OmniSeek: Native Tool Integration for Multi-turn Audio-Visual Reasoning", "updated": "2026-10-01T17:58:16Z"}
|
|
{"abstract": "Feedforward foundation models have recently shown remarkable 3D reconstruction capabilities. However, existing models exhibit large tracking drift in long-context streaming reconstruction due to error accumulation. In this paper, we revisit loop closure with streaming reconstruction foundation models to enable accurate, drift-free, kilometer-scale reconstruction. Specifically, our method detects loop candidates through global descriptor retrieval, and constructs loop-conditioned windows to estimate the relative poses between looped frames. Given the observation that our adopted streaming reconstruction backbone produces a globally consistent scale, we optimize all frame poses on the SE(3) manifold with sequential and loop closure constraints, avoiding the pose graph optimization on the Sim(3) or higher-dimensional SL(4) manifolds employed in prior works. Extensive experiments show that our method reduces drift and produces consistent geometry on kilometer-scale sequences, significantly outperforming the state of the art. Code is available at https://github.com/MoyangLi00/CLoSeR.git.", "authors": ["Moyang Li", "Zihan Zhu", "Wei Zhang", "Marc Pollefeys", "Daniel Barath"], "base_id": "2610.01927", "categories": ["cs.CV", "cs.RO"], "id": "2610.01927v1", "pdf_url": "https://arxiv.org/pdf/2610.01927v1", "primary_category": "cs.CV", "published": "2026-10-01T15:59:53Z", "query_hits": ["q3_longctx_cv"], "seq": 38, "title": "CLoSeR: Closing the Loop for Long-Context Streaming Reconstruction", "updated": "2026-10-01T15:59:53Z"}
|
|
{"abstract": "While recent Multimodal Large Language Models (MLLMs) have shown promise in affective computing, their reasoning capabilities are largely confined to short video clips with limited interactions. However, real-world emotions are not merely isolated instantaneous reactions but dynamic and cumulative processes deeply shaped by past experiences and ongoing events. To bridge this gap, we introduce LongEmoBench, a benchmark dedicated to emotion understanding and reasoning in long videos. It assesses progressive capabilities scaling from continuous scene interactions to complex episodic developments. Furthermore, we propose LongEmo, a novel memory-augmented agentic framework designed to tackle the immense challenges of long-range affective reasoning. LongEmo processes continuous video streams to construct an Event Memory Graph, explicitly modeling long-range dependencies and capturing emotional dynamics across discrete events. Given a question, the agent retrieves a query-relevant event stream from the graph, iteratively integrating multimodal memories and relational dependencies to deduce the final answer. Extensive evaluations of 17 representative methods reveal that they struggle significantly with emotion understanding and reasoning in long videos. In contrast, LongEmo achieves state-of-the-art performance, demonstrating the efficacy of its event-centric memory architecture.", "authors": ["Shuo Zhang", "Yifan Zhou", "Han Wang", "Jinsong Zhang", "Jingyu Li", "Hongbing Li", "Zhejun Zhang", "Chengyi Zhao", "Yuquan Hao", "Yitong Liu", "Jiyin Li", "Ruiqi Tang", "Zixuan Lin", "Yi Luo", "Xurui Zhang", "Ronghao Chen", "Huacan Wang", "Lei Li"], "base_id": "2609.40079", "categories": ["cs.CV", "cs.AI"], "id": "2609.40079v1", "pdf_url": "https://arxiv.org/pdf/2609.40079v1", "primary_category": "cs.CV", "published": "2026-09-30T16:27:25Z", "query_hits": ["q2_long_video"], "seq": 39, "title": "LongEmo: Towards Emotion Understanding and Reasoning in Long Videos", "updated": "2026-09-30T16:27:25Z"}
|
|
{"abstract": "Ultra-long video temporal grounding requires balancing long-range evidence search with fine-grained event understanding under a limited visual budget, yet existing agentic methods still rely largely on predefined policies and tool capabilities. Motivated by this, we propose a novel policy-tool coevolution framework that jointly evolves high-level policies and executable media tools from the agentic reasoning trajectories of a VLM, forming a reusable skill without updating model parameters. During evolution, an external skill updater distills transferable task experience in long-video temporal grounding, accordingly refining the orchestration of long-range image-based and fine-grained video-based observations. Alongside these policy updates, the updater employs its coding capabilities to upgrade existing tools or create new ones, adapting the tools to long-video evidence acquisition. Equipped with the evolved skill, the VLM autonomously orchestrates tools under the guidance of the evolved policy, coordinating image and video observations for agentic inference without relying on a separate, stronger planning model. Extensive experiments spanning five benchmarks and three VLMs show that policy-tool coevolution consistently improves temporal grounding accuracy in ultra-long videos while reducing visual token cost at inference, and that the evolved skill yields substantial performance gains on general long-video QA without additional task-specific evolution, demonstrating the effectiveness and generalizability of our framework for long-video understanding.", "authors": ["Yiduo Jia", "Muzhi Zhu", "Jinchuan Shi", "Hao Zhong", "Yuling Xi", "Ke Liu", "Hao Chen"], "base_id": "2609.40048", "categories": ["cs.CV"], "id": "2609.40048v1", "pdf_url": "https://arxiv.org/pdf/2609.40048v1", "primary_category": "cs.CV", "published": "2026-09-30T16:14:14Z", "query_hits": ["q2_long_video"], "seq": 40, "title": "CoEvoWhen: Policy-Tool Coevolution for Ultra-Long Video Temporal Grounding", "updated": "2026-09-30T16:14:14Z"}
|
|
{"abstract": "Vision-language models face a fundamental scaling bottleneck: the number of visual tokens grows with both temporal duration and spatial resolution, making long-video understanding expensive for the vision encoder and the language model. Existing methods often compress visual tokens after dense encoding, creating a mismatch between the representation used during training and the compact interface required at deployment. We present CoVisco, a codec-native vision encoder with native token compression for unified image-video understanding. By combining codec-native input support with segmented attention, CoVisco can encode long visual inputs in a single forward pass without forming dense patch-to-patch interactions across all frames. Each temporal segment is equipped with learnable abstract tokens that learn a compact segment-level representation, while fine-grained patch tokens remain available throughout the encoder. Alternating intra-segment and abstract-communication layers preserve video-level context through the abstract-token channel. A lightweight selector further exposes either abstract tokens alone or abstract tokens augmented with a runtime-selected subset of patch tokens, yielding a compact visual interface that reduces the visual context and prefill burden of downstream MLLMs while retaining fine-grained evidence when needed. Pretrained with contrastive objectives on 565M image--text pairs and 6.4M videos, CoVisco shows competitive performance on video-oriented embedding and multimodal understanding benchmarks. In the evaluated four-segment, 64-frame setting, abstract-only inference uses only 400 visual tokens while achieving video-understanding performance close to, and on some benchmarks exceeding, OneVision-Encoder. Selected patch tokens further improve fine-grained video reasoning. Project URL: https://github.com/ernie-research/CoVisco.git", "authors": ["Yulong Liu", "Xiaotian Han", "Junyuan Shang", "Yuchen Ding", "Zhenyu Zhang", "Shuohuan Wang", "Guibo Zhu", "Sirui Han", "Dianhai Yu"], "base_id": "2609.39924", "categories": ["cs.CV", "cs.AI"], "id": "2609.39924v1", "pdf_url": "https://arxiv.org/pdf/2609.39924v1", "primary_category": "cs.CV", "published": "2026-09-30T15:12:10Z", "query_hits": ["q2_long_video"], "seq": 41, "title": "CoVisco: Codec-Native Vision Encoder with Native Token Compression for Unified Image-Video Understanding", "updated": "2026-09-30T15:12:10Z"}
|
|
{"abstract": "Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware front-ends read the motion vectors and residuals that encoding produced, but in their deployed form each predictive frame is still tokenized on its own: the tokens are a function of the current primitives, not of a carried reference. We argue that a more natural function is of both---the current primitives and a carried reference. A clip and its time reversal share the same frames and differ only in the order of changes---an axis that symmetric pooling discards by construction, and that is non-empty in the frozen vision features VideoLMs use---and the codec recurrence already composes those changes in order against a reference state. We introduce RESUME, a stateful codec representation: an anchor I-frame initializes a compact latent state, each subsequent predictive frame is consumed as an update to that state, and a shared readout exposes VideoLM-compatible tokens from the accumulated state. Codec prediction is thereby kept at the representation level and handed to the language model as a trajectory, not as a set of independent token groups. At the same per-predictive-frame token budget as prior codec-aware methods, a predictive frame enters the language model as a readout of what the front-end already knows, not as an encoding of the current primitives alone. Across ten benchmarks, the gains concentrate on temporal reasoning: on all three temporal benchmarks RESUME improves over both the RGB-frame baseline LLaVA-Video-7B (by 2.8, 5.1, and 3.9 points on TempCompass, TOMATO, and MVBench) and the codec-based baseline CoPE-7B, while staying competitive on general and long-form QA. Frozen-transition tests further show anchor dependence, order sensitivity, and useful rollout behavior beyond the training horizon.", "authors": ["Can Zhang", "Xiaotian Han", "Junyuan Shang", "Yuchen Ding", "Zhenyu Zhang", "Shuohuan Wang", "Dianhai Yu", "Ruirui Li"], "base_id": "2609.39563", "categories": ["cs.CV"], "id": "2609.39563v1", "pdf_url": "https://arxiv.org/pdf/2609.39563v1", "primary_category": "cs.CV", "published": "2026-09-30T12:00:41Z", "query_hits": ["q5_videolang_long"], "seq": 42, "title": "RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling", "updated": "2026-09-30T12:00:41Z"}
|
|
{"abstract": "Recent advances have enabled unified omni-modal models in understanding audio, vision, and language. However, existing benchmarks, training data, and learning methods largely treat the modalities independently, leaving the capability of audio-visual joint reasoning poorly evaluated and insufficiently elicited. We address this gap with a benchmark, data engine, and learning method. First, we introduce OmniReasoningBench, a benchmark where both audio and visual evidence are indispensable. It comprises 1,150 multiple-choice and open-ended questions across two tasks, reasoning over video and reasoning beyond video. Second, we develop a data engine OmniQA. It automatically constructs evidence-grounded QA pairs that explicitly necessitate audio-visual joint reasoning, together with time-stamped clue chains that guide the annotation of thinking process. Besides our benchmark, this engine produces training data OmniReasoning-SFT-112K and OmniReasoning-RL-19K. Finally, we propose an on-policy self-distillation method Modality-Factored Self-Distillation (MFSD). It evaluates each sampled response under modality-specific clue contexts, disentangling the contributions of individual clues and their cross-modal interactions for token-level credit assignment. With our training data and learning method, our model OmniReasoning-30B-A3B achieves 50.0% on OmniVideoBench and 42.5% on OmniReasoningBench, improving the base model Qwen3-Omni-30B-A3B-Thinking by 12.8 and 9.3 percentage points, respectively. Moreover, it delivers substantial gains on general and long-video benchmarks, including Video-MME-v2. We hope our work offers a solid step for facilitating future research in omni-modal joint reasoning.", "authors": ["Junming Lin", "Yuxuan Wang", "Zhenxin Lei", "Yuxin Liu", "Ruixun Liu", "Yinsong Yan", "Ling Wang", "Minghao Han", "Yunfei Chu", "Shun Lei", "Xueyao Zhang", "Qize Yang", "Jin Xu", "Yiwu Zhong"], "base_id": "2609.39490", "categories": ["cs.CV", "cs.AI"], "id": "2609.39490v1", "pdf_url": "https://arxiv.org/pdf/2609.39490v1", "primary_category": "cs.CV", "published": "2026-09-30T10:51:24Z", "query_hits": ["q2_long_video"], "seq": 43, "title": "OmniReasoning: Pushing the Limits of Audio-Visual Joint Reasoning", "updated": "2026-09-30T10:51:24Z"}
|
|
{"abstract": "Vision-language models (VLMs) can answer questions about hour-long videos, but processing every frame is prohibitively expensive, even though the evidence for a question usually spans only a few seconds. Video agents, i.e., harness programs wrapped around a frozen VLM, address this by observing the video selectively, yet existing harnesses are hand-crafted by experts through slow build-and-test cycles. We propose VidHarness, a framework that automates harness design for cost-efficient long video understanding, in which a harness proposer iteratively evolves harnesses based on execution feedback from an evolution environment. To escape the local optima of greedy refinement, we organize the evolution as Monte Carlo tree search (MCTS), and to reduce the evaluation cost, we integrate uncertainty-aware multi-fidelity validation, which screens new harnesses on a few questions and promotes only the promising ones. Since the best harness varies with the frame budget, we further introduce a mixture-of-harness that routes each question to a harness specialized for its budget. VidHarness sets new state-of-the-art results on LongVideoBench, Video-MME, and Video-Holmes, outperforms the strongest hand-crafted video agent by up to $11.2$ points, and generalizes to the knowledge-intensive benchmarks Video-MMMU and MMVU with fewer than half of the frames of uniform sampling.", "authors": ["Susan Liang", "Jianmin Wu", "Daxiang Dong"], "base_id": "2609.38413", "categories": ["cs.CV"], "id": "2609.38413v1", "pdf_url": "https://arxiv.org/pdf/2609.38413v1", "primary_category": "cs.CV", "published": "2026-09-29T19:06:54Z", "query_hits": ["q2_long_video"], "seq": 44, "title": "VidHarness: Evolving Agent Harnesses for Cost-Efficient Long Video Understanding", "updated": "2026-09-29T19:06:54Z"}
|
|
{"abstract": "Video diffusion models are increasingly developed into specialized models for diverse downstream tasks, and this development often includes a distillation stage, for example to accelerate sampling or to improve long-video generation. This stage is typically repeated for every specialized model. We introduce LongLive-Plug, a once-for-all distillation framework that learns reusable capabilities as LoRAs on a base model for training-free, plug-and-play deployment to compatible downstream models. These capabilities include single-pass classifier-free guidance, few-step sampling, and long-context error correction for autoregressive generation. The adapters remain reusable even when downstream models add conditioning branches, expand output channels. Despite training at a fixed guidance scale, our dedicated CFG LoRA provides text guidance control through its inference weight. Combining it with a few-step LoRA simultaneously preserves few-step generation and CFG controllability on downstream tasks. We verify training-free deployment on 54 downstream models across three backbone families and eight task categories, including world modeling, robotics, editing, and multimodal generation. The approach may support additional compatible models. Each capability can thus be distilled once per backbone family and reused without per-target retraining.", "authors": ["Shuai Yang", "Luozhou Wang", "Wei Huang", "ZhiFei Chen", "Bohan Zhang", "Xiao Fu", "Qianli Ma", "Chen-Hsuan Lin", "Weian Mao", "Bryan Chu", "Song Han", "Yukang Chen"], "base_id": "2609.38154", "categories": ["cs.CV"], "id": "2609.38154v1", "pdf_url": "https://arxiv.org/pdf/2609.38154v1", "primary_category": "cs.CV", "published": "2026-09-29T17:58:49Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 45, "title": "LongLive-Plug: Once-for-All Distillation for Video Generation", "updated": "2026-09-29T17:58:49Z"}
|
|
{"abstract": "Modern vision-language models (VLMs) have shown promising results in long-video understanding due to the rich semantic information they can capture. However, most methods focus on coarse captioning of extracted image frames that are computationally inefficient and require models with large context windows. While past work has explored efficient methods through multimodal retrieval-augmented generation (RAG), they rely on lossy embeddings that lose temporal context and fine-grained detail. Few works to date have investigated how VLM-based query-relevant information retrieval can be optimized. We introduce LazySloth, an efficient tree-based search method that speeds up video comprehension and retrieval tasks 2.9-8.3x (compared to existing agentic methods) through bounded captioning of portions of the video considered irrelevant by a VLM of the video. Compared to contemporary specialized video-understanding VLMs and RAG-based methods, LazySloth achieved similar or better final task accuracy across two recent open-source base VLMs--Gemma 4 31B and Qwen3.6 27B--across four benchmarks. LazySloth reduced the gap between the base open-source model and a closed-source model, GPT-4o. Ablations showed that replacing VLM scene understanding with CLIP-based retrieval cost 8.8-19.9% in accuracy, while lazy tree construction matches eager construction at a fraction of the captioning cost. With LazySloth, we demonstrate the possibility of faster long-video comprehension without substantial loss in performance.", "authors": ["Arka Mukherjee", "Kaleen Shrestha", "Larissa Zhu", "Maja Matari\u0107"], "base_id": "2609.37426", "categories": ["cs.CV", "cs.AI"], "id": "2609.37426v1", "pdf_url": "https://arxiv.org/pdf/2609.37426v1", "primary_category": "cs.CV", "published": "2026-09-29T12:53:57Z", "query_hits": ["q2_long_video"], "seq": 46, "title": "LazySloth: Bounded LLM-based Lazy Tree Search for Fast Long Video Comprehension", "updated": "2026-09-29T12:53:57Z"}
|
|
{"abstract": "Recent Recommendation Agents (RecAgents) offer a promising alternative by shifting recommendation to an active, user-side paradigm, where generative agents autonomously perceive external platforms, reason over user preferences, and execute decisions. However, existing RecAgents still suffer from two critical limitations: brittle item perception based on noisy and heterogeneous item pages, and inefficient long-context reasoning over extended user histories and multi-step interaction traces. To address these challenges, we propose a novel recommendation agent framework, termed as ReMem, that combines OCR-based multimodal perception with time-evolving dynamic memory. Instead of parsing raw HTML, ReMem observes item pages through screenshots and extracts structured multimodal information via an OCR tool, enabling a more humanoid and platform-agnostic perception mechanism. To support long-horizon preference modeling, ReMem further introduces a chunk-wise sequential memory update strategy, where the agent selectively maintains a fixed-size memory of informative historical interactions while processing arbitrarily long contexts with linear inference complexity and bounded context length. This design allows the agent to preserve evolving user preferences without relying on external memory modules or disrupting the standard autoregressive generation process. To enhance the dynamic memory instruction, we further develop a multi-memory GRPO variant, which propagates the final-answer advantage to all intermediate conversations that contribute to the final response. Extensive experiments on three datasets demonstrate that ReMem consistently outperforms state-of-the-art baselines, achieving an average improvement of 5.16\\% across three recommendation agent tasks, namely searching, ranking, and judging.", "authors": ["Haohao Qu", "Yongcheng Jing", "Chun Hin Chan", "Shanru Lin", "Wenqi Fan", "Dacheng Tao"], "base_id": "2609.37311", "categories": ["cs.AI", "cs.IR"], "id": "2609.37311v2", "pdf_url": "https://arxiv.org/pdf/2609.37311v2", "primary_category": "cs.AI", "published": "2026-09-29T11:46:23Z", "query_hits": ["q1_longctx_multimodal"], "seq": 47, "title": "ReMem: Rethinking Perception and Memory in Long-Context Recommendation Agents", "updated": "2026-10-03T13:53:12Z"}
|
|
{"abstract": "Long-context reasoning in large language models incurs substantial computation and memory costs. Visual text compression (VTC) reduces input length by rendering text as images, but fixed-resolution rendering creates a compression-performance trade-off: low DPI saves tokens at the expense of legibility, whereas high DPI spends tokens on irrelevant content. We introduce FocusVTC, which breaks this trade-off through adaptive resolution while preserving general multimodal capabilities. It combines compressed low-DPI global views with selective region enhancement, integrating enhanced views into ongoing reasoning. We construct 29.4K high-quality Reasoning-Evidence Localization (REL) chain-of-thought examples (REL-CoT) that link reasoning traces to page indices and bounding boxes. Multi-resolution REL supervised fine-tuning (REL-SFT) teaches the model to localize relevant regions, and Group Relative Policy Optimization learns when to enhance resolution and how to use the resulting observations, without a separate continual-pretraining stage. At 72 DPI on RULER v1, FocusVTC scores 87.4 at $2.9\\times$ input compression, including tool observations, versus 57.5 for Glyph at $3.0\\times$ input compression. It surpasses its text-input backbone on LongBench (56.40 versus 55.86), improves the MRCR macro-average by 13.91 points, and achieves a 51.19 macro-average on VTCBench. The MRCR latency evaluation also shows a $2.79\\times$ online end-to-end speedup over Text. General multimodal capabilities are preserved, with MMMU increasing from 65.12 to 66.73 and MME from 2424.02 to 2457.62.", "authors": ["FangZhi Zhong", "Xuerui Qiu", "Yuqi Pan", "Ya Liu", "Shaowei Gu", "Bo Xu", "Guoqi Li"], "base_id": "2609.36651", "categories": ["cs.CV", "cs.AI"], "id": "2609.36651v1", "pdf_url": "https://arxiv.org/pdf/2609.36651v1", "primary_category": "cs.CV", "published": "2026-09-29T03:51:06Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 48, "title": "FocusVTC: Efficient and High-Performance Visual Text Compression with Adaptive Resolution", "updated": "2026-09-29T03:51:06Z"}
|
|
{"abstract": "Standard video generators do not natively compact historical context into reusable memory tokens. As generation continues, the growing history makes it increasingly difficult to retain information from earlier frames due to long-context degradation. Key-frame-based approaches address this challenge by retaining selected past frames, but can discard information needed for future generation. Rather than relying on frame selection alone, we study whether a frozen video generator can supply the supervision needed to learn a compact representation of the history. We propose Prediction-Aligned Context Compaction (PACC), which uses a learned compressor to aggregate information across past frames into compact memory tokens. We train the compressor through on-policy distillation, using the same frozen generator both as a student when conditioned on compressed memory and as a teacher when conditioned on the full history. The student generates continuations, while the teacher provides targets for the same noisy inputs at each denoising step. Only the compressor is updated to align the student's predictions with these targets. We evaluate PACC on MBench, which jointly measures memory-event coverage and consistency. PACC outperforms the strongest baseline by 6.63 points on Causal-rCM and 3.19 points on Causal Forcing. Evaluation on VBench-Long using MovieGen prompts further shows that PACC produces minute-long videos with generation quality competitive with baselines. Together, these results show that learning to compact historical context can improve long-video memory without modifying the underlying generator.", "authors": ["Xiaoyu Wu", "Weihang Guo", "Yifei Wang", "Xinze Feng", "Lydia E. Kavraki", "Zhiwei Steven Wu"], "base_id": "2609.36364", "categories": ["cs.CV", "cs.AI", "cs.LG"], "id": "2609.36364v1", "pdf_url": "https://arxiv.org/pdf/2609.36364v1", "primary_category": "cs.CV", "published": "2026-09-28T22:52:57Z", "query_hits": ["q3_longctx_cv"], "seq": 49, "title": "Compress to Remember: Learning Compact Memory via On-Policy Distillation for Long Video Generation", "updated": "2026-09-28T22:52:57Z"}
|
|
{"abstract": "Long video understanding relies on video memory to overcome the context limits of multimodal large language models. Existing methods follow a build-then-reasoning pipeline: memory is built offline for the entire video, then reasoned over as a static source. In practice a long video is shared by several questions, and this pipeline is costly at both ends: with few questions, building memory for the whole video costs far more than answering them; with many questions, the memory is never updated, so what is learned while answering questions is lost to the next question. To alleviate these, we introduce Sprout, an agentic framework that builds memory while reasoning: a temporal tree that sprouts detailed nodes as questions are answered. The agent watches the video segment by segment at a low frame rate, stopping when the current question can be answered, remembers each segment as a coarse node of the tree, and revisits key intervals at a higher frame rate to refine the tree with the recovered details. Once a segment is recorded as text, its video input is removed from the context history, while the original video remains reachable through the video tools. The memory tree and prior question--answer records persist across questions, so the memory is online and dynamic: built from the first question onward and updated by every question thereafter. We find that replacing accumulated video inputs with textual memory substantially reduces context usage while maintaining accuracy, with slight improvements in some settings. Across benchmarks on three models, Sprout achieves competitive or improved accuracy relative to representative offline memory methods, with no upfront construction stage and lower context cost per question.", "authors": ["Wei Chen", "Xuanyu Zheng", "Yancheng Long", "Haoyang Xu", "Kaiyu Jiang", "Bin Wen", "Tingting Gao", "Han Li", "Long Chen"], "base_id": "2609.35497", "categories": ["cs.CV"], "id": "2609.35497v1", "pdf_url": "https://arxiv.org/pdf/2609.35497v1", "primary_category": "cs.CV", "published": "2026-09-28T16:01:59Z", "query_hits": ["q2_long_video"], "seq": 50, "title": "Sprout: Building Dynamic Memory While Reasoning for Agentic Video Understanding", "updated": "2026-09-28T16:01:59Z"}
|
|
{"abstract": "Video Large Language Models (Video LLMs) have achieved remarkable progress in video understanding, but their inference efficiency is constrained by the large number of visual tokens produced by long videos. Recent video token compression methods increasingly introduce sophisticated strategies for token selection, pruning, and merging. This raises a fundamental question: how much of compression performance can be obtained by simply preserving the structure encoded in the visual representations? We investigate this question with SimpleCluster, a simple and training-free baseline that performs position-aware cross-frame clustering in the visual feature space and represents each cluster using the mean of its original visual features. Extensive experiments across four video understanding benchmarks and three representative Video LLMs show that SimpleCluster achieves competitive or superior performance over recent compression methods across a wide range of token retention ratios, with particularly strong robustness under extremely low retention rates (e.g., 1%). To understand this behavior, we analyze the feature space preserved by different compression methods in terms of local approximation fidelity and global coverage. The results show that stronger downstream performance is consistently associated with better preservation of the original visual feature distribution, especially its global coverage. These findings highlight feature-space preservation as an important consideration for video token compression under highly constrained token budgets. Our code is available at https://github.com/xiaozhang79/SimpleCluster.", "authors": ["Xiao Zhang", "Wang Zeng", "Sheng Jin", "Wentao Liu", "Chen Qian", "Shichao Kan"], "base_id": "2609.35394", "categories": ["cs.CV"], "id": "2609.35394v1", "pdf_url": "https://arxiv.org/pdf/2609.35394v1", "primary_category": "cs.CV", "published": "2026-09-28T15:21:37Z", "query_hits": ["q2_long_video"], "seq": 51, "title": "Rethinking Visual Token Compression for Video Large Language Models: A Simple Yet Strong Baseline", "updated": "2026-09-28T15:21:37Z"}
|
|
{"abstract": "Long video understanding requires reasoning over hours of content that far exceed the context window of multimodal large language models (MLLMs). Existing approaches compress video into fixed-length clips, yet events are fragmented and unrelated content is merged, yielding noisy retrieval. To overcome this challenge, we propose EMMA, an Event-driven Multimodal Memory system with Multi-Agent retrieval, which organizes memory around semantic events in three stages: (1) Event segmentation: a dual-stream detector adaptively locates event boundaries by fusing visual semantic drift with spoken transitions, rather than fixed windows. (2) Memory construction: built on the detected events, a three-level hierarchy of raw video, clip-level memory, and event-level memory stores content at progressively higher semantic abstraction. (3) Memory retrieval: a multi-agent framework collaboratively queries this hierarchy through query decomposition, tiered search, evidence validation, and answer generation, narrowing evidence from coarse events to precise clips. Extensive experiments on three challenging benchmarks demonstrate substantial improvements, showing the effectiveness of our event-aligned memory system. Code, prompt templates, and reproduction scripts are available at https://github.com/BAI-LAB/EMMA.", "authors": ["Ruochen Zhao", "Jiazheng Kang", "Gezi Li", "Dongying Lin", "Ting Bai"], "base_id": "2610.23892", "categories": ["cs.CV", "cs.AI"], "id": "2610.23892v1", "pdf_url": "https://arxiv.org/pdf/2610.23892v1", "primary_category": "cs.CV", "published": "2026-09-28T11:16:45Z", "query_hits": ["q2_long_video"], "seq": 52, "title": "EMMA: Event-Driven Multimodal Memory System with Multi-Agent Retrieval", "updated": "2026-09-28T11:16:45Z"}
|
|
{"abstract": "Temporal video grounding is a key capability of advanced Multimodal Large Language Models (MLLMs) for the thorough understanding of video events, which is however often limited by repeated actions and visually similar contexts in long videos. In this paper, we study this issue from the perspective of On-policy distillation (OPD) and propose a new training regime for MLLMs termed Counterfactual Attention Policy Distillation (CAPD). In particular, OPD is a viable solution for MLLMs via providing dense teacher supervision on student-generated trajectories. But its next-token based teacher-student distillation is hard to identify the specific video segments supporting each predicted timestamp, which is critical for temporal grounding. In this case, CAPD measures how masking each temporal group changes the teacher's output distribution. The resulting counterfactual influence calibrates the teacher's attention and weights token-level distillation, allowing the student to learn the temporal evidence that affects boundary prediction. To validate CAPD, we trained it on Qwen3-VL-8B-Instruct using only 2,500 samples for one epoch, and evaluated it on the TimeLens and multiple general video benchmarks. Experimental results show that CAPD improves average recall by 12.0% relative to GRPO on TimeLens while preserving general video understanding, achieving comparable accuracy to the base model.", "authors": ["Shaobo Ju", "Haiyang Yu", "Xuecheng Wu", "Qiong Wu", "Jiacong Wang", "Fan Shi", "Jun Peng", "Yiyi Zhou"], "base_id": "2609.34581", "categories": ["cs.CV"], "id": "2609.34581v2", "pdf_url": "https://arxiv.org/pdf/2609.34581v2", "primary_category": "cs.CV", "published": "2026-09-28T08:24:34Z", "query_hits": ["q2_long_video"], "seq": 53, "title": "Counterfactual Attention Policy Distillation for Temporal Video Grounding", "updated": "2026-09-29T08:12:10Z"}
|
|
{"abstract": "Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether their judgments remain reliable when the evidence is buried in day-long videos has yet to be established. Existing benchmarks cannot answer this. Their videos are typically only a few minutes long, many answer pairs can be separated from the transcript alone, and collecting human judgments does not scale to ultra-long videos. We introduce PlaylistEval, an agentic framework that builds video-language judge benchmarks over 100-hour playlist collection without human annotation. It automatically generates questions with paired answers whose differences are controlled by causal degradation, so that every pair demands retrieval across the collection. The resulting benchmark contains 630 pairs across seven domains spanning both static and dynamic knowledge, and on a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (IAA 0.781). Evaluating 17 omnimodal and multimodal models from eight families reveals that frontier judges reach only 75.4% pairwise accuracy, while open-source judge models perform far behind. We further show that both retrieval and final judgment depend on using multiple modalities, and that judge accuracy degrades as the playlist set grows. We release our pipeline, benchmark, and evaluation code at https://playlisteval.github.io.", "authors": ["Shayekh Bin Islam", "Hwanjun Song"], "base_id": "2609.34314", "categories": ["cs.CV", "cs.AI", "cs.CL"], "id": "2609.34314v1", "pdf_url": "https://arxiv.org/pdf/2609.34314v1", "primary_category": "cs.CV", "published": "2026-09-28T04:54:29Z", "query_hits": ["q5_videolang_long"], "seq": 54, "title": "PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?", "updated": "2026-09-28T04:54:29Z"}
|
|
{"abstract": "Unified multimodal models combine understanding, generation, and editing within a single network, offering a promising foundation for versatile multimodal applications. However, growing multimodal contexts make KV cache storage and access increasingly costly. Existing KV cache compression methods are typically tailored to specific tasks and single-modality caches, while overlooking changes in cache importance across tasks and timesteps. However, in unified multimodal models, each task involves multiple KV cache types, and both their composition and dynamics differ across tasks. As a result, a single compression policy overlooks task- and type-specific requirements, leading to the loss of critical information and degraded quality across tasks. Based on these findings, we propose UniCache, a training-free framework for task- and type-aware KV cache compression. UniCache identifies the cache segments activated by each task and assigns suitable compression policies through offline calibration. It coordinates their parallel execution under a shared storage budget through attention-guided allocation and task-aware temporal scheduling. Experiments show that UniCache achieves $5\\times$ KV cache compression for understanding and editing and $2.5\\times$ for generation with negligible quality loss, while increasing throughput by up to $1.78\\times$ in long-context settings, significantly improving the practicality of scaling unified multimodal models to longer context.", "authors": ["Wanqi Yang", "Yuexiao Ma", "Mei Xie", "Xiawu Zheng", "Shiwei Liu"], "base_id": "2609.32831", "categories": ["cs.LG"], "id": "2609.32831v1", "pdf_url": "https://arxiv.org/pdf/2609.32831v1", "primary_category": "cs.LG", "published": "2026-09-26T18:02:55Z", "query_hits": ["q1_longctx_multimodal"], "seq": 55, "title": "UniCache: Task- and Type-Aware KV Cache Compression for Unified Multimodal Models", "updated": "2026-09-26T18:02:55Z"}
|
|
{"abstract": "Vision-language models are increasingly used to understand long videos and continuous streams. However, dense visual tokens accumulate with video duration, making long-context inference prohibitively expensive. Existing training-free visual-token selection methods reduce this cost by retaining informative tokens, but may lose coherent event evidence and fail to distinguish detailed recent observations from long-range history. We propose \\textbf{KeyRec}, a training-free framework for constructing bounded visual memory. During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank. Candidate events are proposed according to their novelty relative to previously stored events and maintained through an online add--merge--evict update. When a question arrives, a text-only router adaptively allocates a fixed readout budget between recent and event memory, without reprocessing historical frames. KeyRec operates on model-facing visual embeddings and supports both modular encoder--projector VLMs and the encoder- and projector-free NEO-ov architecture. Across four streaming and long-video benchmarks and three VLM backbones, KeyRec achieves the best compressed performance in 13 of 15 settings using only 10\\% of the dense decoder-facing visual-token budget. It outperforms the strongest compressed baseline by 2.21--18.37 points on real-time questions, achieves the best compressed result in five of six long-video settings, and performs best in every NEO-ov 2B setting.", "authors": ["Zihan Chen", "Xuejian Rong", "Xiaojuan Wang", "Boqing Gong", "Adi Zicher", "Yael Pritch", "Nikhil Karnad"], "base_id": "2609.32182", "categories": ["cs.CV", "cs.AI"], "id": "2609.32182v1", "pdf_url": "https://arxiv.org/pdf/2609.32182v1", "primary_category": "cs.CV", "published": "2026-09-26T03:15:58Z", "query_hits": ["q3_longctx_cv"], "seq": 56, "title": "KeyRec: Bounded Visual Memory for Streaming and Long-Video Understanding", "updated": "2026-09-26T03:15:58Z"}
|
|
{"abstract": "Recent video-language models increasingly adopt hybrid architectures that interleave linear and full attention layers for efficient long-context processing. While the recurrent state of linear attention remains fixed in size, the KV cache of full attention continues to grow with the video stream, making eviction necessary under a bounded memory budget. The key challenge in streaming is that eviction must occur before the question arrives, so what to retain has to be decided without the question. Existing eviction methods derive token scores from the KV cache itself, using position, attention, or key-value representations, and attention-based scores further require proxy queries or extra computation. Hybrid backbones offer another source of signal. In gated-delta linear attention, the recurrent state is updated by the residual between each input and what can already be retrieved from the state, so its change over a chunk of frames reflects how much new information the chunk brings. We propose DeltaS, a query-agnostic, training-free method that retains video chunks inducing larger normalized state change, or state drift. In a controlled comparison with the budget and retention policy held fixed, state drift outperforms position-, attention-, and key-value-based signals. With a signal costing only 1.9% of the forward pass, DeltaS surpasses the strongest query-agnostic bounded-memory baseline by 2.1 points on average across six long-video benchmarks and by 5.6 points on the longest benchmark. These results suggest that the two memories of hybrid architectures can work cooperatively. Code is available at https://github.com/MaumAI-Company/DeltaS.", "authors": ["Taeyoun Kwon", "Seungjin Kim", "Hyeonyu Kim", "Moon Hwan Kim"], "base_id": "2609.27470", "categories": ["cs.CV", "cs.CL", "cs.LG"], "id": "2609.27470v1", "pdf_url": "https://arxiv.org/pdf/2609.27470v1", "primary_category": "cs.CV", "published": "2026-09-23T07:35:52Z", "query_hits": ["q3_longctx_cv", "q5_videolang_long"], "seq": 57, "title": "DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video", "updated": "2026-09-23T07:35:52Z"}
|
|
{"abstract": "We introduce Qwen3.8-Omni-Flash, a natively multimodal agentic model for real-world multimodal productivity. Compared with previous omni models, which primarily emphasized perception and interaction, Qwen3.8-Omni-Flash substantially improves multimodal understanding and reasoning, as well as performance on long-horizon agentic tasks. These capabilities are supported by a native multimodal co-training strategy that preserves strong text-domain capabilities while facilitating the transfer of agentic capabilities from text to audio and video tasks. The model inherits the sparse mixture-of-experts (MoE) architecture of Qwen3.8-Next and extends the context window to one million tokens, supporting long-context multimodal reasoning and long-horizon planning. These advances enable integration into production workflows as a primary agent or a specialized sub-agent, supporting video editing, long-form audio and video translation, music-conditioned music video or movie generation, and video-based note or omni-skill creation. To address the lack of native audio and video support in existing agent harnesses, we release Qwen-MM-Plugins, a lightweight open-source plugin framework for multimodal productivity. We further frame real-time multimodal interaction as a system-level challenge requiring orchestration of context and memory management, tool use, and sub-agent delegation. Accordingly, we release Qwen-Live-Harness, an open-source framework for building responsive, real-time multimodal agents based on Qwen3.8-Omni-Flash. Extensive evaluations demonstrate that Qwen3.8-Omni-Flash achieves strong performance across multimodal understanding, reasoning, long-horizon agentic execution, and video productivity tasks. These results and the accompanying open-source tools support Qwen3.8-Omni-Flash as a practical foundation for deploying natively multimodal agents in research and production.", "authors": [" Qwen Team"], "base_id": "2609.25611", "categories": ["cs.CL", "cs.CV", "cs.MM"], "id": "2609.25611v1", "pdf_url": "https://arxiv.org/pdf/2609.25611v1", "primary_category": "cs.CL", "published": "2026-09-22T03:07:59Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 58, "title": "Qwen3.8-Omni: Towards Native Omni-Modal Agents", "updated": "2026-09-22T03:07:59Z"}
|
|
{"abstract": "The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.", "authors": [" DeepSeek-AI", " :", "Anyi Xu", "B. Li", "Bangcai Lin", "Bing Xue", "BingCheng Xian", "Bingzheng Xu", "Bochao Wu", "Bowei Zhang", "Boyi Deng", "C. C. Yu", "Chao Jin", "Chaofan Lin", "Chen Dong", "Chenbing Wang", "Chenfan Feng", "Chengda Lu", "Chenggang Zhao", "Chengqi Deng", "Chengyuan Zhang", "Chenhao Xu", "Chenqi Zhao", "Chenze Shao", "Chuhao Wang", "Chuqi Zhang", "Damai Dai", "Dejian Yang", "Deli Chen", "Di Huang", "Di Wu", "Donghao Li", "Erhang Li", "Eric Fu", "F. Zhou", "Fangwei Zhou", "Fangyun Lin", "Fangzhou Yuan", "Feiyu Xia", "Fucong Dai", "Guangbo Hao", "Guanglin Li", "Guanting Chen", "Guoai Cao", "Guofan Fan", "Guolai Meng", "Guowei Li", "Haichuan Zhang", "Haiyang Ma", "Haiyang Shen", "Han Li", "Han Yu", "Han Zhang", "Hangyuan Deng", "Hanwei Xu", "Hanxiang Xu", "Hanxun Zhong", "Hao Guo", "Hao Jiang", "Hao Li", "Hao Qin", "Haodong Wen", "Haofen Liang", "Haofeng Huang", "Haohua Liu", "Haoling Zhang", "Haoming Luo", "Haoran Yang", "Haotian Xu", "Haotian Yuan", "Haoting Huang", "Haowen Luo", "Haoyang Cai", "Haoyu Chen", "Haozhe Ji", "Hengran Zhang", "Hengrui Wang", "Hengxu Wu", "Honghui Ding", "Hongxuan Tang", "Huadong Wang", "Huanqi Cao", "Huazuo Gao", "Hui Qu", "Hui Zeng", "J. Yang", "J. H. Jin", "J. H. Zhang", "J. X. Zou", "Jia Yu", "Jiahui Zhou", "Jiajun Chen", "Jialiang Huang", "Jialin Zhao", "Jiamin Tang", "Jian Zhou", "Jianan Tong", "Jianwen Li", "Jiaqi Zhu", "Jiarui Wang", "Jiasheng Ye", "Jiashi Li", "Jiaxin Xu", "Jiaying Ding", "Jibai Lu", "Jiewen Hu", "Jin Yan", "Jincheng Zhai", "Jingchang Chen", "Jingcheng Hu", "Jingli Zhou", "Jingsheng Xu", "Jingting Xiang", "Jingyan Yun", "Jingyang Yuan", "Jingyuan Cheng", "Jinhua Zhu", "Jinpeng Wang", "Jinyi Chen", "Jinyi Hu", "Jiping Yu", "Jueliang Guo", "Junbo Pei", "Junbo Sun", "Junguang Jiang", "Junjie Qiu", "Junkang Zhou", "Junqi Liu", "Junren Li", "Junxian Li", "Junxiao Song", "Junyi Guo", "Kai Dong", "Kaifeng Chen", "Kaige Gao", "Kang Guan", "Kangdong Yuan", "Ke Hong", "Ke Xu", "Kefan Zhao", "Kexin Ji", "Kexin Zhang", "Kexing Zhou", "Kuai Yu", "Lan Zhang", "Lean Wang", "Lecong Zhang", "Lei Wang", "Letian Gao", "Liang Zhao", "Liansheng Xu", "Lihua Guo", "Lingxiao Luo", "Lingyue Fu", "Litao Deng", "Litong Wang", "Liyue Zhang", "Longhao Chen", "Lu Chen", "Luotian Huang", "Luyao Ma", "Luyao Wang", "M. S. Di", "Max Mei", "Menghao Ye", "Miao Cui", "Mingchuan Zhang", "Minghua Zhang", "Minghui Tang", "Mingjing Zhang", "Mingqi Wei", "Mingshu Chen", "Mingxing Liu", "Mingxu Zhou", "Mingyu Xu", "Mingyu Yang", "Mingze Wang", "Muyang Chen", "Ni Shentu", "Ning Wang", "Niufang Ning", "Panpan Huang", "Peixin Cong", "Peiyi Wang", "Peiyuan Xin", "Pengfei Ren", "Pengfei Yan", "Pengle Zhang", "Qi Kang", "Qi Tang", "Qiancheng Wang", "Qiang Li", "Qihao Zhu", "Qingyang Li", "Qinyu Chen", "Qiushi Du", "Qizhou Guo", "Rongxian Xu", "Rui Ding", "Rui Hu", "Rui Tian", "Rui Yu", "Ruidong Zhu", "Ruifan Xu", "Ruihan Yang", "Ruihang Xia", "Ruijie Lu", "Ruilin Geng", "Ruipeng Hong", "Ruiqi Ge", "Ruisong Zhang", "Ruize Sun", "Ruizhe Pan", "Runji Wang", "Runqian Chen", "Runxin Xu", "Ruohong Tian", "Ruomeng Shen", "Ruoyu Zhang", "Ryan X.", "S. H. Liu", "Shanghao Lu", "Shangyan Zhou", "Shanhuang Chen", "Shaofei Cai", "Shaoheng Nie", "Shaoyuan Chen", "Shengding Hu", "Shengkai Lin", "Shengwen Ran", "Shengyu Liu", "Shengyuan Jia", "Shi Bai", "Shi Feng", "Shicheng Xu", "Shichun Liu", "Shiqiang Hu", "Shirong Ma", "Shiyu Wang", "Shiyuan Feng", "Shufan Gong", "Shuhan Lin", "Shuiping Yu", "Shunfeng Zhou", "Shuo Yang", "Shuomeng Wang", "Shuting Guo", "Shuting Pan", "Shuying Yu", "Sinuo Cao", "Siyi Lin", "Sizhe Chen", "Songyang Chen", "Songyang Zhou", "Tao Ni", "Tao Yun", "Tian Jin", "Tian Pei", "Tian Ye", "Tianle Lin", "Tianran Ji", "Tianyi Cui", "Tianyuan Yue", "Tingting Yu", "Tongrui Xiong", "Wangding Zeng", "Wei Liu", "Wei Zhang", "Weibin Xu", "Weihao Zeng", "Weilin Zhao", "Wen Liu", "Wenfeng Liang", "Wenjie Pang", "Wenjing Luo", "Wenjing Yao", "Wenjun Gao", "Wenkai Shao", "Wenkai Yang", "Wenli Zhang", "Wenlu Wang", "Wenlve Huang", "Wenqian Yan", "Wentao Zhang", "Xi Gao", "Xiang He", "Xiang Li", "Xiangli Li", "Xiangwen Wang", "Xiangying Zhang", "Xiankui Wei", "Xiao Bi", "Xiaodong Liu", "Xiaohan Wang", "Xiaojian Qu", "Xiaokang Chen", "Xiaokang Zhang", "Xiaotao Nie", "Xiaoyao Zou", "Xiaoyuan Li", "Xicheng Guo", "Xieting Chu", "Xin Cheng", "Xin Liu", "Xin Xie", "Xinbo Xu", "Xingchao Liu", "Xingchen Liu", "Xingkai Yu", "Xingyou Li", "Xintong Yao", "Xinyang Chen", "Xinyong Jiang", "Xinyu Yang", "Xinyu Yang", "Xu Chen", "Xuanyu Wang", "Xubei Zhong", "Xuecheng Su", "Xuejie Liu", "Xuheng Lin", "Xujie Fan", "Xuncheng Zhao", "Xuwei Fu", "Y. C. Yan", "Y. H. Jiang", "Y. T. Wu", "Y. W. M.", "Y. Z. Wang", "Yafei Gao", "Yang Yang", "Yang Zhang", "Yanru Ma", "Yanwen Huang", "Yao Li", "Yao Li", "Yao Meng", "Yao Zhao", "Yaofeng Sun", "Yaohui Wang", "Yaoyang Ye", "Yehang Yin", "Yexinrui Wu", "Yi Qian", "Yi Tao", "Yi Yu", "Yichao Zhang", "Yichen Jiang", "Yicheng Wang", "Yifan Ding", "Yifan Shi", "Yifeng Peng", "Yifeng Zhai", "Yijia Wu", "Yiliang Xiong", "Yilun Wang", "Ying He", "Ying Zhou", "Yingjia Luo", "Yinmin Zhong", "Yiping Wang", "Yisong Wang", "Yixiang Zhang", "Yixiao Chen", "Yixuan Tan", "Yixuan Wei", "Yiyang Ma", "Yiyao Yang", "Yiyuan Liu", "Yizai Cai", "Yizhen Wei", "Yizhi Wang", "Yonglun Yang", "Yongqi Zhuo", "Yongqiang Guo", "Yongtong Wu", "Yu Wu", "Yu Zhang", "Yuan Bian", "Yuan Cheng", "Yuan Ou", "Yuan Sun", "Yuanfan Xu", "Yuanhang Sun", "Yuanhao Li", "Yuchen Liu", "Yuchen Yao", "Yudong Han", "Yuduan Wang", "Yuhan Wu", "Yuhao Meng", "Yuheng Zou", "YuKun Li", "Yunchuan Wang", "Yunfan Xiao", "Yunfan Xiong", "Yupeng Chen", "Yuqian Cao", "Yuqian Wang", "Yuqing Chen", "Yushun Zhang", "Yutong Lin", "Yuwei Xiao", "Yuxian Gu", "Yuxiang Chen", "Yuxiang Huang", "Yuxiang Luo", "Yuxiang You", "Yuxin Chen", "Yuxin Xiang", "Yuxuan Liu", "Yuxuan Zhou", "Yuyang Zhou", "Yuzhe Guo", "Yuzhen Huang", "Yuzhuo Bai", "Z. Y. Z.", "Zanlin Ni", "Zehao Wang", "Zehua Zhao", "Zehui Ren", "Zejun Zhao", "Zhangli Sha", "Zhanying Wang", "Zhaochen Zhang", "Zhaoshuai Du", "Zhe Fu", "Zhean Xu", "Zhenda Xie", "Zheng Liu", "Zhengyan Zhang", "Zhenhua Dong", "Zhewen Hao", "Zhibang Wang", "Zhibin Gou", "Zhicheng Ma", "Zhihao Li", "Zhihong Shao", "Zhihuan Huang", "Zhijie Li", "Zhirui Lu", "Zhixian Huang", "Zhixuan Chen", "Zhixuan Chen", "Zhixuan Pan", "Zhiyu Wu", "Zhizhou Ren", "Zhu He", "Zhuoshu Li", "Zhuping Zhang", "Zian Xu", "Zihao Wang", "Zihui Gu", "Zijia Zhu", "Zili Zhang", "Zilin Li", "Zilong Hou", "Zilong Lyu", "Ziqiao Wang", "Ziwei Xie", "Ziya Zhang", "Ziyi Gao", "Zizheng Pan", "Zonglin Li", "Zongqing Yao", "Zui Chen", "Zuofan Wu", "Chenchen Ling", "Chengyu Hou", "Chong Chen", "D. Li", "Di Qi", "Dongjie Ji", "Fang Wei", "Fanyi Xia", "Fei Xie", "Feiyi Tan", "Hailong Guo", "Haiyan Zhai", "Hui Zhou", "Huihui Tan", "Huijie Li", "Jia Luo", "Jia Song", "Jialu Cai", "Jian Liang", "Jiangting Zhou", "Jiaqi Gao", "Jiayi Shao", "Jie Chen", "Jieyu Yang", "Jin Chen", "Jingde Zhang", "Jingzi Zhou", "Jinqian Wang", "Jinyang Liu", "JinZhao Sun", "Junhua Ling", "Junmin Zheng", "Kaicheng Yang", "Ke Xu", "Le Su", "Leyi Xia", "Liangfeng Ding", "Lin Zhuo", "Linwang Ma", "Linyan Zhu", "Liyu Cai", "Luqi Yao", "M. K. Zhang", "Meng Li", "Miao Lin", "Miaojun Wang", "Min Zhang", "Mingming Li", "Mingming Wang", "Mingze Yin", "Minmin Han", "Nan Cao", "Ning Wang", "Ningxin Ma", "Panpan Wang", "Peihan Lin", "Peng Sun", "Peng Zhang", "Qian Ying", "Qiang Xiang", "Qiao Wang", "Qingmiao Mao", "Qiwei Jiang", "Rongli Jin", "Ruyi Chen", "Sha Tao", "Shangmian Sun", "Shaoqing Wu", "Shichao Zou", "Si Lei", "Tianyang Zhang", "Tianyu Sun", "Tingting Yin", "W. L. Xiao", "Wei An", "Wei Li", "Wei Wang", "Weiwei Lin", "Wenqing Hou", "X. Lin", "Xiangfei Meng", "Xianzhu Huang", "Xiao Peng", "Xiaoqian Li", "Xiaoting Zhang", "Xiaowen Sun", "Xiaoxiang Wang", "Xiaoyu Ye", "Xinrou Zhang", "Xinyu Zhang", "Xue Cao", "Xueyin Chen", "Yanan Zhou", "Yanhong Xu", "Yao Xia", "Yao Xu", "Yi Shao", "Yihong Zhang", "Yiling Ma", "Ying Tang", "Yining Lou", "Yiru Chen", "Yishi Piao", "Yixuan Chen", "Yong Xiong", "Yuchen Xuan", "Yuehan Yang", "Yuer Xu", "Yukun Zha", "Yunxian Ma", "Yuping Lin", "Yuting Yan", "Yutong Xie", "Yuwen Sheng", "Yuxuan Zhu", "Zekai Zhang", "Zhe Ju", "Zhenzhen Lin", "Zheren Gao", "Zheyang Sun", "Zhigang Yan", "Zhongyu Wu", "Zi Wang", "Zihua Qu", "Ziling Yan", "Ziyi Wan"], "base_id": "2609.19969", "categories": ["cs.CL"], "id": "2609.19969v1", "pdf_url": "https://arxiv.org/pdf/2609.19969v1", "primary_category": "cs.CL", "published": "2026-09-17T09:43:10Z", "query_hits": ["q1_longctx_multimodal"], "seq": 59, "title": "DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression", "updated": "2026-09-17T09:43:10Z"}
|
|
{"abstract": "Wearable assistants require episodic memory over egocentric video, yet current vision-language models face bounded frame budgets, growing visual-token costs, and long-context retrieval failures. Under these practical constraints, we study whether textual captions can serve as reusable episodic memory. We define the Episodic Memory Video Caption QA task and introduce CapMem, a human-annotated benchmark with 75 videos totaling 33.7 hours, and 1,000 multiple-choice questions across 16 scenarios. On long videos (>20 min), full-coverage CaptionQA with 30s and 60s caption windows outperforms direct VideoQA for 10/12 and 8/12 models, respectively. On the same video subset, a matched-frame control across six Qwen models retains mean accuracy gains of 3.22 and 2.55 points, respectively. Our caption-guided retrieve-and-verify harness further improves accuracy by up to 5.3 points. These results support the effectiveness of caption memory for episodic reasoning over long egocentric video.", "authors": ["Dingli Liang", "Yiqiao Xie", "Yukai Huang", "Zhaokai Wang", "Weitong Cai", "Guangwen Feng", "Jifei Song", "Zhensong Zhang", "Hang Zhang"], "base_id": "2609.17688", "categories": ["cs.AI", "cs.CV"], "id": "2609.17688v1", "pdf_url": "https://arxiv.org/pdf/2609.17688v1", "primary_category": "cs.AI", "published": "2026-09-15T18:03:04Z", "query_hits": ["q3_longctx_cv"], "seq": 60, "title": "CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video", "updated": "2026-09-15T18:03:04Z"}
|
|
{"abstract": "Multimodal Large Language Models have demonstrated impressive video understanding, yet their ability to reason over long-form narratives is often masked by visual-centric evaluations and inefficient context processing. Existing benchmarks over-rely on visual heuristics while marginalizing auditory cues, effectively reducing models to \"silent observers\" that bypass genuine cross-modal reasoning. Moreover, standard dense sampling creates an evidence-context trade-off: increasing frames to capture evidence inevitably leads to attention distraction and token explosion. To bridge these gaps, we present Video-HolmesV2, a novel benchmark designed for Deep Audio-Visual Coupling. Unlike previous works, it enforces an Evidence-Based Evaluation, requiring models to justify answers with precise spatio-temporal audio-visual evidence, thereby reducing confounding effects of guessing and hallucinated evidence. To support this, we introduce: (1) a Multi-Model Cross-Verification pipeline to ensure task rigor; (2) a Spatio-temporal Evidence-Aware Metric for fine-grained calibration. Furthermore, we propose an Audio-Text Guided Token Compression framework. By fusing task intent with auditory anchors, our method distills high-value reasoning cues to mitigate long-context noise. In our evaluation, even strong proprietary models achieve below 60% accuracy, while our approach outperforms comparable open-source omni-models.", "authors": ["Zhaoyang Wei", "Zipeng Wang", "Yushe Cao", "Chenhui Qiang", "Shuaibing Cheng", "Xuesong Yang", "Sen Nie", "Bowen Jiang", "Wenchao Ding", "Yanchao Hao", "Zheng Wei", "Xuehui Yu", "Zhenjun Han"], "base_id": "2609.17248", "categories": ["cs.CV"], "id": "2609.17248v1", "pdf_url": "https://arxiv.org/pdf/2609.17248v1", "primary_category": "cs.CV", "published": "2026-09-15T14:30:00Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 61, "title": "Video-HolmesV2: Can MLLMs Reason with Spatio-Temporal Audio-Visual Evidence in Long Videos?", "updated": "2026-09-15T14:30:00Z"}
|
|
{"abstract": "Stable and reliable 4D spatial understanding is fundamental for autonomous driving systems. While feedforward reconstruction networks can estimate camera motion and 3D structure in one pass, pose estimation over long videos remains challenged by computational cost, long-context ambiguity, and temporal instability. To address these challenges, we propose Feedforward Visual Odometry (FFVO), a pose-specialized adaptation of joint reconstruction architectures for efficient and temporally stable camera-pose estimation. FFVO uses (i) a compact camera-token representation for computationally efficient temporal aggregation, (ii) a hierarchical local-to-global temporal decoder that mitigates geometric ambiguity by separating short-range motion aggregation from sequence-level integration, and (iii) intermediate trajectory supervision that promotes temporal stability. Extensive evaluation on the Waymo Open Dataset (WOD), KITTI, and a large-scale proprietary benchmark demonstrates that our method performs favorably against existing feedforward approaches, and greatly reduces jitter and drift. These results support FFVO as an effective feedforward camera-pose decoder in long-horizon visual odometry settings.", "authors": ["Meng-Li Shih", "Shih-Yang Su", "Yuliang Zou", "Hao Xiang", "Haidong Zhu", "Vincent Casser", "Brian Curless", "Dmitry Kalenichenko", "Mingxing Tan", "Dragomir Anguelov"], "base_id": "2609.13733", "categories": ["cs.CV"], "id": "2609.13733v1", "pdf_url": "https://arxiv.org/pdf/2609.13733v1", "primary_category": "cs.CV", "published": "2026-09-12T05:56:52Z", "query_hits": ["q3_longctx_cv"], "seq": 62, "title": "FFVO: A Feedforward Pose Decoder for Long-Horizon Visual Odometry", "updated": "2026-09-12T05:56:52Z"}
|
|
{"abstract": "Many emerging video language modeling tasks require systems to move beyond clip-level abstraction and model visual content as it unfolds over extended time horizons. However, most existing video datasets rely on coarse or sparsely aligned supervision, which compresses temporal variation and limits the ability of models to learn reusable representations of continuous visual dynamics. We introduce Kairos, a video dataset for video-language modeling with time-resolved annotations. Kairos consists of long-duration videos, ranging from ten minutes to half an hour, annotated with fine-grained temporal alignment. The annotations capture ongoing actions, entity appearances and attributes, interactions, and evolving contextual cues along the video timeline. This time-resolved structure supports fine-grained evaluation, long-range modeling and reasoning, instruction data construction, representation learning, and video generation. Kairos provides a general-purpose foundation for modeling visual experiences over time.", "authors": ["Ruibo Ming", "Lei Sun", "Deheng Zhang", "He Zhang", "Jialu Li", "Jian Wang", "Zhendong Li", "Mengshun Hu", "Danda Pani Paudel", "Luc Van Gool", "Jinjin Gu"], "base_id": "2609.08755", "categories": ["cs.CV", "cs.AI"], "id": "2609.08755v1", "pdf_url": "https://arxiv.org/pdf/2609.08755v1", "primary_category": "cs.CV", "published": "2026-09-08T13:50:30Z", "query_hits": ["q5_videolang_long"], "seq": 63, "title": "Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics", "updated": "2026-09-08T13:50:30Z"}
|
|
{"abstract": "Long-form instructional videos require automatic chaptering to support browsing, navigation, and knowledge access. Recent long-context language models can perform chaptering from textualized video inputs, but they remain costly and brittle for content-dense lecture videos with long transcripts, smooth topic transitions, and detailed chapter outputs. A scalable segment-then-caption paradigm reduces this cost, but introduces two new challenges: boundary error propagation and fragmented cross-chapter context. We propose \\textbf{CausalChapter}, an intervention-inspired framework for long-video chaptering that estimates prediction-level influence through lightweight masking and removal interventions. For boundary localization, our Local Dependency Shift module detects drops in predictive dependency between adjacent temporal windows; for chapter description generation, our Cross-Segment Support Selection module reranks historical contexts according to their support for the current prediction. Experiments on long-video chaptering benchmarks show that CausalChapter improves boundary localization, chapter description quality, and cross-chapter coherence.", "authors": ["Xinran Duan", "Guozhang Li", "Yaoyao Zhong", "Mei Wang", "Lizhi Wang", "Hua Huang"], "base_id": "2609.08686", "categories": ["cs.CV", "cs.AI"], "id": "2609.08686v1", "pdf_url": "https://arxiv.org/pdf/2609.08686v1", "primary_category": "cs.CV", "published": "2026-09-08T12:54:57Z", "query_hits": ["q3_longctx_cv"], "seq": 64, "title": "CausalChapter: Improving Long-Video Chaptering with Interventional Dependency Modeling", "updated": "2026-09-08T12:54:57Z"}
|
|
{"abstract": "Annual reports, diligence packs, and infographic dashboards bury numbers in page images: axes, cell grids, and footnotes that OCR pipelines flatten and that page-level visual retrievers still treat as interchangeable in-context examples. We keep a frozen Qwen2.5-VL-7B-Instruct generator and a ColPali / VisRAG-Ret page index, and insert three modules. A capability-aware visual router (CAVR) tags each retrieved page as text, table, chart, layout, or mixed and mixes specialist experts before generation. Weak-to-strong page selection (WSPS) distils a frozen 7B answerability teacher into a 3B selection head so ranking is no longer a single InfoNCE score. Visual evidence threading (VET) builds layout-anchored paths of length at most three and lets the generator read the thread rather than a flat top-$k$ list. On gold-page DocVQA / ChartQA / InfographicVQA the 7B system reaches 96.3 / 90.1 / 85.4. Under the VisRAG top-3 protocol the mean generation accuracy is 62.74 versus 59.39 for the same backbone with concatenation. On MMLongBench-Doc retrieve-then-read, F1 moves from 19.2 to 22.6 and multi-page accuracy from 16.4 to 21.2. ViDoRe nDCG@5 after WSPS reranking is 83.6, with TAT-DQA financial reports at 70.4.", "authors": ["Amirul Rahman", "Aisha Karim", "Kenji Nakamura", "Yi-Fan Ng"], "base_id": "2609.13268", "categories": ["cs.CV"], "id": "2609.13268v1", "pdf_url": "https://arxiv.org/pdf/2609.13268v1", "primary_category": "cs.CV", "published": "2026-09-07T16:56:47Z", "query_hits": ["q3_longctx_cv"], "seq": 65, "title": "Capability-Routed Visual Retrieval and Evidence Threading for Long-Context Document Question Answering", "updated": "2026-09-07T16:56:47Z"}
|
|
{"abstract": "Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \\textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \\textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\\%$ success in semantic trajectory retrieval and $48.7\\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\\%$ and $34.4\\%$). LTE achieves trajectory compression by factors of $8.7\\times$ to $26.1\\times$ with sub-second query latency on $24$\\,h video. On Ego4D natural-language queries, the system reaches $28.75\\%$ / $55.10\\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.", "authors": ["Tianyidan Xie", "Shenyi Wang", "Qiang Tang", "Mingjie Wang", "Zhicheng Qiu", "Xuanfu Li", "Zhan Xu", "Jian Yang", "Lanjun Wang", "Zili Yi"], "base_id": "2609.04802", "categories": ["cs.CV", "cs.AI"], "id": "2609.04802v3", "pdf_url": "https://arxiv.org/pdf/2609.04802v3", "primary_category": "cs.CV", "published": "2026-09-04T07:00:48Z", "query_hits": ["q5_videolang_long"], "seq": 66, "title": "Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents", "updated": "2026-09-29T08:04:58Z"}
|
|
{"abstract": "Diffusion language models (DLMs) offer a non-autoregressive alternative for mobile edge agentic artificial intelligence (AI) by refining tokens through iterative denoising rather than left-to-right decoding. Compared with autoregressive Transformer-based large language models (LLMs), DLMs can update multiple uncertain tokens in parallel and exploit bidirectional context throughout the generation process, enabling more flexible quality-latency trade-offs beyond fixed sequential decoding. These properties are particularly attractive for edge agents, where partial refinement, early exit, and constraint-guided correction can reduce response delay and communication overhead while improving robustness under noisy, incomplete, or dynamic contexts. This survey reviews DLM foundations and analyzes their suitability for edge settings under latency, memory, energy, bandwidth, privacy, and reliability constraints. We cover resource-efficient architectures, training and inference acceleration, compression, edge/cloud deployment, communication-aware serving, Internet of Things (IoT)/wireless applications, and evaluation of DLM-based agents. We further discuss open issues in long-context state management, split inference, trustworthy execution, multimodal grounding, and reproducible benchmarking. The goal is to connect DLM modeling properties, including bidirectionality, parallel refinement, controllability, and quality-latency elasticity, with system-level requirements of future mobile edge intelligence.", "authors": ["Chenqi Li", "Minghui Min", "Dusit Niyato", "Wei Ni"], "base_id": "2609.04778", "categories": ["cs.AI"], "id": "2609.04778v1", "pdf_url": "https://arxiv.org/pdf/2609.04778v1", "primary_category": "cs.AI", "published": "2026-09-04T06:16:49Z", "query_hits": ["q1_longctx_multimodal"], "seq": 67, "title": "Diffusion Language Models for Mobile Edge Agentic AI: Foundations, Applications, and Challenges", "updated": "2026-09-04T06:16:49Z"}
|
|
{"abstract": "Video-language benchmarks are usually constructed by the dataset authors without published reliability statistics, leaving the noise floor of the construct unknown. We argue that multimodal benchmarking benefits from methods taken from research communities that have already invested in strategies to ensure reliability. We illustrate the case with the Classroom Observation Protocol for Undergraduate STEM (COPUS): a 24-code multi-label observation instrument with a decade of peer-reviewed reliability literature. We recast COPUS as a video benchmark for multimodal foundation models, where it provides a dense set of structured labels (a 24-dimensional binary vector every 2 minutes across a 50-90 minute lecture), an externally validated vocabulary, and established literature that provides a per-code reliability target based on human evaluators. Annotations in our evaluation corpus are produced by a 5-person human-evaluator panel whose consensus matrix is our reference. We propose VISTA, a baseline that runs MiniCPM-V-4.5 over a dense sliding window, refines its per-window outputs with a lightweight multi-layer perceptron (MLP) head trained on top of the frozen backbone, and max-pools the resulting predictions onto the 2-minute COPUS grid. On three held-out chemistry lectures, VISTA reaches 80.1% restricted macro accuracy versus 74.9% for the zero-shot variant, with the largest residual errors on visually similar instructor codes and on rare audio-dependent codes. We characterize three systematic failure modes (audio-partial observability, fine-grained group-work discrimination, long-tail recall) and release the benchmark tooling, prompts and baseline code at https://github.com/ajfranck/VISTA.", "authors": ["Andrew Franck", "Brendan Ng", "Ben Fitzgerald", "Zane Derrod", "Chris Cianci", "Chris Craney"], "base_id": "2609.04550", "categories": ["cs.CV"], "id": "2609.04550v1", "pdf_url": "https://arxiv.org/pdf/2609.04550v1", "primary_category": "cs.CV", "published": "2026-09-03T23:13:56Z", "query_hits": ["q5_videolang_long"], "seq": 68, "title": "VISTA: Dense Multi-Label Classroom Coding with Vision-Language Models", "updated": "2026-09-03T23:13:56Z"}
|
|
{"abstract": "Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.", "authors": ["Prakhar Khatri"], "base_id": "2609.03820", "categories": ["cs.CV", "cs.CL"], "id": "2609.03820v1", "pdf_url": "https://arxiv.org/pdf/2609.03820v1", "primary_category": "cs.CV", "published": "2026-09-03T13:20:48Z", "query_hits": ["q5_videolang_long"], "seq": 69, "title": "Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs", "updated": "2026-09-03T13:20:48Z"}
|
|
{"abstract": "Large vision-language models have achieved remarkable success in vision-language tasks. However, they remain prone to Visual Hallucinations (VHs), undermining their reliability in real-world applications. Existing solutions typically require curated datasets, additional training, or multi-round decoding, resulting in considerable computational overhead. In this paper, we propose \\textbf{RVSD} (\\underline{R}etrieval \\underline{V}ision \\underline{S}parse \\underline{D}ecoding), a training-free and plug-and-play decoding framework that, for the first time, unifies token sparsification and \\textbf{Semantic-Space Visual Retrieval} (SSVR) within a single decoding pass. Within RVSD, we introduce a \\textbf{semantics-directed token selection} strategy that selectively sparsifies redundant tokens while preserving critical visual information. We further propose the SSVR mechanism, which reformulates visual compensation as an on-demand cross-modal retrieval process within a shared semantic space. Extensive experiments demonstrate that RVSD achieves state-of-the-art performance in mitigating VHs while maintaining robust suppression capabilities under long-context generation settings. Our code is available here.\\footnote{https://github.com/canjie-liu/RVSD}", "authors": ["Canjie Liu", "Jiawen Kang", "Jinbo Wen", "Zishao Zhong"], "base_id": "2609.02731", "categories": ["cs.CV", "cs.AI"], "id": "2609.02731v1", "pdf_url": "https://arxiv.org/pdf/2609.02731v1", "primary_category": "cs.CV", "published": "2026-09-02T15:40:40Z", "query_hits": ["q3_longctx_cv"], "seq": 70, "title": "RVSD: Retrieval Vision Sparse Decoding for Mitigating Visual Hallucinations in Large Vision-Language Models", "updated": "2026-09-02T15:40:40Z"}
|
|
{"abstract": "Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods. Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query. Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.", "authors": ["Shiyu Li", "Zi-Yuan Hu", "Shijia Huang", "Yanyang Li", "Yiwu Zhong", "Liwei Wang"], "base_id": "2609.01004", "categories": ["cs.CV", "cs.AI", "cs.CL", "cs.LG"], "id": "2609.01004v1", "pdf_url": "https://arxiv.org/pdf/2609.01004v1", "primary_category": "cs.CV", "published": "2026-09-01T09:52:10Z", "query_hits": ["q5_videolang_long"], "seq": 71, "title": "SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models", "updated": "2026-09-01T09:52:10Z"}
|
|
{"abstract": "Autoregressive large models have recently advanced Text-to-SVG generation from simple icons to complex, long-context graphics, yet standard autoregressive decoding often fails to maintain global consistency across geometry, layout, occlusion, and composition. We introduce CANVAS (Consistency-Aware Navigation via Visual Adaptive Sampling), a training-free, render-aware inference framework that combines power-sharpened trajectory likelihood with visual feedback from rendered futures and derives a stroke-wise navigation rule. It effectively estimates each candidate stroke's future value under a limited generation and rendering budget and adaptively allocates samples according to candidate uncertainty, decision influence, and rollout cost. Experiments across multiple autoregressive SVG backbones and complementary benchmarks demonstrate improvements in global consistency, which includes sound geometric relationships, spatial layouts, occlusion ordering, and overall composition, without additional training, demonstrating the effectiveness and generalization ability of our framework.", "authors": ["Yichen Wu", "Haoxuan Qu", "Yihang Lou", "Hossein Rahmani", "Jun Liu"], "base_id": "2608.30689", "categories": ["cs.CV"], "id": "2608.30689v1", "pdf_url": "https://arxiv.org/pdf/2608.30689v1", "primary_category": "cs.CV", "published": "2026-08-31T12:27:46Z", "query_hits": ["q3_longctx_cv"], "seq": 72, "title": "CANVAS: Consistency-Aware Navigation via Visual Adaptive Sampling for Long-Context Text-to-SVG Generation", "updated": "2026-08-31T12:27:46Z"}
|
|
{"abstract": "Autoregressive video diffusion enables scalable long-video generation by producing chunks from a bounded recent context. While recency-based caching preserves local continuity, it evicts historical cues needed when subjects, objects, scenes, or attributes reappear. Existing memory mechanisms expose models to nonlocal history, but access alone does not ensure effective use. Our analysis reveals that video DiT layers exhibit distinct preferences for current, recent, and distant context, suggesting that long-range memory requires deciding both what to retrieve and where to use it. We introduce LayerRecall, a current-conditioned, layer-selective memory router that retrieves relevant historical K/V states and injects them only into backbone-specific memory-sensitive layers while preserving local attention elsewhere. To reduce reliance on scarce high-quality long-horizon videos and explicit memory-allocation labels, we further propose Cross-Horizon Prediction Matching (CHPM), which uses a privileged long-context reference to supervise the bounded-memory router in prediction space. Across 100 multi-shot evaluation prompts, LayerRecall achieves the best overall results on MemoBench and MovieBench while matching its backbone on VBench-Long, demonstrating stronger long-range recovery without sacrificing local continuity. Qualitative analyses further reveal memory-guided self-correction, whereby initially mismatched local attributes return to their historical appearance without resetting ongoing motion or scene structure. Additional analyses show cross-backbone portability and negligible inference overhead.", "authors": ["Yixuan Ding", "Jiahao Kong", "Wei Huang", "Ruijie Quan", "Yi Yang"], "base_id": "2608.28460", "categories": ["cs.CV"], "id": "2608.28460v1", "pdf_url": "https://arxiv.org/pdf/2608.28460v1", "primary_category": "cs.CV", "published": "2026-08-28T15:49:13Z", "query_hits": ["q3_longctx_cv"], "seq": 73, "title": "LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation", "updated": "2026-08-28T15:49:13Z"}
|
|
{"abstract": "Scaling video generation to long durations reveals a critical bottleneck: current models lack robust long-term memory. This deficiency can be studied along two critical aspects: object permanence, the ability to precisely reproduce the appearance of objects upon re-entry; and memory capacity, the ability to process ultra-long context and use information from distant history. Robust long-term memory requires both: object permanence without sufficient context handling limits the temporal scope, while long context length without permanence fails to maintain identity. To address this, we present Ring Forcing, an autoregressive video diffusion framework designed to robustly construct and precisely utilize long-term memory. Our ring-structured training strategy enforces retrieval from distant history, effectively reconciling the trade-off between strict historical adherence and generative diversity. To expand memory capacity, we introduce a compression and timestep composition strategy. Under fixed sequence length constraints, this method extends the effective historical span to minutes-long durations and achieves a comprehensive receptive field over the entire history. Furthermore, we present a sparse RoPE mechanism to enable flexible, scalable memory adaptation while fully exploiting pre-trained priors. Extensive experiments demonstrate that Ring Forcing achieves superior minutes-long coherence and object permanence, significantly outperforming state-of-the-art methods.", "authors": ["Bowen Xue", "Brandon Y. Feng", "Chenguo Lin", "Yuchen Lin", "Yujia Zeng", "Lvmin Zhang", "Maneesh Agrawala", "Honglei Yan", "Panwang Pan"], "base_id": "2608.26794", "categories": ["cs.CV"], "id": "2608.26794v1", "pdf_url": "https://arxiv.org/pdf/2608.26794v1", "primary_category": "cs.CV", "published": "2026-08-27T08:25:32Z", "query_hits": ["q3_longctx_cv"], "seq": 74, "title": "Ring Forcing: Towards Precise Long-Term Memory for Autoregressive Video Diffusion", "updated": "2026-08-27T08:25:32Z"}
|
|
{"abstract": "Navigation World Models (NWMs) predict action-conditioned visual futures for planning. Two practical challenges are central to their design: selecting a suitable visual representation and efficiently modeling observation history for repeated candidate queries. Standard Global-Softmax attention provides flexible interactions but repeatedly processes the same history, leading to increasing computation and memory costs for long contexts and multi-query planning. We study both problems within a unified conditional flow-transformer framework. We first compare five frozen visual representations under the same dynamics model and evaluation. To reduce redundant history computation, we design Cached-Linear, a hybrid architecture that combines local and shifted-window attention for target mixing with linear attention for reusable history access. We further develop Balanced Gated Delta Network (GDN), which augments this design with frame-wise recurrent memory for temporal history modeling. Experiments on RECON, SACSoN, and SCAND show that representation choice depends on the prediction objective: PAE-L performs best for reconstruction, RAE-B for direct prediction, and V-JEPA for long-horizon rollout. Under shared-history workloads, Cached-Linear substantially reduces computation and memory compared with Global-Softmax, while Balanced GDN improves selected direct-prediction endpoints with efficient context reuse. Overall, we systematically study visual representation and history modeling for NWMs and develop hybrid reusable-history architectures for efficient long-context and multi-query prediction.", "authors": ["Guangfu Guo", "Xiaoqian Lu", "Rui Liu", "Yutong Chen", "Kunpeng Liu", "Long Cheng"], "base_id": "2609.29555", "categories": ["cs.CV", "cs.RO"], "id": "2609.29555v1", "pdf_url": "https://arxiv.org/pdf/2609.29555v1", "primary_category": "cs.CV", "published": "2026-08-26T06:17:52Z", "query_hits": ["q3_longctx_cv"], "seq": 75, "title": "Visual Representation and History Modeling for Navigation World Models", "updated": "2026-08-26T06:17:52Z"}
|
|
{"abstract": "While Vision Large Language Models (VLLMs) have achieved remarkable success in multimodal reasoning, their long-context inference remains prohibitively expensive due to the massive computation and memory overhead of visual Key-Value (KV) caches. Existing KV compression methods often apply uniform pruning across visual tokens and layers, leading to substantial information loss and degraded performance.To address this challenge, we propose \\textbf{VisCache}, a plug-and-play framework for coarse-to-fine \\textbf{Vis}ual KV \\textbf{Cache} pruning without training, which consists of two synergistic stages. First, a lightweight VLM filters temporal redundancy by selectively forwarding semantically informative keyframes. Second, we introduce {PruneKV}, a surgical KV compression algorithm tailored to the attention dynamics of VLLMs. Unlike rigid pruning strategies, PruneKV adopts a parabolic layer-wise budget allocation together with an asymmetric update mechanism that selectively prunes keys while fusing values, thereby preserving critical contextual information. Extensive experiments demonstrate that VisCache substantially improves inference efficiency, achieving up to {2.35$\\times$ speedup} and significant memory reduction while maintaining competitive performance with only {19--28\\%} KV cache retention. VisCache consistently outperforms existing baselines, establishing a new Pareto frontier between efficiency and performance for long-context VLLM inference. Code is available at https://github.com/Wlklk/VisCache", "authors": ["Lyuke Wang", "Zhuo Li", "Guangxu Zhu"], "base_id": "2608.24063", "categories": ["cs.CV", "cs.AI"], "id": "2608.24063v1", "pdf_url": "https://arxiv.org/pdf/2608.24063v1", "primary_category": "cs.CV", "published": "2026-08-25T04:52:05Z", "query_hits": ["q1_longctx_multimodal", "q3_longctx_cv"], "seq": 76, "title": "VisCache: Visual KV Cache Pruning for Efficient Vision Large Language Model Inference", "updated": "2026-08-25T04:52:05Z"}
|
|
{"abstract": "In domain-specific multimodal long documents, images and text jointly convey complex knowledge that cannot be fully captured by plain text alone. However, existing paradigms like DocVQA primarily focus on generating textual answers or localizing evidence regions, rather than outputting query-specific textual attribute values and corresponding images. To address this gap, we propose query-driven image-text joint extraction from long documents, requiring models to output query-requested textual attribute values and corresponding image bounding boxes. Based on challenges related to both user intent and document content, we designed a two-level taxonomy that operates at the query and instance levels. Further, we construct ITJoint, the first high-quality, manually annotated benchmark for this new task, comprising 2,455 pages of domain-specific documents with numerous non-decorative images, 316 queries, and 910 answer instances. Finally, we evaluate representative standalone Vision-Language Models from different providers and further design Q2IT, a multi-agent collaborative framework consisting of three progressively collaborating agents for evidence collection, page selection, and target-image localization. Using a joint evaluation approach that assesses both text extraction and image localization, our experiments show that standalone VLMs struggle with this task, while Q2IT significantly improves performance on ITJoint, although a substantial gap remains toward perfect results.", "authors": ["Yikai Gao", "Ding Xia", "Xi Yang"], "base_id": "2608.22214", "categories": ["cs.AI", "cs.MM"], "id": "2608.22214v1", "pdf_url": "https://arxiv.org/pdf/2608.22214v1", "primary_category": "cs.AI", "published": "2026-08-23T04:37:34Z", "query_hits": ["q4_long_document"], "seq": 77, "title": "Query-Driven Multimodal Information Extraction from Long Documents", "updated": "2026-08-23T04:37:34Z"}
|
|
{"abstract": "Streaming video understanding requires models to causally update state as video arrives and organize growing history into semantic units that can evolve, persist, and be recalled under bounded computation and memory. This challenge is pronounced in live soccer commentary, where a system must describe completed events, summarize recent play, recall earlier events, or remain silent using only information available before each utterance. We present StreamSoccer, an event-driven system that uses event memory as its intermediate representation. A fixed-budget active memory integrates the stream; completed event states are retained locally and consolidated into retrievable historical records. A unified generator uses current, recent, and historical context to produce three commentary modes, while a rule-assisted scheduler selects a mode or silence. Unlike streaming video-language models organized around frames, visual tokens, or caches, and soccer-commentary methods based on predefined clips or output timestamps, StreamSoccer explicitly models event lifecycles. We construct a three-track streaming soccer commentary dataset and a layered evaluation protocol. At common reference anchors, StreamSoccer obtains CIDEr scores of 38.62, 23.96, and 17.39 for current-event, recent-window, and historical-memory commentary, ranking first on the current-event and historical-memory tracks and second on recent-window. Controlled ablations show that local completed events improve all tracks and that the full system performs best on all three. Across 174 raw-video runs on 58 matches, per-minute RTF p95 ranges from 0.10 to 0.22 without sustained growth with match history. These results indicate that event memory supports streaming soccer commentary across temporal scopes while controlling long-history computation.", "authors": ["Chenxi Shao", "Bozhong Wang", "Jiaxin Huang", "Zhao Liu", "Sunwei Zhu", "Tianxin Hang", "Gaoqi He", "Yang Li", "Changbo Wang"], "base_id": "2608.19723", "categories": ["cs.CV", "cs.CL"], "id": "2608.19723v1", "pdf_url": "https://arxiv.org/pdf/2608.19723v1", "primary_category": "cs.CV", "published": "2026-08-20T07:19:29Z", "query_hits": ["q5_videolang_long"], "seq": 78, "title": "StreamSoccer: Event-Driven Memory for Streaming Soccer Commentary", "updated": "2026-08-20T07:19:29Z"}
|
|
{"abstract": "Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.", "authors": ["Muhammad Haseeb Aslam", "Alessandro Koerich", "Marco Pedersoli", "Ali Etemad", "Eric Granger"], "base_id": "2608.18386", "categories": ["cs.CV", "cs.AI"], "id": "2608.18386v1", "pdf_url": "https://arxiv.org/pdf/2608.18386v1", "primary_category": "cs.CV", "published": "2026-08-18T23:40:28Z", "query_hits": ["q5_videolang_long"], "seq": 79, "title": "TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs", "updated": "2026-08-18T23:40:28Z"}
|
|
{"abstract": "Whole-slide pathology reasoning requires models to integrate gigapixel-scale visual evidence across complete case-linked slides, yet current question-answering benchmarks primarily measure final answer accuracy--a metric vulnerable to linguistic priors and benchmark regularities, and insufficient to establish that predictions are grounded in the supplied tissue. We introduce PathoArgus-Bench, a benchmark and evaluation protocol that explicitly tests the full evidence chain: availability, accessibility, use, and responsiveness. PathoArgus-Bench comprises 22,078 four-choice questions from 4,913 patients across 15 TCGA projects, covering six pathology capabilities across three levels of evidence demand, and operates under a fixed reader budget that retains only a small fraction of the gigapixel context. To further isolate evidence-grounded reasoning, we contribute ESG (Evidence State Quartets), a controlled set of 483 quartets where the question text is fixed while the target WSI set is moved, replaced, or removed, requiring consistent predictions across all states. Evaluating 20 general-purpose, medical, and pathology-specific systems reveals a stark gap: while GPT-5.6 achieves 57.09% overall accuracy and 57.04% on ESG, it correctly completes only 19 of 483 quartets (3.93% QExact), exposing that row-level accuracy does not translate into reliable evidence grounding. We also introduce PathoArgus, a fixed-budget reader that allocates context via question relevance and spatial coverage, attaining 50.39% overall accuracy yet only 1.86% QExact--demonstrating that improved context access alone does not ensure consistent evidence-based prediction. Our benchmark and diagnostics establish that acquiring useful whole-slide context is necessary but far from sufficient, and call for a shift from answer-centric to evidence-grounded evaluation in computational pathology.", "authors": ["Bowen Liu", "Qixiang Zhang", "Xiaomeng Li"], "base_id": "2608.17607", "categories": ["cs.CV"], "id": "2608.17607v1", "pdf_url": "https://arxiv.org/pdf/2608.17607v1", "primary_category": "cs.CV", "published": "2026-08-18T10:16:33Z", "query_hits": ["q3_longctx_cv"], "seq": 80, "title": "PathoArgus: Advancing Evidence-Grounded Long-Context Visual Reasoning across Gigapixel Whole-Slide and Multi-Slide Case Contexts", "updated": "2026-08-18T10:16:33Z"}
|
|
{"abstract": "Long-context reasoning remains computationally expensive for large language models due to the quadratic complexity of attention over text tokens. Visual-text compression offers a promising alternative by rendering text into images and processing them with vision-language models, often reducing token usage. However, existing approaches apply uniform compression regardless of query relevance, potentially sacrificing precision where detailed extraction is required. We present SEER, a framework that learns to select query-relevant images through visual scanning and retrieve textual content only where needed, combining the efficiency of visual compression with the precision of text-based reasoning. Through supervised fine-tuning on tool-interaction trajectories, SEER learns adaptive tool invocation for selection and retrieval. Experiments on long-context benchmarks show that SEER improves extraction precision through selective text retrieval while retaining average prompt-token savings relative to full-text baselines. On LongBench, SEER achieves 51.11% average accuracy, outperforming the visual-text baseline Glyph-9B by 2.33 points and Qwen3-8B by 3.49 points. Code can be accessed at https://github.com/jiaweixu98/SEER", "authors": ["Jiawei Xu", "Zhilin Zhai", "Jinrui Fang", "Ruohan Xu", "Mingfei Lu", "Yi Zhang", "Guanchu Wang", "Tianlong Chen", "Ying Ding"], "base_id": "2608.15962", "categories": ["cs.CL", "cs.CV"], "id": "2608.15962v1", "pdf_url": "https://arxiv.org/pdf/2608.15962v1", "primary_category": "cs.CL", "published": "2026-08-16T23:30:41Z", "query_hits": ["q3_longctx_cv"], "seq": 81, "title": "SEER: Long-Context Reasoning via Selective Visual-Text Compression", "updated": "2026-08-16T23:30:41Z"}
|
|
{"abstract": "Multimodal retrieval-augmented generation (RAG) systems often rely on long unstructured contexts or aggressively expanded evidence graphs, which can introduce noisy evidence, weaken multi-hop reasoning, and increase unsupported generation. We present GraphLoom, a reliability-calibrated multimodal knowledge-graph RAG framework for compact and faithful evidence routing. Given a question and its associated multimodal input, GraphLoom constructs an instance-level multimodal knowledge graph from grounded scene descriptions, extracted relational triples, and external commonsense knowledge. Instead of injecting all retrieved evidence into the generator, GraphLoom performs reliability-aware subgraph retrieval with bounded expansion and selectively routes high-utility evidence through hierarchical graph memory slots and joint graph-sequence attention in a frozen language model. To improve robustness in complex reasoning settings, GraphLoom further combines interleaved retrieval with budgeted corrective retrieval, enabling adaptive multi-hop evidence refinement under noisy retrieval conditions. We evaluate GraphLoom on ScienceQA, MultiModalQA, and OK-VQA, including large distractor evidence pools that approximate noisy external knowledge retrieval. Experimental results show consistent gains in answer quality and evidence faithfulness over strong multimodal RAG, graph-retrieval, and open-source vision-language baselines, with improved retrieval quality on MultiModalQA and stable performance under noisy evidence pools. Additional analyses using MiniCheck-based verification, human evaluation, and latency profiling show that reliability-calibrated graph evidence routing provides an effective alternative to long-context multimodal evidence injection.", "authors": ["Zafar Ali", "Asad Khan", "Aalia Malik", "Pavlos Kefalas"], "base_id": "2608.15056", "categories": ["cs.AI"], "id": "2608.15056v1", "pdf_url": "https://arxiv.org/pdf/2608.15056v1", "primary_category": "cs.AI", "published": "2026-08-15T06:02:23Z", "query_hits": ["q1_longctx_multimodal"], "seq": 82, "title": "GraphLoom: Reliability-Calibrated Graph Evidence Routing for Multimodal KG-RAG", "updated": "2026-08-15T06:02:23Z"}
|
|
{"abstract": "Long-document visual question answering (VQA) over documents of tens to hundreds of pages mixing text, tables, charts, and figures typically follows retrieve-then-read pipelines. In our setting, the bottleneck shifts from retrieval recall to reranker-side evidence selection: on MMLongBench-Doc, BGE-M3 reaches Recall@20 = 0.86 but only F1@5 = 0.254, and even the visual retriever ColPali reaches only F1@5 = 0.332; a text-only rerank LLM seeing only raw snippets misses table, chart, and layout evidence even when the upstream retriever encoded images. We propose Trident, with two complementary components: Trident-R, a retriever-agnostic LLM reranker that converts each candidate into an LLM-readable semantic record, including a visual caption, section path, entity tags, multi-axis concept hits, and a text snippet, then performs a single adaptive-K rerank call; and Trident-S, a generation-side module that prompts the VLM under topical, entity, and structural lenses before synthesis. On two long-document datasets, the annotation+rerank protocol substantially improves retrieval F1 across five heterogeneous pools, with every reranked pool exceeding the strongest adaptive-K baseline PageIndex. An LLM rerank without the annotation barely changes first-hit ranking, indicating the lift comes from the structured annotation. Trident-S targets open-ended synthesis questions by design, adding up to 6.6 points in generation accuracy on these questions. The best Trident configuration is the strongest downstream QA pipeline in our evaluation, with rankings consistent across two LLM judges (kappa = 0.913).", "authors": ["Guanchen Wu", "Jiayuan Ding", "Subhabrata Mukherjee", "Carl Yang"], "base_id": "2608.14841", "categories": ["cs.AI"], "id": "2608.14841v1", "pdf_url": "https://arxiv.org/pdf/2608.14841v1", "primary_category": "cs.AI", "published": "2026-08-14T19:29:42Z", "query_hits": ["q4_long_document"], "seq": 83, "title": "What the Reranker Sees: Multi-Aspect Page Annotation for Long-Document Multimodal Question Answering", "updated": "2026-08-14T19:29:42Z"}
|
|
{"abstract": "Part-aware 3D object generation is essential for graphics applications such as controllable modeling, editing, and articulation, where objects are represented as coherent assemblies of semantic parts. However, existing part-aware generation methods, do not scale well to highly complex objects. As the number of parts increases, generating detailed geometry becomes prohibitively expensive in token length and memory. We introduce MegaParts, a scalable autoregressive 3D generation framework to address this challenge by combining structured sequence modeling with a token-efficient vector-quantized shape tokenizer. Our tokenizer learns discrete latent representations for part-level geometry by minimizing token usage subject to high-fidelity reconstruction, enabling adaptive-length tokenization based on geometric complexity. On top of this compact representation, we train a large language model to generate object bounding boxes, part bounding boxes, and part shape tokens within a unified structured sequence. Combined with efficient long-context training strategy, our token-efficient formulation scales to objects with up to 300 parts and sequence lengths up to 256k tokens. This substantially extends the scale of part-aware 3D generation while preserving compositional structure and enabling fine-grained part-level control. Our method achieves higher mesh quality than baseline autoregressive and diffusion models, showing that compressed discrete part tokens improve not only scalability but also the achievable fidelity of generated geometry. These results suggest that LLM native token-efficient autoregressive modeling is a compelling alternative to diffusion for large-scale part-aware 3D generation. The project page is available at https://expmaster.github.io/megaparts_webpage.", "authors": ["Manwen Liao", "Xinyu Lian", "Jian Mao", "Kaixu Chen", "Li Luo", "Jinghao Yan", "Wanshui Gan", "Qiao Yu", "Weitian Zhang", "Chunhua Shen", "Guang Chen", "Bo Dai", "Xudong Xu", "Zhaoyang Lyu"], "base_id": "2608.14783", "categories": ["cs.CV", "cs.GR"], "id": "2608.14783v1", "pdf_url": "https://arxiv.org/pdf/2608.14783v1", "primary_category": "cs.CV", "published": "2026-08-14T18:00:02Z", "query_hits": ["q3_longctx_cv"], "seq": 84, "title": "MegaParts: Scaling Part-Aware 3D Object Generation to 300 Parts via Token-Efficient Autoregressive Modeling", "updated": "2026-08-14T18:00:02Z"}
|
|
{"abstract": "Transformer-based language models achieve strong performance but incur substantial inference cost due to repeated high-dimensional matrix multiplications. We propose Reduced Matrix Multiplication (RMM), a training-free, input-adaptive inference method that reduces Transformer matrix products by selecting informative slices along their contraction dimensions, without modifying model weights. Under a simple retention-ratio control, RMM provides a smooth and predictable accuracy-efficiency trade-off. Across language models ranging from 1B to 70B parameters, we find that reduction tolerance depends on the model family, task, component, and retention ratio, although it often improves with model scale. Under moderate reduction, RMM remains robust across the evaluated discriminative, autoregressive generation, and long-context settings. We further show that the same principle extends to multimodal vision-language inference. Mechanistic ablations reveal a structural asymmetry within Transformers: attention-side computations are substantially more reducible than MLP components. Finally, wall-clock benchmarks with custom kernels on an NVIDIA A100 show that these computational savings can translate into practical runtime gains, especially at longer sequence lengths. Together, these results position RMM as a scalable direction for input-adaptive inference-time optimization.", "authors": ["Zixuan Lan", "Yanhong Li", "Jiawei Zhou"], "base_id": "2608.13426", "categories": ["cs.LG", "cs.AI", "cs.CL"], "id": "2608.13426v2", "pdf_url": "https://arxiv.org/pdf/2608.13426v2", "primary_category": "cs.LG", "published": "2026-08-13T16:16:04Z", "query_hits": ["q1_longctx_multimodal"], "seq": 85, "title": "Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inference", "updated": "2026-08-31T13:12:05Z"}
|
|
{"abstract": "Long-horizon egocentric memory transforms continuous first-person video and audio into a searchable record of past experiences. We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent. To address both bottlenecks, we introduce EgoCITE (Egocentric Context-augmented Indexing and Time-aware Evidence retrieval), a long-horizon agentic memory framework for egocentric QA. EgoCITE comprises three components. EgoScheme uses local multimodal context to turn fragmentary video captions and speech transcripts into self-contained atomic memory indices. EgoIndex organizes complementary action, activity, utterance, and conversation representations into searchable multi-view memory indices at multiple granularities. EgoRetrv combines semantic search with question-conditioned temporal relevance scoring and curation of retrieved evidence. We evaluate EgoCITE on EgoLifeQA, EgoMem, and EgoR1-Bench in terms of answer accuracy and target-event retrieval alignment. EgoCITE improves accuracy over agentic memory baselines by at least 4.4--14.2% while achieving 36$\\times$ lower cost than long-context LLM agents.", "authors": ["Le Zhang", "Hao Chen", "Vlad Roznyatovskiy", "Jianzhong Zhang", "Ke Sun"], "base_id": "2608.12627", "categories": ["cs.CV", "cs.AI", "cs.CL", "cs.HC"], "id": "2608.12627v3", "pdf_url": "https://arxiv.org/pdf/2608.12627v3", "primary_category": "cs.CV", "published": "2026-08-12T22:20:29Z", "query_hits": ["q1_longctx_multimodal"], "seq": 86, "title": "EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory", "updated": "2026-08-18T22:47:18Z"}
|
|
{"abstract": "Vision Language Models (VLMs) face significant challenges with ultra-long, interleaved image-text sequences due to the quadratic complexity of self-attention. Current solutions either resort to aggressive token pruning, risking irreversible information loss, or adopt efficient but less precise architectures, while largely ignoring the equally vital textual component. We introduce VLZip, a framework that unifies visual and textual compression for high-fidelity reasoning within a pure Transformer. At its core, VLZip hierarchically distills visual and textual segments into compact, layer-specific \"soft prefixes\" and injects them into each decoder layer's hidden states, drastically shortening the attention sequence while preserving fine-grained global context. To address deficient evaluations in the field, we also introduce LongVLBench, a new benchmark derived from video narratives that demands holistic, narrative-level reasoning. Extensive experiments show VLZip achieves leading performance on long-context multimodal reasoning, enabling training up to 120K tokens, a 6x increase over the baseline, and inference beyond 280K tokens with significantly reduced memory, while demonstrating the memory scalability to handle up to 2M tokens. By excelling at extreme context lengths where existing methods collapse, VLZip establishes an efficient and powerful new standard for long-context multimodal AI. Code is available at https://github.com/ShareLab-SII/VLZip.", "authors": ["Yuqi Zhang", "Cheng Chen", "Yuyu Guo", "Wenjie Yang", "Lingchen Meng", "Peng Di", "Hang Yu", "Zuxuan Wu", "Yu-Gang Jiang"], "base_id": "2608.08630", "categories": ["cs.CV"], "id": "2608.08630v1", "pdf_url": "https://arxiv.org/pdf/2608.08630v1", "primary_category": "cs.CV", "published": "2026-08-09T10:31:03Z", "query_hits": ["q1_longctx_multimodal"], "seq": 87, "title": "VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling", "updated": "2026-08-09T10:31:03Z"}
|
|
{"abstract": "Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.", "authors": ["Omar Coser", "Antonio Orvieto", "Paolo Soda", "Loredana Zollo"], "base_id": "2608.06122", "categories": ["cs.LG", "cs.AI"], "id": "2608.06122v2", "pdf_url": "https://arxiv.org/pdf/2608.06122v2", "primary_category": "cs.LG", "published": "2026-08-06T14:53:23Z", "query_hits": ["q1_longctx_multimodal"], "seq": 88, "title": "Is Self-Pretraining really useful to improve diagnosis in medical Time Series?", "updated": "2026-08-07T08:30:22Z"}
|
|
{"abstract": "Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The evaluation suite employs an ensemble of five state-of-the-art LLM-based embedding models to increase reliability and mitigate single-model bias, and applies two complementary methodologies: (i) coarse-grained semantic matching and (ii) fine-grained semantic matching to compare model-generated descriptions against CLIP-CC-Bench references. Using this framework, we evaluate 17 state-of-the-art video-language models and report both their Borda-aggregated rankings and their average scores on CLIP-CC-Bench. We further quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability. We release standardized evaluation scripts, model outputs, and aggregation tools at https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench to support reproducibility. CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks.", "authors": ["Mukhtiar Ali", "Harsh Dubey", "Sugam Mishra", "Chulwoo Pack"], "base_id": "2608.04302", "categories": ["cs.CV", "cs.IR", "cs.MM"], "id": "2608.04302v1", "pdf_url": "https://arxiv.org/pdf/2608.04302v1", "primary_category": "cs.CV", "published": "2026-08-05T00:20:23Z", "query_hits": ["q5_videolang_long"], "seq": 89, "title": "CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models", "updated": "2026-08-05T00:20:23Z"}
|
|
{"abstract": "Data agents enable natural-language analytics over organizational workspaces, where relevant evidence may be scattered across databases, structured files, long documents, and multimedia. Existing benchmarks largely isolate structured querying, retrieval, or open-ended analysis, leaving heterogeneous evidence discovery, complete tabular outputs, and deterministic evaluation insufficiently unified. We introduce DataSpace, a benchmark in which data agents produce verifiable tabular results from task-local heterogeneous workspaces. It contains 410 cross-language tasks and 7,439 artifacts totaling 15.01 GB across CSV, JSON, SQLite, Markdown, PDF, and video. DataSpace also served as the official evaluation benchmark for the KDD Cup 2026 Data Agents for Complex Data Analysis competition. Each agent receives only a question and workspace and returns the complete requested tabular result. We construct DataSpace with DataSpace-Builder, an execution-grounded framework comprising cross-language transformation, constraint-aware relational sampling, modality routing and artifact rendering, and human review and task repair by 11 domain experts. A deterministic evaluator performs header-invariant column alignment, type- and precision-aware normalization, and order-aware row comparison. Across six recently released frontier multimodal models and five widely used agent harnesses, the best accuracy reaches 66.34%, while harness choice creates a 15.36-point spread with the backbone fixed. Multimodal evidence integration and joins consistently reduce accuracy across all six backbones. These results show that DataSpace remains unsaturated and identify key challenges for improving data-agent reliability.", "authors": ["Boyan Li", "Zhuowen Liang", "Yupeng Xie", "Xiaotian Lin", "Tianqi Luo", "Xinyu Liu", "Yizhang Zhu", "Zhangyang Peng", "Yuan Li", "Zhengxuan Zhang", "Jiayi Zhang", "Nan Tang", "Guoliang Li", "Yuyu Luo"], "base_id": "2608.03451", "categories": ["cs.AI"], "id": "2608.03451v1", "pdf_url": "https://arxiv.org/pdf/2608.03451v1", "primary_category": "cs.AI", "published": "2026-08-04T10:48:11Z", "query_hits": ["q4_long_document"], "seq": 90, "title": "DataSpace: Benchmarking Data Agents for Verifiable Analytics over Heterogeneous Workspaces", "updated": "2026-08-04T10:48:11Z"}
|
|
{"abstract": "Long Document Visual Question Answering (LongDocVQA) requires Multimodal Large Language Models (MLLMs) to locate, integrate, and reason over heterogeneous document elements distributed across multiple pages. Existing approaches, including end-to-end MLLMs, retrieval-augmented generation (RAG) pipelines, and document agents, often lack explicit mechanisms to represent and verify how grounded evidence is progressively composed during reasoning, limiting both answer accuracy and traceability. In this paper, we cast LongDocVQA as an explicit evidence graph reasoning problem rather than implicit answer prediction. To this end, we propose DocTrace, a hierarchical framework that progressively performs evidence localization, structured document parsing, and evidence graph reasoning to enable explicit evidence provenance. To effectively learn these capabilities, we develop a two-stage training framework: joint Supervised Fine-Tuning (SFT) first initializes evidence localization and graph reasoning abilities, followed by task-specific Group Relative Policy Optimization (GRPO) with dedicated rewards to further optimize these capabilities. Extensive experiments on MMLongBench-Doc, LongDocURL, and SlideVQA demonstrate that DocTrace consistently outperforms both existing open-source baselines and proprietary MLLMs. Compared with the Qwen3-VL-8B-Instruct backbone, DocTrace achieves absolute improvements of 14.4, 11.3, and 11.7 points on the three benchmarks, respectively. Beyond competitive performance, DocTrace constructs traceable evidence graphs with explicit node-level provenance, enabling transparent and verifiable reasoning for long document understanding.", "authors": ["Le Xiang", "Zhicheng Guan", "Hong Chen", "Xiaocong Lin", "Zhenghua Lei", "Teng Hu", "Bolei He", "Long Zeng"], "base_id": "2608.03292", "categories": ["cs.AI"], "id": "2608.03292v1", "pdf_url": "https://arxiv.org/pdf/2608.03292v1", "primary_category": "cs.AI", "published": "2026-08-04T08:04:59Z", "query_hits": ["q4_long_document"], "seq": 91, "title": "DocTrace: Towards Traceable Long Document VQA via Hierarchical Evidence Graph Reasoning", "updated": "2026-08-04T08:04:59Z"}
|
|
{"abstract": "Recent visual-text compression (VTC) methods, typified by DeepSeek-OCR, report impressive high token compression ratios for long-context modeling tasks by leveraging text-to-image rendering. However, existing evaluation protocols heavily rely on downstream task performance. Such evaluation metrics fail to accurately measure text preservation due to the strong inherent linguistic priors of Multimodal Large Language Models (MLLMs). In this work, we introduce a new evaluation framework that decouples MLLMs' capabilities to faithfully assess VTC quality. Within this framework, we further introduce the ZeroSense Benchmark to ensure low semantic correlation of testing samples. By eliminating textual dependencies, our benchmark guarantees that the evaluation results are purely reflective of VTC quality, unaffected by the semantic inference capabilities of downstream models. Extensive experiments across multiple datasets demonstrate that VTC quality and downstream task accuracy diverge significantly, highlighting the necessity of our decoupled evaluation framework.", "authors": ["Yonghan Gao", "Zehong Chen", "Lijian Xu", "Jingzhi Chen", "Jingwei Guan", "Xingyu Zeng"], "base_id": "2608.01848", "categories": ["cs.CV"], "id": "2608.01848v1", "pdf_url": "https://arxiv.org/pdf/2608.01848v1", "primary_category": "cs.CV", "published": "2026-08-03T07:56:43Z", "query_hits": ["q1_longctx_multimodal"], "seq": 92, "title": "Decoupling semantics from vision: A framework for faithful visual-text compression evaluation", "updated": "2026-08-03T07:56:43Z"}
|
|
{"abstract": "Multimodal large language models (MLLMs) increasingly rely on long chain-of-thought reasoning for complex tasks. However, as reasoning sequences lengthen, models may gradually rely less on visual evidence and more on accumulated textual context, leading to visual forgetting. Existing approaches do not directly constrain how visual evidence is used and maintained along the original reasoning trajectory, leaving long-context visual forgetting insufficiently addressed. To address this issue, we propose Remember-R1, a reinforcement learning framework that mitigates long-context visual forgetting by applying process-level supervision directly on the original reasoning trajectory. Specifically, Remember-R1 introduces rewards that encourage broader coverage of matched visual keywords, stronger persistence of visual dependence in later reasoning steps, and greater focus on question-relevant image regions. Experiments across multiple model scales and diverse multimodal benchmarks demonstrate that Remember-R1 consistently improves reasoning performance. Additional analyses further show that it slows the decline of visual attention during generation, supporting its effectiveness in mitigating long-context visual forgetting.", "authors": ["Jianmin Chen", "Jiaqi Tang", "Wei Wei", "Xiaogang Xu", "Jiafei Wu", "Zhe Liu", "Qianzhou Wang", "Yingying Yan", "Botong Geng", "Yuyang Xia", "Lei Zhang", "Qifeng Chen"], "base_id": "2608.01314", "categories": ["cs.CV"], "id": "2608.01314v1", "pdf_url": "https://arxiv.org/pdf/2608.01314v1", "primary_category": "cs.CV", "published": "2026-08-02T15:31:31Z", "query_hits": ["q1_longctx_multimodal"], "seq": 93, "title": "Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning", "updated": "2026-08-02T15:31:31Z"}
|
|
{"abstract": "Vision-language-action models (VLAs), which leverage the cognition of multimodal information to infer physical-world actions, provide a generalized solution for embodied AI applications. Conventional VLAs usually concentrate on current digital cognition. While some efforts are made to enhance VLAs' reasoning capabilities by capturing temporal information, encoding the long-context history causes an efficiency-decreasing issue. To reconcile the conflict between capturing temporal information and maintaining inference efficiency in VLAs, this paper introduces FibVLA, an efficient framework featuring temporal perception of long-context history. Specifically, we leverage logarithmic hindsight sampling to both proprioceptive states and visual frames to capture long-term temporal dependencies with minimal redundancy. For the action expert, we introduce the flow matching to produce action distributions, and the Fibonacci recurrent inference strategy to generate long-range planning steps based on real-time closed-loop feedback. Experiments demonstrate that FibVLA significantly improves action smoothness and success rates without retraining large-scale visual encoders. Efficiency analysis demonstrates superior real-time responsiveness compared to video-based baselines in real-world evaluations.", "authors": ["Li Lin", "Wujun Xu", "Weiwei Meng", "Kaiwen Xia", "Kang Hao Cheong", "Shuai Wang"], "base_id": "2607.29596", "categories": ["cs.RO", "cs.CV"], "id": "2607.29596v1", "pdf_url": "https://arxiv.org/pdf/2607.29596v1", "primary_category": "cs.RO", "published": "2026-07-31T16:23:07Z", "query_hits": ["q1_longctx_multimodal"], "seq": 94, "title": "FibVLA: An Efficient Temporal Vision-Language-Action Model with Fibonacci Sampling", "updated": "2026-07-31T16:23:07Z"}
|
|
{"abstract": "We introduce DiffusionGemma, an experimental open-weight language model that uses discrete diffusion to generate text at exceptionally high speed. Rather than decoding one token at a time, DiffusionGemma iteratively refines blocks of 256 tokens in parallel, avoiding the sequential decoding bottleneck of conventional autoregressive (AR) large language models. Instead of training from scratch, we obtain DiffusionGemma by fine-tuning the mixture-of-experts Gemma 4 model with 3.8B activated and 25.2B total parameters. Our compute-efficient two-stage training pipeline uses fewer than 10% of the starting AR model's total training token budget. The first stage uses supervised fine-tuning to teach bidirectional denoising, while the second stage combines reinforcement learning with sampler distillation to jointly improve generation quality and inference efficiency. DiffusionGemma establishes a new Pareto frontier for the trade-off between generation speed and model capability. Averaged across our full evaluation suite, it generates around 20 tokens per forward pass and achieves roughly 1,500 output tokens per second on a single NVIDIA H100 GPU, which is substantially faster than AR models even with state-of-the-art speculative decoding. DiffusionGemma also retains the starting model's support for thinking mode, multimodal inputs, and long contexts. Despite diffusion fine-tuning, it remains capable of AR generation with only minor performance degradation, suggesting a path toward hybrid diffusion-AR decoding.", "authors": [" DiffusionGemma Team", "Adrien Ali Ta\u00efga", "James Assiene", "Daniele Calandriello", "Rahma Chaabouni", "Jo\u00e3o Gante", "Tamara von Glehn", "Nate Keating", "Chris Knutsen", "Martin Kukla", "Tianlin Liu", "Ivan Lobov", "Ofir Nabati", "Jo\u00e3o Gabriel Oliveira", "Nicolas Perez-Nieves", "Nastasia Prutianova", "Bobak Shahriari", "Jean Tarbouriech", "Pavel Tyletski", "\u00c7a\u011flar \u00dcnl\u00fc", "Cindy Wu", "Glenn Cameron", "Jerome Connor", "Sertan Girgin", "Maarten Grootendorst", "Alon Levkovitch", "Eliya Nachmani", "Omar Sanseviero", "Piotr Stanczyk", "Quentin Berthet", "Andrew Campbell", "Cl\u00e9ment Crepy", "Valentin De Bortoli", "Arnaud Doucet", "Romuald Elie", "Alexandre Galashov", "Klaus Greff", "Alexis Jacq", "David Ruhe", "Yu-Han Wu", "Sebastian Flennerhag", "Brendan O'Donoghue", "George Scrivener", "Shantanu Thakoor"], "base_id": "2608.00146", "categories": ["cs.CL", "cs.AI"], "id": "2608.00146v1", "pdf_url": "https://arxiv.org/pdf/2608.00146v1", "primary_category": "cs.CL", "published": "2026-07-31T16:11:46Z", "query_hits": ["q1_longctx_multimodal"], "seq": 95, "title": "DiffusionGemma Technical Report", "updated": "2026-07-31T16:11:46Z"}
|
|
{"abstract": "Visual generation increasingly requires high-resolution images, long videos, and multimodal context, making the quadratic cost of full attention prohibitive. We introduce Chimera, a hybrid visual diffusion backbone with a principled scaling recipe. Chimera processes text, image, and video tokens in one raster-ordered stream without positional embeddings. It combines Kimi Delta Attention (KDA) for long-context state tracking with O(N) complexity, interleaved Multi-head Latent Attention (MLA) for direct global interaction, and modality-aware short convolutions for local spatiotemporal context. Sparse Mixture-of-Experts (MoE) layers expand capacity while controlling activated compute. To scale this heterogeneous architecture, we introduce HeteroP, a module-wise scheme that transfers hyperparameters across width and depth according to each tensor's functional fan-in and model depth. HeteroP yields a consistently tuned family used to fit Chinchilla-style compute-optimal laws for activated model size, training-token count, and image-video data ratio. Guided by these laws, we train an 11B-parameter Chimera with 2B activated parameters. Experiments show three results. First, measured by pretraining diffusion loss, the dense backbone is 1.7x as compute-efficient as a matched full-attention Wan-2.1 2B baseline, while the complete system reaches 7.3x. Second, without length-specific fine-tuning, Chimera extrapolates zero-shot from 5-second training clips to 30-second videos, with only 6.5% FID degradation in the last five seconds. Third, the fitted laws show that compute-optimal image pretraining divides compute nearly evenly between activated model size and training-token count, whereas video pretraining modestly favors model size at higher budgets. These results establish a foundation for designing and scaling efficient long-context diffusion architectures.", "authors": ["Chongjian Ge", "Hanwen Jiang", "Tianyu Wang", "Jiuxiang Gu", "Yiran Xu", "Ziwen Chen", "Shaoteng Liu", "Jing Shi", "Yicong Hong", "Zefan Cai", "Hailin Jin", "Hao Tan"], "base_id": "2607.28611", "categories": ["cs.CV"], "id": "2607.28611v1", "pdf_url": "https://arxiv.org/pdf/2607.28611v1", "primary_category": "cs.CV", "published": "2026-07-30T17:58:02Z", "query_hits": ["q1_longctx_multimodal"], "seq": 96, "title": "Chimera: Designing and Chinchilla-Scaling Hybrid Visual Diffusion Transformers", "updated": "2026-07-30T17:58:02Z"}
|
|
{"abstract": "Multimodal long documents are core carriers of professional knowledge, where critical evidence is sparsely distributed across paragraphs and modalities. This easily causes key information omission and cross-modal hallucinations in summarization by multimodal LLMs. These issues stem from attention drift in long-range dependency modeling and gaps in inter-modal alignment. To address this, we introduce MMLDSum-Bench, a high-quality benchmark for multimodal long-document summarization, covering multiple domains, context-length scales, and visual-textual modality distributions. We further propose MMLDSum-LLM, a reproducible two-stage training framework that combines supervised fine-tuning with visual-alignment weighted loss and keyword-aware weighted loss, followed by GRPO with a multi-objective reward (keyword coverage, image-text alignment, ROUGE, and length control). Extensive experiments on MMLDSum-Bench, comparing against leading closed-source and open-source multimodal models under a unified evaluation protocol - including LLM-as-a-judge scoring, atomic-claim precision/recall, image-text alignment (ITA), and ROUGE - demonstrate that our approach significantly improves key-information coverage and cross-modal consistency.", "authors": ["Xianpeng Zhang", "Jiahua Yang", "Dongyu Chen", "Lei zhang", "Jian Ma", "Xu guohuan", "Haonan Lu", "Tianhuang Su", "Chuangchuang Wang", "Kai Tang"], "base_id": "2607.28006", "categories": ["cs.AI"], "id": "2607.28006v1", "pdf_url": "https://arxiv.org/pdf/2607.28006v1", "primary_category": "cs.AI", "published": "2026-07-30T10:54:09Z", "query_hits": ["q4_long_document"], "seq": 97, "title": "MMLDSum-LLM: Multimodal Long-Document Summarization with Visual-Alignment and Keyword-Aware", "updated": "2026-07-30T10:54:09Z"}
|
|
{"abstract": "Real-world tasks often require models to learn from task-specific context rather than relying only on pre-trained knowledge. While recent work has highlighted this capability as context learning, existing evaluations mainly focus on textual contexts. In many practical settings, however, the context to be learned from is multimodal: scientific findings are conveyed through figures and tables, financial indicators are scattered across converted reports, and spatial decisions depend on maps, scenes, or web pages. We introduce CLBench-V, a benchmark for multimodal context learning that addresses the difficulty of localizing where context use breaks down by organizing tasks around three dimensions: context grounding, new information application, and new knowledge learning. CLBench-V combines converted public benchmarks with newly constructed datasets spanning domains such as science, finance, long-document understanding, spatial reasoning, and web-based visual question answering. To reduce the cost of constructing domain-specific context-learning tasks, we further use automated construction and filtering procedures for our newly built datasets. Across 3,443 instances and six recent multimodal models, the best overall score is only 0.2847, indicating that multimodal context learning remains far from saturated. Moreover, InternVL3.5-30B-A3B performs best on context grounding and new knowledge learning, while Qwen3.5-Plus performs best on new information application. We further analyze judge reliability, context length, image count, and representative failure cases. Code is available at https://github.com/IamLihua/CLBench-V.", "authors": ["Lai Wei", "Chengqi Li", "Jiapeng Li", "Ruina Hu", "Yue Wang", "Weiran Huang"], "base_id": "2607.25294", "categories": ["cs.CV", "cs.AI", "cs.CL", "cs.LG"], "id": "2607.25294v1", "pdf_url": "https://arxiv.org/pdf/2607.25294v1", "primary_category": "cs.CV", "published": "2026-07-28T05:06:43Z", "query_hits": ["q4_long_document"], "seq": 98, "title": "CLBench-V: Evaluating Multimodal Context Learning from Grounding to Knowledge Acquisition", "updated": "2026-07-28T05:06:43Z"}
|
|
{"abstract": "Reliable visual document understanding requires a model to attribute each answer to the evidence regions that support it. Recent benchmarks and systems express this step through a coordinate interface: the model outputs the coordinates of bounding boxes that mark the evidence regions in the document. Under this interface, vision-language models often fail to identify the right regions even when the answer is correct, a failure known as Attribution Hallucination. We present a study that investigates whether this failure is partially limited by what the model can express through coordinates. On a verified bilingual CiteVQA subset, we compare the coordinate interface with a language interface in which the model outputs only text, quoting its evidence verbatim, and a multimodal retriever returns the location of each quote as a page region proposed by a layout parser (tables and figures are quoted through their captions or notes); the comparison is repeated over six open vision-language models. Compared with the coordinate interface, evidence recall rises from at most 8 points to between 26 and 47 and the hallucination rate roughly halves, with little change in answer quality. Building on this comparison, we use the same quote-and-retrieve pipeline as a training scaffold: because region-level evidence labels are expensive to collect for long documents, we introduce a GRPO recipe whose reward is a judge's reading of the gold answer and crops of the retrieved regions, training the model to quote better evidence without any region labels and raising an 8B backbone's strict attributed accuracy from 22.4 to 33.8. These findings indicate a practical path to improve attribution\"without a coordinate interface and without costly region-level supervision.", "authors": ["Zhuchenyang Liu", "Yao Zhang", "Yu Xiao"], "base_id": "2607.24651", "categories": ["cs.CV", "cs.CL", "cs.IR"], "id": "2607.24651v1", "pdf_url": "https://arxiv.org/pdf/2607.24651v1", "primary_category": "cs.CV", "published": "2026-07-27T16:49:36Z", "query_hits": ["q4_long_document"], "seq": 99, "title": "Evidence Attribution in Visual Document Understanding without Coordinates or Region Labels", "updated": "2026-07-27T16:49:36Z"}
|
|
{"abstract": "Modern LLM systems increasingly rely on knowledge-selection processes that produce high-value structured priors, such as ranked evidence, graph topology, multimodal alignment, and confidence signals. Yet LLM serving remains fundamentally oblivious to this rich structure: once such signals are serialized into a prompt, the backend observes only a flat token sequence, forcing dense and uniform consumption of the full key-value (KV) state during decoding. We term this architectural mismatch the Knowledge Selection-Runtime Consumption (KSRC) gap: richer contexts enlarge the full-prompt KV footprint and decode-time memory traffic, increasing latency and degrading throughput even when reasoning depends on only a small fraction of the context. To bridge the gap, we propose Knowledge Access Planning (KAP), a paradigm-shifting execution abstraction that elevates structured knowledge priors from passive prompt-construction hints into first-class physical execution artifacts. KAP establishes a universal intermediate representation (IR)-the runtime access plan-which compiles structured knowledge signals to govern physical KV access without altering logical prompt semantics, model weights, or training procedures. Through this IR, KAP shifts LLM serving from token-aware context consumption to plan-driven, knowledge-aware runtime consumption. We instantiate KAP with GraphSpec, a compiler-executor realization connecting structured knowledge selection to an LLM serving backend. We derive a phase-boundary model for the positive-speedup regime of plan-guided execution. Across 4K-128K long-context QA workloads, GraphSpec maintains answer quality comparable to full-context decoding while decoupling physical KV consumption from prompt length, reducing proposal-time KV access to 5.5% of source KV state at 128K, and fundamentally shifting the scaling trajectory of long-context generation.", "authors": ["Shuo Wang", "Fang Xi", "Wenyuan Huang", "Qing Wang", "Junming Su"], "base_id": "2607.24260", "categories": ["cs.LG"], "id": "2607.24260v1", "pdf_url": "https://arxiv.org/pdf/2607.24260v1", "primary_category": "cs.LG", "published": "2026-07-27T10:51:38Z", "query_hits": ["q1_longctx_multimodal"], "seq": 100, "title": "KAP: Bridging the Knowledge Selection-Runtime Consumption Gap in LLM Systems", "updated": "2026-07-27T10:51:38Z"}
|
|
{"abstract": "Electronic Theater Programs (ETPs) serve as critical promotional media in the performing arts, comprising a multi-page collection of heterogeneous visual assets such as theatrical posters, performance details, and character portraits. However, existing text-to-image paradigms struggle with such complex design tasks due to their inability to comprehend long-context narratives and maintain visual consistency across multiple distinct pages. To address this, we introduce ETPDesigner, a collaborative Multi-Agent framework that directly synthesizes high-quality ETPs from raw dramatic scripts. Emulating a professional design pipeline, our framework orchestrates specialized agents for semantic script analysis, core poster synthesis, functional background generation, and the stratified composition of character assets. Central to ETPDesigner is a global style anchor mechanism that extracts visual priors from the core poster to enforce strict aesthetic uniformity across all generated components. Furthermore, we elevate the ETP from a static publication to an immersive interactive companion. By integrating portrait animation, customized speech synthesis, and persona-grounded Large Language Models (LLMs), our system enables users to engage in real-time, voice-enabled conversations with the generated virtual characters. To rigorously benchmark this task, we construct ETP-Pro, a domain-specific benchmark of professional theater posters and high-quality character portraits. Extensive evaluations demonstrate our method's superiority in producing semantically faithful, aesthetically consistent, and highly interactive program sets.", "authors": ["Mengtian Li", "Xinru Guo", "Xiaoru Lin", "Xiao Rong", "Zhifeng Xie", "Chaofeng Chen"], "base_id": "2607.19947", "categories": ["cs.CV"], "id": "2607.19947v1", "pdf_url": "https://arxiv.org/pdf/2607.19947v1", "primary_category": "cs.CV", "published": "2026-07-22T09:25:03Z", "query_hits": ["q1_longctx_multimodal"], "seq": 101, "title": "ETPDesigner: Multi-Agent Orchestration for Interactive Multimodal Electronic Theater Program", "updated": "2026-07-22T09:25:03Z"}
|
|
{"abstract": "Long-document multimodal question answering requires more than retrieving relevant chunks from a large document. Different queries require different evidence behavior. Existing multimodal RAG systems improve evidence access through text chunks, page images, graph links, or heterogeneous document elements, but they often apply a largely query-agnostic evidence-use strategy. We present TAP-RAG, a task-aware policy-controlled RAG framework for long-document multimodal QA. TAP-RAG contains a main controller, the Task-Aware Policy Controller (TAPC), and two policy-guided evidence executors: Task-Aware Query-Guided Flow Diffusion (TA-QFD) and Task-Aware Visual Enhancement (TAVE). For each query, TAPC predicts the task prior, estimates visual/local/global evidence signals, and produces an executable policy. TA-QFD then expands textual and structural evidence over the multimodal document graph, while TAVE selectively inspects page images when visual or layout evidence is needed. A guarded synthesis stage fuses text, visual, and structural evidence and abstains when support is insufficient. On DocBench and MMLongBench-Doc, TAP-RAG achieves the best overall accuracy among the compared systems, improving over a matched multimodal-RAG baseline by +9.1 points (61.1 to 70.2) and +4.5 points (42.2 to 46.7), respectively.", "authors": ["Zhong Ji", "Keqi Jin", "Yan Zhang", "Jiasheng Li"], "base_id": "2607.18917", "categories": ["cs.CV"], "id": "2607.18917v1", "pdf_url": "https://arxiv.org/pdf/2607.18917v1", "primary_category": "cs.CV", "published": "2026-07-21T09:59:13Z", "query_hits": ["q4_long_document"], "seq": 102, "title": "TAP-RAG: Task-Aware Policy Control for Long-Document Multimodal Question Answering", "updated": "2026-07-21T09:59:13Z"}
|
|
{"abstract": "While Multimodal Large Language Models (MLLMs) excel in general tasks, rigorous scientific reasoning remains challenging due to the limitations of monolithic, linear planning. Such sequential designs often suffer from visual-semantic misalignment, long-context hallucinations, and brittle execution under fixed task granularity. We propose TopoAgent, a self-evolving topological framework that replaces linear trajectories with dynamic, state-isolated graph evolution. TopoAgent first employs a front-end decomposer to fracture complex queries into visually-grounded atoms. These atoms are organized into a Directed Acyclic Graph (DAG) based on their dependencies, enabling strict context isolation to shield the reasoning engine from irrelevant historical noise. Furthermore, we introduce adaptive atomic fission, which dynamically splits bottleneck nodes into finer-grained sub-atoms at runtime when tool capability boundaries are exceeded. Extensive experiments across mathematics, physics, and chemistry benchmarks demonstrate that TopoAgent significantly outperforms state-of-the-art linear agent frameworks, providing a robust, noise-resistant, and self-correcting paradigm for autonomous scientific reasoning.", "authors": ["Mingze Xu", "Yinghui Li", "Jiayi Kuang", "Zhanhui Kang", "Di Yin", "Ying Shen", "Xing Sun", "Yuxing Han"], "base_id": "2607.14658", "categories": ["cs.AI"], "id": "2607.14658v1", "pdf_url": "https://arxiv.org/pdf/2607.14658v1", "primary_category": "cs.AI", "published": "2026-07-16T07:22:59Z", "query_hits": ["q1_longctx_multimodal"], "seq": 103, "title": "TopoAgent: A Self-Evolving Topological Agent for Multimodal Scientific Reasoning", "updated": "2026-07-16T07:22:59Z"}
|
|
{"abstract": "A companion study ran a 35B mixture-of-experts model on a 2011 NVIDIA Tesla C2075 (Fermi, sm_20, 6GB) as a GPU-prefill/CPU-decode hybrid, because the 4-bit model did not fit in device memory (arXiv:2606.24031). This report keeps the hardware and asks what a model that fits can do: we deploy MiniCPM-V-4.6, a modern multimodal assistant pairing a SigLIP2 vision encoder and window-attention merger (16x visual token compression) with a compact hybrid gated-delta-net backbone, entirely on the GPU. Three results. (i) An all-GPU engine built on measured foundations: projections that dequantize 8-bit weights once and call the vendor SGEMM still in the last Fermi toolchain (64% of FP32 peak; our best hand-written GEMM hit 37%, wrongly called the ceiling); a chunked delta-rule rewrite of the recurrent layers, 2.8x faster than the sequential scan once attribution exposed one bad kernel; and a measured negative: 4-bit weights make decode slower than 8-bit here, since Fermi issues nibble-unpacking shifts at half rate. (ii) The vision side is a port with a proof obligation: we translate tower, merger, and projector to sm_20 CUDA, validating every stage against a locally generated reference forward (full tower 1.4e-5). One failure, position-embedding bucketization differing on exact rational ties, generalizes to a rule: float tie-breaking in index arithmetic is implementation-defined; call the reference operator, do not reimplement it. (iii) Long context exposes an O(N^2) wall short benchmarks hide: prefill falls from 114 tok/s at 2k tokens to 21 at 10k in a naive attention kernel; per-head vendor-GEMM calls writing into the existing score buffer (zero extra memory) restore a flat profile (408 at 2k, 361 at 10k; 17x), verified by exact needle retrieval from 60% depth. The same rewrite cuts image encoding 6x, to 0.93s. The system answers an image question end-to-end in 1.7s.", "authors": ["A. C. Opus", "J. Q. Lu"], "base_id": "2607.14568", "categories": ["cond-mat.other", "cs.AI"], "id": "2607.14568v1", "pdf_url": "https://arxiv.org/pdf/2607.14568v1", "primary_category": "cond-mat.other", "published": "2026-07-16T04:48:44Z", "query_hits": ["q1_longctx_multimodal"], "seq": 104, "title": "A Modern Multimodal Assistant on a 6 GB 2011 GPU: Stage-Validated, All-GPU CUDA Inference for Fermi", "updated": "2026-07-16T04:48:44Z"}
|
|
{"abstract": "Long-form audio description (AD) requires more than describing visible actions: it must preserve characters, events, relationships, and story context across scenes so that blind and low-vision (BLV) audiences can follow a film. Modern video-language models (VLMs) are effective on short clips, but they often treat each moment independently, producing descriptions that miss who characters are, why events matter, and how the current scene connects to earlier narrative context. We propose StoryTeller, a training-free framework for story-aware long-form AD. Instead of relying only on local visual cues, StoryTeller maintains a verified narrative memory that carries forward story-relevant information across scenes, enabling later descriptions to remain coherent, grounded, and contextually informative. Given only raw video and a movie title, StoryTeller can optionally retrieve public movie metadata to resolve names and story context, while accepting only facts that are supported by the video through semantic filtering and VLM verification. The method requires no subtitles, scripts, AD transcripts, aligned captions, character banks, precomputed face identities, or task-specific fine-tuning. To evaluate whether generated AD preserves narrative information, we introduce StoryAD-QA, a question-answering benchmark that tests whether a language model can answer story-context questions using only the generated descriptions. Experiments on standard AD benchmarks and diverse long-form videos show that StoryTeller consistently improves narrative coherence, factual grounding, and story comprehension over strong baselines in automatic, QA-based, and human evaluations.", "authors": ["Seung Hyun Hahm", "Minh T. Dinh", "SouYoung Jin"], "base_id": "2607.11798", "categories": ["cs.CV", "cs.AI"], "id": "2607.11798v1", "pdf_url": "https://arxiv.org/pdf/2607.11798v1", "primary_category": "cs.CV", "published": "2026-07-13T16:50:03Z", "query_hits": ["q5_videolang_long"], "seq": 105, "title": "StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description", "updated": "2026-07-13T16:50:03Z"}
|
|
{"abstract": "Retrieval-Augmented Generation (RAG) streamlines long-document understanding by leveraging retrieval mechanisms to restrict input images to a highly curated subset. However, existing multimodal RAG pipelines primarily face two critical challenges: first, standard semantic similarity retrievers frequently fetch topically overlapping yet answer-void distractor pages that mislead downstream generation; second, rigid single-pass pipelines heavily depend on initial retrieval success, where any omission of core evidence inevitably causes cascading errors. To address these challenges, we introduce HIEVI-RAG, a hierarchical, evidence-driven multimodal RAG framework for closed-domain document understanding. HIEVI-RAG systematically factorizes complex queries into a cooperative four-stage pipeline: (1) hierarchical question decomposition to break multi-hop root queries into atomic child questions; (2) coarse visual page retrieval leveraging a multimodal retriever to fetch candidate pages based on semantic similarity; (3) fine-grained page verification via EVIAGENT, a specialized multi-page verifier trained with GRPO to execute cross-page reasoning over multi-image blocks; and (4) memory-guided iterative generation that leverages accumulated sub-question context to execute multi-round, dynamic reasoning over the prioritized sequence. Extensive evaluations across four benchmarks demonstrate the robust efficacy and synergy of our framework, which significantly outperforms existing open-source baselines and exceeds the strongest reported baseline by an average of 8.05% in accuracy.", "authors": ["Junyu Xiong", "Yonghui Wang", "Rongjian Gu", "Chenyu Liu", "Bing Yin", "Wengang Zhou", "Houqiang Li"], "base_id": "2607.04625", "categories": ["cs.CV", "cs.AI"], "id": "2607.04625v1", "pdf_url": "https://arxiv.org/pdf/2607.04625v1", "primary_category": "cs.CV", "published": "2026-07-06T03:08:28Z", "query_hits": ["q4_long_document"], "seq": 106, "title": "Hierarchical Evidence-Driven Reasoning for Long Document Understanding", "updated": "2026-07-06T03:08:28Z"}
|
|
{"abstract": "As grounded QA systems are increasingly deployed in AI assistants, accurately attributing generated answers to evidence is critical for user trust and model safety. While unimodal attributions have been explored in depth, the multimodal setting remains relatively under-researched. As a result, we introduce MultAttnAttrib, a training-free attribution-generation method that leverages a model's prefill pass, selected attention heads, and calibrated thresholds to locate source evidence within a document. To establish baseline results for the method, we introduce MultAttrEval, a complementary benchmark dataset annotated with fine-grained, ground-truth attributions for answer components grounded in multimodal source documents. To our knowledge, this is the first evaluation dataset designed specifically for multimodal attribution in long-form documents. Experimental results show that MultAttnAttrib consistently outperforms a variety of attribution-generation methods, including several strong prompting-based approaches and matches the latest frontier models such as GPT 5.4. Our method not only substantially improves attribution accuracy for both unimodal and multimodal attribution types, but also produces attributions at up to one-seventh of the direct inference latency compared to prompting on the same base model.", "authors": ["Dang Quang Thien Tran", "Quang V. Dang", "Vinamra Tyagi", "Sai Soorya Rao Veeravalli", "Trang Nguyen", "Ryan A. Rossi", "Franck Dernoncourt", "Nedim Lipka", "Koustava Goswami", "Samyadeep Basu"], "base_id": "2607.01420", "categories": ["cs.CL", "cs.AI", "cs.CV"], "id": "2607.01420v3", "pdf_url": "https://arxiv.org/pdf/2607.01420v3", "primary_category": "cs.CL", "published": "2026-07-01T19:29:19Z", "query_hits": ["q4_long_document"], "seq": 107, "title": "MultAttnAttrib: Training-Free Multimodal Attribution in Long Document Question Answering", "updated": "2026-07-08T16:20:17Z"}
|
|
{"abstract": "Surgical vision-language foundation models typically adopt educational materials, such as surgical lecture videos, to transfer surgical knowledge encoded in language into visual representations. These knowledge are multi-dimensional and hierarchical: fine-grained action cues appear in narration, mid-level key steps are summarized in subsection headings, and global procedural context, such as patient history and surgical strategy, is described in abstract texts. Prior work largely collapses these heterogeneous signals into a single flat embedding space, implicitly assuming independence across hierarchy levels. However, this is suboptimal because it ignores cross-level semantic containment, e.g., actions belong to steps, steps compose phases, weakens long-range dependency modeling. To this end, we propose a hyperbolic surgical video-language pre-training framework that explicitly preserves the hierarchical structure by mitigating structural false negatives induced by procedural context and enforcing semantic consistency between parent phases and their constituent child steps. Extensive experiments on multiple surgical benchmarks show consistent gains in zero- and few-shot phase recognition across procedures and institutions.", "authors": ["Yaojun Hu", "Kun Yuan", "Nassir Navab", "Haochao Ying", "Jian Wu", "Nicolas Padoy"], "base_id": "2606.31245", "categories": ["cs.CV"], "id": "2606.31245v1", "pdf_url": "https://arxiv.org/pdf/2606.31245v1", "primary_category": "cs.CV", "published": "2026-06-30T07:21:06Z", "query_hits": ["q5_videolang_long"], "seq": 108, "title": "HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space", "updated": "2026-06-30T07:21:06Z"}
|
|
{"abstract": "Multimodal large language models (MLLMs) are widely applied to visual document understanding. However, comprehending long documents remains an issue by the limited context window. Though recent multimodal retrieval-augmented generation (MMRAG) can address this challenge by retrieving relevant pages. It still struggles with the visual question answering (VQA) requiring holistic comprehension of a document. To cope with this, knowledge graph (KG) that summarizes global knowledge of a document can provide an effective solution. However, most existing LLM-based KG construction methods handle only the language modality, leaving the automatic creation of multimodal KGs (MMKGs) for visually rich documents largely unexplored. In this paper, we introduce a multimodal graph-based RAG approach to tackle this problem. Existing LLM-based KG methods evaluate the QA performance relying on indirect evidence such as comprehensiveness, diversity, empowerment, and so on. The lack of annotated datasets for comprehensive document-level VQA poses a significant challenge to effective model evaluation. To overcome this limitation, we also introduce a new benchmark, DLVQA (document-level VQA), which provides reference summaries and corresponding supporting facts for global document-level questions. Experimental results show that our approach outperforms existing MMRAG or KG-based approaches on multi-hop QA/VQA benchmarks and DLVQA.", "authors": ["Yi-Cheng Wang", "Chu-Song Chen"], "base_id": "2606.28780", "categories": ["cs.IR", "cs.CV"], "id": "2606.28780v1", "pdf_url": "https://arxiv.org/pdf/2606.28780v1", "primary_category": "cs.IR", "published": "2026-06-27T07:14:31Z", "query_hits": ["q4_long_document"], "seq": 109, "title": "Multimodal Graph RAG for Long-range Visually Rich Document Understanding", "updated": "2026-06-27T07:14:31Z"}
|
|
{"abstract": "Despite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory. These obstacles undermine real-time responsiveness and cause severe forgetting throughout prolonged interactions. In this work, we introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams. The design of LiveStarPro rests on three complementary components. The first component is Streaming Verification Decoding (SVeD), an inference framework that identifies the appropriate response timing through single-pass perplexity verification, thereby eliminating the dependency on explicit silence tokens. The second component is Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams. The third component is Tree-Structured Hierarchical Memory (TSHM), a recursive memory architecture that organizes evicted historical information into event chains and consequently enables efficient retrieval from effectively unbounded video streams. To facilitate a comprehensive evaluation under realistic online conditions, we further present OmniStarPro, a large-scale benchmark that spans 15 diverse real-world scenarios and that extends to hour-scale streams for the assessment of long-term recall. Extensive experiments demonstrate that LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error, while its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching. The model and the code are publicly available at https://github.com/sotayang/LiveStarPro.", "authors": ["Zhenyu Yang", "Kairui Zhang", "Bing Wang", "Shengsheng Qian", "Changsheng Xu"], "base_id": "2606.17798", "categories": ["cs.CV", "cs.AI"], "id": "2606.17798v1", "pdf_url": "https://arxiv.org/pdf/2606.17798v1", "primary_category": "cs.CV", "published": "2026-06-16T11:18:05Z", "query_hits": ["q5_videolang_long"], "seq": 110, "title": "LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams", "updated": "2026-06-16T11:18:05Z"}
|
|
{"abstract": "Long-document multimodal question answering requires a system to locate sparse evidence in long PDFs and integrate clues from text, tables, images, charts, and complex layouts. Existing RAG methods mostly rely on fixed Top-k retrieval over text chunks or pages. Text retrieval can compress the context but often loses visual and layout information; page-level visual retrieval preserves the original page, yet it also sends large irrelevant regions to the reader, leading to a static trade-off among evidence coverage, noise, and inference cost. This paper proposes MAGE-RAG, a multigranular adaptive graph evidence framework for long-document multimodal QA. MAGE-RAG uses page retrieval as the entry point for query-time evidence construction. Offline, it builds an evidence graph with page nodes and element nodes, encoding containment, reading order, layout adjacency, section hierarchy, and semantic-neighbor relations. At query time, an online evidence controller iteratively activates, opens, searches, and prunes evidence under explicit budgets. The resulting evidence subgraph is then rendered into structured multimodal reader input, allowing the LVLM to consume compact and relevant evidence within a limited context. On LongDocURL and MMLongBench-Doc, we establish a unified comparison and analysis protocol covering Direct MLLM, Text RAG, Page-level Visual RAG, and Graph/Agentic RAG. Experiments show that MAGE-RAG achieves 52.75 overall accuracy on LongDocURL, and 53.26 accuracy with 51.19 F1 on MMLongBench-Doc. Fine-grained breakdowns, budget-performance curves, ablations, and trace-based analysis further show that query-time evidence subgraph construction can balance dispersed evidence coverage with context-noise control. Our code is available at https://github.com/laonuo2004/MAGE-RAG.git.", "authors": ["Yilong Zuo", "Xunkai Li", "Jing Yuan", "Qiangqiang Dai", "Hongchao Qin", "Ronghua Li"], "base_id": "2606.15906", "categories": ["cs.IR", "cs.AI", "cs.CL", "cs.DB", "cs.MM"], "id": "2606.15906v1", "pdf_url": "https://arxiv.org/pdf/2606.15906v1", "primary_category": "cs.IR", "published": "2026-06-14T16:34:15Z", "query_hits": ["q4_long_document"], "seq": 111, "title": "MAGE-RAG: Multigranular Adaptive Graph Evidence for Agentic Multimodal RAG in Long-Document QA", "updated": "2026-06-14T16:34:15Z"}
|
|
{"abstract": "Retrieving evidence pages from visually rich long documents is a key challenge in document question answering. Existing page-level visual retrievers operate under an independent matching paradigm: each page is scored in isolation based on query-page similarity. This paradigm can under-rank evidence pages whose signals are localized in fine-grained chunks or depend on document-internal associations. We propose EviProp, a retrieval method that recovers such pages via seeded relevance diffusion. EviProp models each document as a multimodal Chunk-Page graph with hierarchical, sequential, and similarity links. Given a query, it combines dense visual page priors with sparse chunk seeds, then runs Personalized PageRank to diffuse relevance over the graph. Experiments on MMLongBench-Doc and LongDocURL show consistent gains in evidence-page retrieval over independent visual retrieval and text-visual fusion baselines. Downstream QA results further show that improved retrieval translates into better answer accuracy, with negligible online retrieval overhead. Our code is released at https://github.com/Flyecnu/EviProp.", "authors": ["Hongwei Zhang", "Xiaoman Wang", "Zehui Ling", "Ruicheng Zhu", "Yue Zhang", "Pinlong Cai", "Fuke Shen", "Botian Shi", "Tongquan Wei", "Guohang Yan"], "base_id": "2606.08979", "categories": ["cs.IR"], "id": "2606.08979v1", "pdf_url": "https://arxiv.org/pdf/2606.08979v1", "primary_category": "cs.IR", "published": "2026-06-08T03:25:20Z", "query_hits": ["q4_long_document"], "seq": 112, "title": "EviProp: Seeded Relevance Diffusion on Chunk-Page Graphs for Long Multimodal Document Retrieval", "updated": "2026-06-08T03:25:20Z"}
|
|
{"abstract": "Iterative retrieval-reasoning agents have recently shown promise for multimodal long-document question answering. However, most existing systems maintain a single growing context that mixes retrieval traces, observations, and intermediate reasoning. As interactions accumulate, key evidence becomes scattered and diluted, making multi-hop reasoning noisy. We propose MARDoc, a Memory-Aware Refinement Agent framework that decouples long-document QA into three specialized agents: an Explorer for multi-granularity multimodal retrieval, a Refiner for distilling interaction traces into structured evidence and reasoning memories, and a Reflector for checking evidence sufficiency and providing targeted feedback. Across iterations, the agents rely on a dynamically updated structured memory rather than a full accumulated interaction history. This design reduces context noise while preserving answer-critical facts and their logical dependencies. Experiments on MMLongBench-Doc and DocBench show that MARDoc achieves strong results, outperforming same-backbone baselines and demonstrating the effectiveness of structured memory for agentic document QA.", "authors": ["Kaifeng Chen", "Hongtao Liu", "Qiyao Peng", "Jian Yang", "Yongqiang Liu", "Xiaochen Zhang", "Qing Yang"], "base_id": "2606.05749", "categories": ["cs.CL", "cs.AI"], "id": "2606.05749v1", "pdf_url": "https://arxiv.org/pdf/2606.05749v1", "primary_category": "cs.CL", "published": "2026-06-04T06:23:01Z", "query_hits": ["q4_long_document"], "seq": 113, "title": "MARDoc: A Memory-Aware Refinement Agent Framework for Multimodal Long Document QA", "updated": "2026-06-04T06:23:01Z"}
|
|
{"abstract": "Retrieval over visually-rich documents, pages that interleave text with figures, tables, and charts, is essential for multimodal retrieval-augmented generation, yet most retrievers still discard the visual channel. The \\emph{Multimodal Document Retrieval Challenge}, Track~1 of the MIR Challenge at the first EReL@MIR workshop, co-located with The Web Conference 2025, asks participants to build a \\emph{single} retrieval system that handles two complementary regimes: closed-set document page retrieval within long documents from a text query (MMDocIR), and open-domain retrieval of Wikipedia-style passages from an image or image-plus-text query (M2KR). Systems are ranked by the macro-average of mean Recall@$\\{1,3,5\\}$ over the two tasks. The challenge drew 455 entrants and 586 submissions across 22 teams. This report describes the challenge design, datasets, and evaluation protocol; reports the final standings; and analyses the three winning teams' systems. All three build on decoder-based Multimodal-LLM embedders from the Qwen2-VL family rather than on CLIP-style encoders, and differ chiefly in whether they reach the top through fine-tuned ensembles, training-free multi-route fusion with a strong vision-language re-ranker, or zero-shot late interaction. The training-free system finished within $0.1$ point of the fine-tuned winner.", "authors": ["Jingbiao Mei"], "base_id": "2606.04240", "categories": ["cs.CV", "cs.AI", "cs.CL"], "id": "2606.04240v1", "pdf_url": "https://arxiv.org/pdf/2606.04240v1", "primary_category": "cs.CV", "published": "2026-06-02T21:39:32Z", "query_hits": ["q4_long_document"], "seq": 114, "title": "Overview of the EReL@MIR 2025 Multimodal Document Retrieval Challenge (Track 1)", "updated": "2026-06-02T21:39:32Z"}
|
|
{"abstract": "Accurately predicting future events is fundamental to content understanding and decision-making across various domains. While prior research has primarily focused on text or short-video scenarios, long-video event prediction, characterized by vast multimodal context and more complex narratives, remains underexplored. Meanwhile, although recent Long-Video Language Models (LVLMs), built on Large Language Models (LLMs) and Vision-Language Models (VLMs), have shown promise in long-video question answering and summarization, they struggle to generalize to event prediction, as they can neither precisely extract event-related details nor perform fine-grained analysis of event development. To address this gap, we propose VISTA, a multi-level event semantics mining framework for long-video event prediction. Initially, VISTA applies a character-centric visual prompt to precisely extract event-related visual details, enhancing detail-level semantics; subsequently, it employs a knowledge-enhanced iterative retrieval strategy, guiding the LLM to progressively construct logically coherent event chains, thereby improving event-level narratives; ultimately, VISTA adopts a human-like propose-then-retrieve strategy to generate diverse future-oriented proposals and integrate multi-level clues, producing robust and accurate predictions. Extensive experiments on real-world datasets validate the effectiveness of VISTA for long-video event prediction.", "authors": ["Bo Peng", "YuanJie Lyu", "PengGang Qin", "Tong Xu"], "base_id": "2605.31069", "categories": ["cs.CV", "cs.CL"], "id": "2605.31069v1", "pdf_url": "https://arxiv.org/pdf/2605.31069v1", "primary_category": "cs.CV", "published": "2026-05-29T09:38:43Z", "query_hits": ["q5_videolang_long"], "seq": 115, "title": "Towards Effective Long-Video Event Prediction via Multi-Level Event Semantics Mining", "updated": "2026-05-29T09:38:43Z"}
|
|
{"abstract": "In the era of Large Video-Language Models (LVLMs), the computational necessity of sparse frame sampling creates a fundamental ``temporal gap'', rendering models blind to critical causal transitions. Existing solutions relying on generative hallucination (e.g., latent diffusion) or autoregressive extrapolation often fail to maintain semantic consistency over long horizons, suffering from object vanishing and energetic instability. We propose a paradigm shift from probabilistic generation to variational mechanics with the \\textbf{Semantic Least Action Principle (SLAP)}. Drawing a rigorous isomorphism between classical mechanics and semantic dynamics, we model the latent video trajectory as a path on a Riemannian manifold governed by a Semantic Lagrangian. By formulating the interpolation task as a Boundary Value Problem (BVP) solved via the discrete Euler-Lagrange equations, SLAP naturally enforces object persistence without pixel-level rendering. Extensive experiments show the effectiveness of our proposed SLAP.", "authors": ["Xiang Fang", "Wanlong Fang"], "base_id": "2605.30750", "categories": ["cs.CV"], "id": "2605.30750v1", "pdf_url": "https://arxiv.org/pdf/2605.30750v1", "primary_category": "cs.CV", "published": "2026-05-29T02:28:33Z", "query_hits": ["q5_videolang_long"], "seq": 116, "title": "SLAP: The Semantic Least Action Principle for Variational Video-Language Modeling", "updated": "2026-05-29T02:28:33Z"}
|
|
{"abstract": "Video spatial reasoning requires accumulating viewpoint-dependent evidence over time while retaining information useful to the question being asked. Existing spatial video-language models improve geometric perception and long-range context modeling, but often treat memory as a generic temporal cache, which can introduce redundant or irrelevant evidence and weaken long-horizon reasoning. We propose Q-GeoMem, a question-guided geometric memory framework for video spatial reasoning. Q-GeoMem injects camera-conditioned geometry into visual tokens and maintains two complementary memories: a Fine-Grained Context Bank for recent dense features and camera states, and a Semantic-Geometric Evidence Bank for compact long-range evidence. For each candidate frame, a calibrated Q-Former estimates question relevance, while novelty and evidence utility are recomputed with respect to the active evidence bank. The resulting relevance-novelty utility controls capacity-based replacement and serves as an attention bias during memory reading. During reasoning, both memories are read before update and adaptively fused with the current frame representation. Extensive experiments across two in-domain and five out-of-distribution benchmarks, and controlled memory analyses show that Q-GeoMem achieves state-of-the-art performance in the evaluated settings and validate the effectiveness of question-guided geometric evidence selection.", "authors": ["Xianqiang Gao", "Qizhi Chen", "Delin Qu", "Haoming Song", "Zhigang Wang", "Bin Zhao", "Dong Wang", "Xuelong Li"], "base_id": "2605.27318", "categories": ["cs.CV"], "id": "2605.27318v2", "pdf_url": "https://arxiv.org/pdf/2605.27318v2", "primary_category": "cs.CV", "published": "2026-05-26T17:26:29Z", "query_hits": ["q5_videolang_long"], "seq": 117, "title": "Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning", "updated": "2026-07-06T17:18:57Z"}
|
|
{"abstract": "Visually-rich documents such as reports, slides, and manuals often distribute the evidence needed to answer a question across multiple pages, mixing text with layout cues, tables, charts, and figures. This work studies multimodal retrieval-augmented generation for question answering over such visually-rich long documents, where retrieval must select evidence pages that include both textual and visual signals. We present VLD-RAG, an agentic multimodal RAG framework for multi-page evidence retrieval and cross-page reasoning over long documents. VLD-RAG builds a page-preserving multimodal index that stores parsed text, page-level metadata, and dense visual representations, and uses a hybrid retrieval strategy that combines keyword-based sparse search with dense semantic queries to identify candidate sources and evidence pages. A verifier-guided agent workflow coordinates a Retrieval Agent, Answer Agent, and Validation Agent to broaden evidence coverage, detect missing citations, and refine retrieval requests when needed. We evaluate retrieval with Top-1 and Top-5 evidence-page accuracy and generation with generalized accuracy, and show that VLD-RAG improves both evidence-page retrieval and end-task question answering on visually-rich long-document benchmarks, including LongDocURL and MMLongBench-Doc, outperforming previous vision-based retrieval baselines. These findings highlight that coordinated agent verification and multimodal hybrid retrieval are crucial for reliable grounding when correct answers depend on evidence scattered across pages.", "authors": ["Seonok Kim"], "base_id": "2607.24748", "categories": ["cs.IR", "cs.AI", "cs.CL"], "id": "2607.24748v1", "pdf_url": "https://arxiv.org/pdf/2607.24748v1", "primary_category": "cs.IR", "published": "2026-05-21T13:33:38Z", "query_hits": ["q4_long_document"], "seq": 118, "title": "VLD-RAG: Agentic Vision-Language Retrieval-Augmented Generation for Long, Visually-Rich Multi-Page Documents", "updated": "2026-05-21T13:33:38Z"}
|
|
{"abstract": "In this report, we present our champion solutions for the Natural Language Queries and GoalStep tracks of the Ego4D Episodic Memory Challenge at CVPR 2026. Both tracks require accurately localizing temporal segments from long untrimmed egocentric videos. To address these tasks, we propose a reranking-based framework that effectively leverages the strong video-language reasoning capability of multimodal large language model (MLLM) while preserving the efficiency and candidate recall of conventional localization pipelines. Specifically, we first obtain a set of candidate segments from existing localization model OSGNet, and then employ MLLM to select the segment that best matches the given query, thereby refining the final prediction. Ultimately, our method achieved first place in both the Natural Language Queries and GoalStep tracks. Our code can be found at https://github.com/iLearn-Lab/CVPR25-OSGNet.", "authors": ["Yisen Feng", "Leigang Qu", "Haoyu Zhang", "Qiaohui Chu", "Meng Liu", "Xuemeng Song", "Weili Guan", "Liqiang Nie"], "base_id": "2605.20818", "categories": ["cs.CV"], "id": "2605.20818v1", "pdf_url": "https://arxiv.org/pdf/2605.20818v1", "primary_category": "cs.CV", "published": "2026-05-20T07:14:11Z", "query_hits": ["q5_videolang_long"], "seq": 119, "title": "OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026", "updated": "2026-05-20T07:14:11Z"}
|
|
{"abstract": "Existing works typically focus on presentation generation under isolated input settings, whereas real-world use cases span diverse scenarios, including vague user prompts, long documents, multimodal materials, and multiple heterogeneous sources. Moreover, current evaluations are often insufficiently scenario-specific. They mainly rely on generic presentation-quality criteria, such as visual appeal, layout quality, and overall coherence, but fail to assess the core capabilities required by different input settings, including grounded compression, visual-text alignment, and cross-source synthesis. Consequently, the field lacks a unified benchmark and a scenario-aware evaluation framework for faithfully diagnosing presentation-generation systems across diverse real-world settings. We present UniPPTBench, a unified benchmark for presentation generation across four representative input settings: vague-prompt, long-document, multimodal-document, and multi-source generation. We further introduce UniPPTEval, a scenario-aware evaluation protocol that combines shared metrics for cross-setting comparison with scenario-specific metrics tailored to the core requirements of each setting. We also provide transparent reference baselines to support reproducible comparison. Experiments on UniPPTBench reveal substantial performance variation across settings and recurring failure modes in content grounding, multimodal integration, and cross-source synthesis. In particular, strong performance on generic presentation-quality metrics does not necessarily imply strong task fulfillment in grounded scenarios. Together, UniPPTBench and UniPPTEval provide a faithful and diagnostic foundation for evaluating presentation generation across diverse real-world scenarios. Code and data will be publicly available.", "authors": ["Bo Zhao", "Maosheng Pang", "Chen Zhang", "Huan Yang", "Yixin Cao", "Wei Ji"], "base_id": "2605.17356", "categories": ["cs.CV"], "id": "2605.17356v1", "pdf_url": "https://arxiv.org/pdf/2605.17356v1", "primary_category": "cs.CV", "published": "2026-05-17T09:50:16Z", "query_hits": ["q4_long_document"], "seq": 120, "title": "UniPPTBench: A Unified Benchmark for Presentation Generation Across Diverse Input Settings", "updated": "2026-05-17T09:50:16Z"}
|
|
{"abstract": "Long-context modeling is becoming a core capability of modern large vision-language models (LVLMs), enabling sustained context management across long-document understanding, video analysis, and multi-turn tool use in long-horizon agents. Yet practical training recipes remain insufficiently explored, particularly for designing and balancing long-context data mixtures. In this work, we present a systematic study of long-context continued pre-training (LongPT) for LVLMs, extending a 7B model from 32K to 128K context with extensive ablations on long-document data. We first show that long-document VQA is substantially more effective than OCR transcription. Building on this observation, our ablations further yield three key findings: i) for sequence-length distribution, balanced data outperforms target-length-focused data (e.g., 128K), suggesting that long-context ability requires generalizable key-information retrieval across various lengths and positions; ii) retrieval remains the primary bottleneck, favoring retrieval-heavy mixtures with modest reasoning data for task diversity; iii) pure long-document VQA largely preserves short-context capabilities, suggesting that instruction-formatted long data reduces the need for short-data mixing. Based on these findings, we introduce MMProLong, obtained by long-context continued pre-training from Qwen2.5-VL-7B with only a 5B-token budget. MMProLong improves long-document VQA scores by 7.11% and maintains strong performance at 256K and 512K contexts beyond its 128K training window, without additional training. It further generalizes to webpage-based multimodal needle retrieval, long-context vision-text compression, and long-video understanding without task-specific supervision. Overall, our study establishes a practical LongPT recipe and an empirical foundation for advancing long-context vision-language models.", "authors": ["Zhaowei Wang", "Lishu Luo", "Haodong Duan", "Weiwei Liu", "Sijin Wu", "Ji Luo", "Shen Yan", "Shuai Peng", "Sihang Yuan", "Chaoyi Huang", "Yi Lin", "Yangqiu Song"], "base_id": "2605.13831", "categories": ["cs.CV"], "id": "2605.13831v2", "pdf_url": "https://arxiv.org/pdf/2605.13831v2", "primary_category": "cs.CV", "published": "2026-05-13T17:52:53Z", "query_hits": ["q4_long_document"], "seq": 121, "title": "Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Context", "updated": "2026-10-18T16:19:51Z"}
|
|
{"abstract": "Listwise reranking is a key yet computationally expensive component in vision-centric retrieval and multimodal retrieval-augmented generation (M-RAG) over long documents. While recent VLM-based rerankers achieve strong accuracy, their practicality is often limited by long visual-token sequences and multi-step autoregressive decoding. We propose ZipRerank, a highly efficient listwise multimodal reranker that directly addresses both bottlenecks. It reduces input length via a lightweight query-image early interaction mechanism and eliminates autoregressive decoding by scoring all candidates in a single forward pass. To enable effective learning, ZipRerank adopts a two-stage training strategy: (i) listwise pretraining on large-scale text data rendered as images, and (ii) multimodal finetuning with VLM-teacher-distilled soft-ranking supervision. Extensive experiments on the MMDocIR benchmark show that ZipRerank matches or surpasses state-of-the-art multimodal rerankers while reducing LLM inference latency by up to an order of magnitude, making it well-suited for latency-sensitive real-world systems. The code is available at https://github.com/dukesun99/ZipRerank.", "authors": ["Yiqun Sun", "Pengfei Wei", "Lawrence B. Hsieh"], "base_id": "2605.11864", "categories": ["cs.IR", "cs.AI", "cs.CV", "cs.MM"], "id": "2605.11864v1", "pdf_url": "https://arxiv.org/pdf/2605.11864v1", "primary_category": "cs.IR", "published": "2026-05-12T09:45:59Z", "query_hits": ["q4_long_document"], "seq": 122, "title": "Very Efficient Listwise Multimodal Reranking for Long Documents", "updated": "2026-05-12T09:45:59Z"}
|
|
{"abstract": "Evaluating whether Multimodal Large Language Models can produce trustworthy, verifiable reasoning over long, visually rich documents requires evaluation beyond end-to-end answer accuracy. We introduce DocScope, a benchmark that formulates long-document QA as a structured reasoning trajectory prediction problem: given a complete PDF document and a question, the model outputs evidence pages, supporting evidence regions, relevant factual statements, and a final answer. We design a four-stage evaluation protocol -- Page Localization, Region Grounding, Fact Extraction, and Answer Verification -- that audits each level of the trajectory independently through inter-stage decoupling, with all judges selected and calibrated via human alignment studies. DocScope comprises 1,124 questions derived from 273 documents, with all hierarchical evidence annotations completed by human annotators. We benchmark 6 proprietary models, 12 open-weight models, and several domain-specific systems. Our experiments reveal that answer accuracy cannot substitute for trajectory-level evaluation: even among correct answers, the highest observed rate of complete evidence chains is only 29\\%. Across all models, region grounding remains the weakest trajectory stage. Furthermore, the primary difficulty stems from aggregating evidence dispersed across long distances and multiple document clusters, while an oracle study identifies faithful perception and fact extraction as the dominant capability bottleneck. Cross-architecture comparisons further suggest that activated parameter count matters more than total scale. The benchmark and code will be publicly released at https://github.com/MiliLab/DocScope.", "authors": ["Xiang Feng", "Jiawei Zhou", "Zhangfeng Huang", "Kewei Wang", "Shanshan Ye", "Jinxin Hu", "Zulong Chen", "Yong Luo", "Jing Zhang"], "base_id": "2605.08888", "categories": ["cs.CL", "cs.CV"], "id": "2605.08888v2", "pdf_url": "https://arxiv.org/pdf/2605.08888v2", "primary_category": "cs.CL", "published": "2026-05-09T11:12:59Z", "query_hits": ["q4_long_document"], "seq": 123, "title": "DocScope: Benchmarking Verifiable Reasoning for Trustworthy Long-Document Understanding", "updated": "2026-05-14T06:57:18Z"}
|
|
{"abstract": "Long video understanding with multimodal language models suffers from three compounding bottlenecks: heavy decode cost to obtain dense RGB frames, token growth with frame count, and weak motion perception under sparse keyframe sampling. Existing remedies either prune visual tokens after decoding, which leaves the expensive RGB pipeline untouched, or they discretise codec motion vectors with a generative tokeniser, which couples motion modelling to a heavy pre-training stage. In this paper we present HIMMEL, a hierarchical video-language framework that allocates semantic and motion capacity along separate paths. A small set of sparse semantic anchor frames is routed to the expensive host ViT to ground object identity and scene layout, while the far denser inter-frame intervals are encoded by a lightweight compressed-domain tri-stream adapter that distils motion evidence from motion-vector maps, residual maps, and an I-frame context branch into aligned motion tokens. These tokens are injected into the LLM via a differentiable placeholder mechanism, after a dedicated Stage 1 contrastive alignment that places the motion representation in a geometry compatible with the frozen visual backbone. We further show that an InfoNCE alignment objective beats MSE regression by +1.5 pp. On Video-MME, HIMMEL surpasses the dense 32-frame baseline by +2.3 pp (61.2 to 63.5%) while using 3.6 times fewer context tokens and running end-to-end in 1.03 s per question instead of 2.75 s. Switching to a stronger Qwen3-VL-8B host pushes HIMMEL to 64.9%, narrowing the gap to much larger proprietary systems while keeping inference on a single NVIDIA H20 GPU. Extensive ablations across stream composition, motion-encoder family, fusion mode, alignment objective, anchor count, LoRA rank, and video duration confirm that the full tri-stream contributes to the observed gains, and that the benefit grows with video length.", "authors": ["Haopeng Jin", "Wenlong Zhao", "Tao Yu", "Tiankun Yang", "Zhenyu Guan", "ShiQuan Dong", "Yujia Yang", "Shani Ye", "Chubin Chen", "Hongzhu Yi", "Jinwen Luo"], "base_id": "2605.08158", "categories": ["cs.CV", "cs.AI"], "id": "2605.08158v3", "pdf_url": "https://arxiv.org/pdf/2605.08158v3", "primary_category": "cs.CV", "published": "2026-05-04T09:35:25Z", "query_hits": ["q5_videolang_long"], "seq": 124, "title": "HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding", "updated": "2026-10-15T09:04:56Z"}
|
|
{"abstract": "Recently, video language models (VLMs) have been applied in various fields. However, the visual token sequence of the VLM is too long, which may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in VLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This paper introduces a novel approach called WindowQuant, which employs window-adaptive mixed-precision quantization to optimize the KV cache. WindowQuant consists of two modules: window-level quantization search and window-level KV cache computation. Window-level quantization search quickly determines the optimal bit-width configuration of the KV cache windows based on the similarity scores between the corresponding visual token windows and the text prompt, maintaining the model accuracy. Furthermore, window-level KV cache computation reorders the KV cache windows before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that WindowQuant outperforms state-of-the-art VLM models and KV cache quantization methods on various datasets.", "authors": ["Wei Tao", "Xiaoyang Qu", "Peiqiang Wang", "Guokuan Li", "Jiguang Wan", "Kai Lu", "Jianzong Wang"], "base_id": "2605.02262", "categories": ["cs.CV", "cs.CL"], "id": "2605.02262v1", "pdf_url": "https://arxiv.org/pdf/2605.02262v1", "primary_category": "cs.CV", "published": "2026-05-04T06:17:13Z", "query_hits": ["q5_videolang_long"], "seq": 125, "title": "WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization", "updated": "2026-05-04T06:17:13Z"}
|
|
{"abstract": "Video temporal grounding (VTG) aims to localize the start and end timestamps of the event described by a given query within an untrimmed video. Despite the strong open-world video understanding and recognition ability of video language large models (Vid-LLMs), outputting precise temporal grounding information remains challenging, since explicit temporal cues are scarce in untrimmed videos, and query-relevant entities are hard to track consistently across the video timeline. In this paper, we present \\MarkIt{}, a training-free framework that transforms an input video into a query-conditioned marked video, which empowers Vid-LLMs to generate more reliable temporal localization predictions. The core component of \\MarkIt{} is an annotation-free query-to-mask grounding bridge (Q2M-Bridge). Given a natural-language query, it automatically derives a compact set of canonical subject tags through linguistic parsing and normalization, then maps these tags to query-conditioned instance masks using text-conditioned open-vocabulary segmentation. The bridge also embeds lightweight semantic instance markers and a persistent frame index into each frame, effectively transforming long-range temporal reasoning into explicit visual cues for Vid-LLMs. \\MarkIt{} adopts an inference-time plug-and-play design, needs no modifications to Vid-LLM weights, and is fully compatible with supervised fine-tuning. Experiments conducted on multiple mainstream moment retrieval and highlight detection benchmarks demonstrate that \\MarkIt {} achieves state-of-the-art results, delivering consistent temporal grounding improvements across a wide range of existing models.", "authors": ["Pengcheng Fang", "Yuxia Chen", "Xiaohao Cai"], "base_id": "2604.25886", "categories": ["cs.MM"], "id": "2604.25886v3", "pdf_url": "https://arxiv.org/pdf/2604.25886v3", "primary_category": "cs.MM", "published": "2026-04-28T17:29:19Z", "query_hits": ["q5_videolang_long"], "seq": 126, "title": "MarkIt: Training-Free Visual Markers for Precise Video Temporal Grounding", "updated": "2026-06-16T05:05:15Z"}
|
|
{"abstract": "Semantic Multi-Object Tracking (SMOT) extends multi-object tracking with semantic outputs such as video summaries, instance-level captions, and interaction labels, aiming to move from trajectories to human-interpretable descriptions of dynamic scenes. Existing SMOT systems are trained end-to-end, coupling progress to expensive supervision, limiting the ability to rapidly adapt to new foundation models and new interactions. We propose TF-SMOT, a training-free SMOT pipeline that composes pretrained components for detection, mask-based tracking, and video-language generation. TF-SMOT combines D-FINE and the promptable SAM2 segmentation tracker to produce temporally consistent tracklets, uses contour grounding to generate video summaries and instance captions with InternVideo2.5, and aligns extracted interaction predicates to BenSMOT WordNet synsets via gloss-based semantic retrieval with LLM disambiguation. On BenSMOT, TF-SMOT achieves state-of-the-art tracking performance within the SMOT setting and improves summary and caption quality compared to prior art. Interaction recognition, however, remains challenging under strict exact-match evaluation on the fine-grained and long-tailed WordNet label space; our analysis and ablations indicate that semantic overlap and label granularity substantially affect measured performance.", "authors": ["Laurence Bonat", "Francesco Tonini", "Elisa Ricci", "Lorenzo Vaquero"], "base_id": "2604.14074", "categories": ["cs.CV"], "id": "2604.14074v1", "pdf_url": "https://arxiv.org/pdf/2604.14074v1", "primary_category": "cs.CV", "published": "2026-04-15T16:44:57Z", "query_hits": ["q5_videolang_long"], "seq": 127, "title": "Training-Free Semantic Multi-Object Tracking with Vision-Language Models", "updated": "2026-04-15T16:44:57Z"}
|
|
{"abstract": "Existing Multimodal Large Language Models (MLLMs) suffer from significant performance degradation on the long document understanding task as document length increases. This stems from two fundamental challenges: 1) a low Signal-to-Noise Ratio (SNR), with crucial evidence buried in irrelevant pages; and 2) supervision scarcity, as datasets offering only final short answers provide a weak learning signal. In this paper, we address these challenges by proposing a paradigm that requires the model to execute a structured Analysis, Localization and Reasoning workflow. To instill this capability, we design a two-stage training framework: we first perform Supervised Fine-Tuning on high-quality data generated via an efficient knowledge distillation strategy. Subsequently, we employ an Evidence-aware Group Relative Policy Optimization which jointly optimizes for both evidence localization and answer accuracy. Additionally, we introduce a Evidence-Guided Resolution Allocation strategy to mitigate memory constraints of training on multi-pages documents. Extensive experiments demonstrate that DocSeeker achieves superior performance on both in-domain and out-of-domain tasks. We show it robustly generalizes from short-page training to ultra-long documents and is naturally synergistic with visual Retrieval-Augmented Generation systems, serving as a solid foundation for their implementation.", "authors": ["Hao Yan", "Yuliang Liu", "Xingchen Liu", "Yuyi Zhang", "Minghui Liao", "Jihao Wu", "Wei Chen", "Xiang Bai"], "base_id": "2604.12812", "categories": ["cs.AI"], "id": "2604.12812v5", "pdf_url": "https://arxiv.org/pdf/2604.12812v5", "primary_category": "cs.AI", "published": "2026-04-14T14:39:26Z", "query_hits": ["q4_long_document"], "seq": 128, "title": "DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding", "updated": "2026-05-11T03:47:15Z"}
|
|
{"abstract": "Video-language foundation models have proven to be highly effective in zero-shot applications across a wide range of tasks. A particularly challenging area is the intraoperative surgical procedure domain, where labeled data is scarce, and precise temporal understanding is often required for complex downstream tasks. To address this challenge, we introduce CliPPER (Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition), a novel video-language pretraining framework trained on surgical lecture videos. Our method is designed for fine-grained temporal video-text recognition and introduces several novel pretraining strategies to improve multimodal alignment in long-form surgical videos. Specifically, we propose Contextual Video-Text Contrastive Learning (VTC_CTX) and Clip Order Prediction (COP) pretraining objectives, both of which leverage temporal and contextual dependencies to enhance local video understanding. In addition, we incorporate a Cycle-Consistency Alignment over video-text matches within the same surgical video to enforce bidirectional consistency and improve overall representation coherence. Moreover, we introduce a more refined alignment loss, Frame-Text Matching (FTM), to improve the alignment between video frames and text. As a result, our model establishes a new state-of-the-art across multiple public surgical benchmarks, including zero-shot recognition of phases, steps, instruments, and triplets. The source code and pretraining captions can be found at https://github.com/CAMMA-public/CliPPER.", "authors": ["Florian Stilz", "Vinkle Srivastav", "Nassir Navab", "Nicolas Padoy"], "base_id": "2603.24539", "categories": ["cs.CV", "cs.AI"], "id": "2603.24539v1", "pdf_url": "https://arxiv.org/pdf/2603.24539v1", "primary_category": "cs.CV", "published": "2026-03-25T17:14:36Z", "query_hits": ["q5_videolang_long"], "seq": 129, "title": "CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition", "updated": "2026-03-25T17:14:36Z"}
|
|
{"abstract": "Multimodal document question answering requires retrieving dispersed evidence from visually rich long documents and performing reliable reasoning over heterogeneous information. Existing multimodal RAG systems remain limited by two bottlenecks: static retrieval that ignores query complexity, and end-to-end Vision-Language Models (VLMs) that couple visual perception with logical reasoning, leading to inefficient computation and unstable answer generation. We propose AutoThinkRAG, a complexity-aware inference architecture for multimodal document QA. It has two components: (1) a Query Complexity Router that analyzes query difficulty and structure to adaptively select retrieval and reasoning paths; and (2) a Perception--Reasoning Decoupling architecture that uses a lightweight VLM as a high-fidelity visual interpreter to convert query-relevant visual cues into textual representations, which are then passed to an LLM for logical reasoning and answer synthesis. This design improves both efficiency and robustness, especially on long-document and unanswerable queries. Experiments on DocBench and MMLongBench show that AutoThinkRAG achieves 82.13\\% and 51.29\\% overall accuracy, respectively, while reducing per-query token consumption by 18.9\\% and monetary cost by 18.2\\%. Further analyses show that the gains are most pronounced on complex queries requiring adaptive retrieval and multi-step reasoning.", "authors": ["Jiashu Yang", "Chi Zhang", "Abudukelimu Wuerkaixi", "Xuxin Cheng", "Cao Liu", "Ke Zeng", "Xu Jia", "Xunliang Cai"], "base_id": "2603.05551", "categories": ["cs.IR", "cs.CV"], "id": "2603.05551v2", "pdf_url": "https://arxiv.org/pdf/2603.05551v2", "primary_category": "cs.IR", "published": "2026-03-05T02:29:33Z", "query_hits": ["q4_long_document"], "seq": 130, "title": "AutothinkRAG: Complexity-Aware Control of Retrieval-Augmented Reasoning for Image-Text Interaction", "updated": "2026-03-17T15:59:36Z"}
|
|
{"abstract": "The expansion of retrieval-augmented generation (RAG) into multimodal domains has intensified the challenge for processing complex visual documents, such as financial reports. While page-level chunking and retrieval is a natural starting point, it creates a critical bottleneck: delivering entire pages to the generator introduces excessive extraneous context. This not only overloads the generator's attention mechanism but also dilutes the most salient evidence. Moreover, compressing these information-rich pages into a limited visual token budget further increases the risk of hallucinations. To address this, we introduce AgenticOCR, a dynamic parsing paradigm that transforms optical character recognition (OCR) from a static, full-text process into a query-driven, on-demand extraction system. By autonomously analyzing document layout in a \"thinking with images\" manner, AgenticOCR identifies and selectively recognizes regions of interest. This approach performs on-demand decompression of visual tokens precisely where needed, effectively decoupling retrieval granularity from rigid page-level chunking. AgenticOCR has the potential to serve as the \"third building block\" of the visual document RAG stack, operating alongside and enhancing standard Embedding and Reranking modules. Experimental results demonstrate that AgenticOCR improves both the efficiency and accuracy of visual RAG systems, achieving expert-level performance in long document understanding. Code and models are available at https://github.com/OpenDataLab/AgenticOCR.", "authors": ["Zhengren Wang", "Dongsheng Ma", "Huaping Zhong", "Jiayu Li", "Wentao Zhang", "Bin Wang", "Conghui He"], "base_id": "2602.24134", "categories": ["cs.CV", "cs.CL"], "id": "2602.24134v1", "pdf_url": "https://arxiv.org/pdf/2602.24134v1", "primary_category": "cs.CV", "published": "2026-02-27T16:09:38Z", "query_hits": ["q4_long_document"], "seq": 131, "title": "AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation", "updated": "2026-02-27T16:09:38Z"}
|
|
{"abstract": "Document parsing is a fundamental task in multimodal understanding, supporting a wide range of downstream applications such as information extraction and intelligent document analysis. Benefiting from strong semantic modeling and robust generalization, VLM-based end-to-end approaches have emerged as the mainstream paradigm in recent years. However, these models often suffer from substantial inference latency, as they must autoregressively generate long, full-page sequences when processing long-form documents. While recent hybrid methods mitigate this issue via region-level parallel decoding with VLMs, independent region decoding loses full-page context and might weaken global coherence. To address this issue, we propose Hierarchical Speculative Decoding (HSD), a two-stage local-to-global framework for document parsing. HSD first employs a lightweight pipeline drafter to predict region partitions and generate coarse drafts for each region. The first stage verifies the generated region-level drafts in parallel for efficiency, while the second stage further performs page-level verification on these refined outputs to preserve full-page coherence. Experimental results show that HSD achieves a near-lossless 2.7x speedup with HunyuanOCR on OmniDocBench v1.5 and up to 7.04x speedup on long-document parsing tasks, demonstrating the effectiveness of the proposed method. The code is available at https://github.com/whlscut/HSD.", "authors": ["Wenhui Liao", "Hongliang Li", "Pengyu Xie", "Xinyu Cai", "Yufan Shen", "Yi Xin", "Qi Qin", "Shenglong Ye", "Tianbin Li", "Ming Hu", "Junjun He", "Yihao Liu", "Wenhai Wang", "Min Dou", "Bin Fu", "Botian Shi", "Yu Qiao", "Lianwen Jin"], "base_id": "2602.12957", "categories": ["cs.CV"], "id": "2602.12957v3", "pdf_url": "https://arxiv.org/pdf/2602.12957v3", "primary_category": "cs.CV", "published": "2026-02-13T14:22:10Z", "query_hits": ["q4_long_document"], "seq": 132, "title": "HSD: Training-Free Acceleration for Document Parsing Vision-Language Models with Hierarchical Speculative Decoding", "updated": "2026-06-29T06:43:50Z"}
|
|
{"abstract": "Retrieval-augmented generation is a practical paradigm for question answering over long documents, but it remains brittle for multimodal reading where text, tables, and figures are interleaved across many pages. First, flat chunking breaks document-native structure and cross-modal alignment, yielding semantic fragments that are hard to interpret in isolation. Second, even iterative retrieval can fail in long contexts by looping on partial evidence or drifting into irrelevant sections as noise accumulates, since each step is guided only by the current snippet without a persistent global search state. We introduce $G^2$-Reader, a dual-graph system, to address both issues. It evolves a Content Graph to preserve document-native structure and cross-modal semantics, and maintains a Planning Graph, an agentic directed acyclic graph of sub-questions, to track intermediate findings and guide stepwise navigation for evidence completion. On VisDoMBench across five multimodal domains, $G^2$-Reader with Qwen3-VL-32B-Instruct reaches 66.21\\% average accuracy, outperforming strong baselines and a standalone GPT-5 (53.08\\%).", "authors": ["Yaxin Du", "Junru Song", "Yifan Zhou", "Cheng Wang", "Jiahao Gu", "Zimeng Chen", "Menglan Chen", "Wen Yao", "Yang Yang", "Ying Wen", "Siheng Chen"], "base_id": "2601.22055", "categories": ["cs.CL"], "id": "2601.22055v1", "pdf_url": "https://arxiv.org/pdf/2601.22055v1", "primary_category": "cs.CL", "published": "2026-01-29T17:52:54Z", "query_hits": ["q4_long_document"], "seq": 133, "title": "$G^2$-Reader: Dual Evolving Graphs for Multimodal Document QA", "updated": "2026-01-29T17:52:54Z"}
|
|
{"abstract": "Document understanding is a long standing practical task. Vision Language Models (VLMs) have gradually become a primary approach in this domain, demonstrating effective performance on single page tasks. However, their effectiveness diminishes when handling long documents. In such scenarios, clues are often scattered across multiple pages and modalities, and redundancy from lengthy inputs can impair the models judgment. While retrieval augmented generation mitigates this issue by filtering for question relevant content, the retrieved results still contain substantial redundancy. To address these limitations, we propose SLEUTH, a multi agent framework. Concretely, SLEUTH orchestrates a retriever and four collaborative agents in a coarse to fine process. The framework identifies key textual and visual clues within the retrieved pages, filters for salient visual evidence such as tables and charts, and analyzes the query to devise a reasoning strategy. It ultimately synthesizes a distilled, evidence dense multimodal context to generate the final prediction. SLEUTH is model agnostic and scalable. When paired with advanced VLM backbones, it consistently improves performance on multiple long document benchmarks, achieving state of the art results. Ablation studies verify each modules effectiveness and confirm the benefits of our hierarchical refinement paradigm.", "authors": ["Keliang Liu", "Zizhi Chen", "Mingcheng Li", "Jingqun Tang", "Dingkang Yang", "Lihua Zhang"], "base_id": "2511.22850", "categories": ["cs.CV"], "id": "2511.22850v1", "pdf_url": "https://arxiv.org/pdf/2511.22850v1", "primary_category": "cs.CV", "published": "2025-11-28T03:09:40Z", "query_hits": ["q4_long_document"], "seq": 134, "title": "Resolving Evidence Sparsity: Agentic Context Engineering for Long-Document Understanding", "updated": "2025-11-28T03:09:40Z"}
|