907 lines
77 KiB
XML
907 lines
77 KiB
XML
<?xml version='1.0' encoding='UTF-8'?>
|
|
<feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:arxiv="http://arxiv.org/schemas/atom" xmlns="http://www.w3.org/2005/Atom">
|
|
<id>https://arxiv.org/api/zXfgRSGa21Sq5idz8y5kVCx9K90</id>
|
|
<title>arXiv Query: search_query=all:"video-language" AND all:long&id_list=&start=0&max_results=30</title>
|
|
<updated>2026-10-26T01:04:47Z</updated>
|
|
<link href="https://arxiv.org/api/query?search_query=all:%22video-language%22+AND+all:long&start=0&max_results=30&id_list=" type="application/atom+xml"/>
|
|
<opensearch:itemsPerPage>30</opensearch:itemsPerPage>
|
|
<opensearch:totalResults>145</opensearch:totalResults>
|
|
<opensearch:startIndex>0</opensearch:startIndex>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.30089v1</id>
|
|
<title>ProactEgoMem: Benchmarking Proactive Memory in Streaming Egocentric Video</title>
|
|
<updated>2026-10-21T17:37:58Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.30089v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.30089v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Today's multimodal assistants are typically prompted to interpret the visual evidence presented to them. To offload the cognitive burden of remembering across long horizons, wearable assistants should instead proactively acquire and maintain useful memory as the user's surroundings evolve. Doing so on the user's behalf requires identifying what matters, understanding the scene in real time, and initiating memory maintenance in the background while the user stays focused on their activities. We introduce ProactEgoMem, a benchmark that evaluates proactive memory in streaming egocentric video. Three tasks ground the evaluation in ambient environmental needs: Information Capture (IC) preserves transient yet dense visual information, State Tracking (ST) follows changing object states, and Location Tracking (LT) records object movements. The benchmark defines its protocol at two levels of proactiveness: the Notification Level asks for a single timely notification that satisfies the user's intent, while the deeper Intervention Level asks the agent to maintain memory on the user's behalf through an active start-to-stop tool lifecycle. Parallel ProactEgoMem-Daily and ProactEgoMem-Sci tracks contain 300/100 clips covering daily and scientific use cases, paired with 1000/500 questions in ProactEgoMem-QA, which measures the quality of the proactively acquired memory. Across proprietary, open-source, video-language, and streaming baselines, even the strongest models fail to capture memory both timely and accurately. ProactEgoMem provides a rigorous evaluation foundation for proactive multimodal assistance and for video agents that co-exist with their users.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-21T17:37:58Z</published>
|
|
<arxiv:comment>30 pages, 22 figures, Project Page: https://proactegomem.github.io/</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Ruotong Liao</name>
|
|
</author>
|
|
<author>
|
|
<name>Meikel Kokowski</name>
|
|
</author>
|
|
<author>
|
|
<name>Xiaoxiao Sun</name>
|
|
</author>
|
|
<author>
|
|
<name>Yongqin Xian</name>
|
|
</author>
|
|
<author>
|
|
<name>Laura Bravo Sánchez</name>
|
|
</author>
|
|
<author>
|
|
<name>Shiye Su</name>
|
|
</author>
|
|
<author>
|
|
<name>Federico Tombari</name>
|
|
</author>
|
|
<author>
|
|
<name>Serena Yeung-Levy</name>
|
|
</author>
|
|
<author>
|
|
<name>Volker Tresp</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.26431v1</id>
|
|
<title>Long-form Video-to-Text Summarization using Recursive Language Models</title>
|
|
<updated>2026-10-19T13:56:33Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.26431v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.26431v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>While VideoLLMs have made significant progress in understanding short video benchmarks, they often struggle to summarize long-form ones. This is mainly due to performance degradation caused by long-context prompts, the inherent difficulty of preserving temporal continuity and a lack of semantically rich concepts in event descriptions. To tackle these issues, this paper introduces the first recursive Video Language Summarization Model (RVLSM) designed for video processing and tailored to long-form video-to-text summarization. RVLSM recursively decomposes videos into semantically coherent segments and delegates chunk-level reasoning to VideoLLM sub-models and external tools. It employs (1) Semantic Temporal Decomposition (STD), which preserves temporal coherence and continuity through a scene-aware chunking strategy that dynamically partitions videos according to semantic boundaries, and (2) Adaptive Multimodal Tool Calling (AMTC), which ensures semantic richness of the summary via a routing mechanism that selectively invokes auxiliary modules such as action recognition, optical character recognition, automatic speech recognition, and object detection. Experiments on the VideoXum and MixTyVSum benchmarks show that RVLSM consistently improves over zero-shot VideoLLM baselines across multiple model settings and performance metrics, and is competitive with a state-of-the-art fine-tuned model for video-to-text summarization. Importantly, the named entities in the RVLSM's summaries align more closely with those detected in the ground-truth summaries than those detected by baseline and state-of-the-art methods, confirming the saliency of the event descriptions it generates.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-19T13:56:33Z</published>
|
|
<arxiv:comment>Paper currently under review</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Alessia Intini</name>
|
|
</author>
|
|
<author>
|
|
<name>Lorenzo Vaiani</name>
|
|
</author>
|
|
<author>
|
|
<name>Enrico Chen</name>
|
|
</author>
|
|
<author>
|
|
<name>Luca Cagliero</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.26910v1</id>
|
|
<title>Memory Internalization via Test-Time Training for Long-Video Understanding</title>
|
|
<updated>2026-10-17T04:06:57Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.26910v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.26910v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-video understanding often requires answering many questions about a single video whose evidence is distributed across extended temporal contexts. Conventional video-language models repeatedly encode frames or retrieve visual evidence for every question. Parameterized video memory offers a write-once alternative, but existing methods rely primarily on synthesized text and fixed adaptation configurations, leaving open where video knowledge should be written and how visual evidence should shape the memory. We introduce Memory Internalization via Test-Time Training (MIT^3), a per-video adaptation framework that writes each long video into compact, video-specific adaptive parameters for video-free, text-only question answering. Under compact multi-scale supervision, causal tracing measures the causal memory utility of each model layer for video-grounded answering and selects video-specific adaptation locations. Visual memory distillation then trains the text-only memory to reproduce the representation change induced by the relevant video clip, preserving the internal effect of visual evidence beyond textual supervision. Across LVBench, LongVideoBench, and Video-MME, MIT^3 achieves data-efficient video-free question answering with substantially fewer synthesized samples than prior parameterized video-memory methods, demonstrating that effective video internalization depends on both where knowledge is written and how visual evidence is distilled into the model.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-17T04:06:57Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Haonan Zhao</name>
|
|
</author>
|
|
<author>
|
|
<name>Jiaxuan Chen</name>
|
|
</author>
|
|
<author>
|
|
<name>Qingyang Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Shuochen Chang</name>
|
|
</author>
|
|
<author>
|
|
<name>Siyu Sun</name>
|
|
</author>
|
|
<author>
|
|
<name>Jiangtong Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Li Niu</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.17355v1</id>
|
|
<title>WIRE: Weight Space Implicit Representation Encoding of Frozen Video LLMs</title>
|
|
<updated>2026-10-12T20:13:21Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.17355v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.17355v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video language models represent a video as a long sequence of visual embeddings in a decoder's input. Every visual embedding is processed by every decoder layer during prefill, stored in the key value cache, and attended by every generated token, so a long video raises prefill, memory, and generation cost at once. We present WIRE (Weight Space Implicit Representation Encoding), which reduces this recurring cost by moving part of the video conditioned computation of a frozen video language model from its input activations into a temporary low rank update of its weights. WIRE keeps a short explicit visual context, so the update acts as a correction rather than a replacement of the video. WIRE first compresses the full resolution embeddings of each frame into a small set of carrier tokens using a question conditioned bidirectional selective scan, and physically contracts the decoder input so every decoder layer, cache entry, and decoding step operates on the shorter sequence. Hypernetworks at selected decoder layers read the compressed forward pass together with a readout of the full resolution features removed by contraction and generate LoRA updates to the attention and feedforward projections. The updates are merged after prefill, so autoregressive decoding runs the unmodified frozen backbone. On Qwen3-VL-8B, WIRE with 16 carrier tokens per frame contracts the visual stream by $3.5\times$, reduces decoder prefill time by $3.35\times$ at 256 frames and $4.14\times$ at 1024 frames, shrinks the key value cache proportionally, and keeps per token decoding latency approximately constant. Across nine benchmark settings at 256 frames, it obtains the highest average accuracy among efficient video language models we evaluate under a common 256 frame budget.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-12T20:13:21Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Ameen Ali</name>
|
|
</author>
|
|
<author>
|
|
<name>Tamim Zoabi</name>
|
|
</author>
|
|
<author>
|
|
<name>Lior Wolf</name>
|
|
</author>
|
|
<author>
|
|
<name>Chaim Baskin</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.08954v1</id>
|
|
<title>RACER: Reflective Agent Coupling Query Interpretation and Tool-Based Retrieval for Frame Selection in Long Video Understanding</title>
|
|
<updated>2026-10-06T18:21:13Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.08954v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.08954v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video large language models (Vid-LLMs) excel at diverse video-language tasks by reasoning over selected frames. However, frame selection for long videos remains challenging, as it requires retrieving relevant frames distributed across segments from a large candidate pool given complex queries. This paper investigates dominant approaches to long-video frame selection from a task-decomposition perspective, identifying two key challenges: the Query Comprehension Gap in similarity-based methods and the Interpretation--Selection Gap in judgment-based methods. To address them, we propose RACER, a training-free reflective agentic framework that decomposes long-video frame selection into query interpretation driven by a lightweight Vid-LLM and evidence localization supported by an embedding model serving as a retrieval tool. Specifically, the Vid-LLM is responsible solely for reformulating the complex query into sub-queries that make implicit information requirements explicit, mitigating the Query Comprehension Gap. Meanwhile, the retrieval tool leverages these sub-queries to localize relevant evidence, relieving the Vid-LLM of direct frame selection and thus addressing the Interpretation--Selection Gap. Finally, the retrieved frames are fed back to the Vid-LLM for sub-query refinement, forming a reflection loop that iteratively improves query interpretation and frame selection. Experiments across multiple benchmarks show that RACER consistently improves long video understanding. Notably, RACER achieves effective frame selection even with limited-capability components, demonstrating that agentic integration enables these components to enhance more capable Vid-LLMs.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-06T18:21:13Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Yiyang Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yitian Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yizhou Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Jianglin Lu</name>
|
|
</author>
|
|
<author>
|
|
<name>Qihua Dong</name>
|
|
</author>
|
|
<author>
|
|
<name>Hailing Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Huimin Zeng</name>
|
|
</author>
|
|
<author>
|
|
<name>Mingyuan Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yun Fu</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.08674v1</id>
|
|
<title>EC-RAG: Event Chain Retrieval-Augmented Generation for Long Video Understanding</title>
|
|
<updated>2026-10-06T16:57:46Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.08674v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.08674v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Current large video-language models (LVLMs) still face challenges when dealing with long videos, mainly because frames are often processed independently, making it difficult to capture temporal dependencies across events. Although retrieval-augmented approaches have been introduced to provide additional context, most of them operate at the frame or snippet level, which limits their ability to model how events evolve over time and relate to each other. In this paper, we propose Event Chain Retrieval-Augmented Generation (EC-RAG), a training-free framework that organizes video content into an explicit event chain before question answering. Instead of retrieving isolated frames or text segments, EC-RAG first partitions the video into semantically coherent segments, represents each segment using multi-modal signals, and then links them into a structured chain that preserves temporal order and captures inter-event relationships. Given a query, the system identifies relevant events within this chain and gathers supporting evidence from the associated modalities. Our approach offers several practical advantages: (i) event-level abstraction that better reflects how video content is naturally structured, enabling more reliable localization compared to frame-level retrieval; (ii) structured multi-modal fusion that aggregates speech, text, and visual cues at the event level, allowing complementary information to be more effectively utilized during reasoning; and (iii) plug-and-play compatibility with existing LVLM backbones, requiring no additional training or reliance on proprietary models. Experiments on Video-MME, MLVU, and LongVideoBench show that this event-centric design consistently outperforms frame-level retrieval baselines, highlighting the importance of modeling temporal structure for long-video understanding.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-06T16:57:46Z</published>
|
|
<arxiv:comment>12 pages, 7 figures, 7 tables, including supplementary material</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Yuhao Qin</name>
|
|
</author>
|
|
<author>
|
|
<name>Junbo Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yuke Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Yining Zhu</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2610.08526v1</id>
|
|
<title>WareFly-VLA: A Vision-Language-Action Framework for UAV Navigation and Human Tracking in Smart Warehouses</title>
|
|
<updated>2026-10-06T15:21:14Z</updated>
|
|
<link href="https://arxiv.org/abs/2610.08526v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2610.08526v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Vision-Language-Action (VLA) models have achieved impressive results in robotic manipulation and ground-mobile navigation, yet language-conditioned control of unmanned aerial vehicles (UAVs) in smart warehouses remains largely unexplored, hindered by the lack of benchmarks that jointly provide continuous low-level flight actions, fine-grained natural-language target descriptions, and realistic industrial environments. This paper introduces WareFly-VLA, a photorealistic UAV VLA framework and dataset for language-guided human search, localization, and tracking in warehouse environments. It contains 507 human-teleoperated flight episodes and 8,504 high-resolution RGB transitions collected in NVIDIA Isaac Sim, each paired with a human-written appearance description of the target worker and a synchronized four-degree-of-freedom control command. Two aerial tasks are covered: target approach and person following, under occlusion, long-range search, altitude variation, and clutter. A unified benchmark of four open-source VLA architectures (SmolVLA, GR00T N1.7, pi_0 and OpenVLA) is established under a leakage-free episode-level protocol at two control rates. The results show that language-conditioned aerial control in warehouses is far from solved: performance drops substantially under strict generalization settings, continuous action modeling consistently outperforms discrete action tokenization, only the forward channel is reliably learnable from a single frame, and current foundation-model interfaces transfer poorly from ground and humanoid embodiments to aerial platforms. The synchronized video, language, action, pose, and difficulty annotations further support world-model research. The dataset, baselines, and evaluation protocol are released to support language-grounded aerial autonomy in smart warehouses.</summary>
|
|
<category term="cs.RO" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-10-06T15:21:14Z</published>
|
|
<arxiv:comment>41 pages, 35 figures, 11 tables</arxiv:comment>
|
|
<arxiv:primary_category term="cs.RO"/>
|
|
<author>
|
|
<name>Thinh D. Le</name>
|
|
</author>
|
|
<author>
|
|
<name>Son T. Nguyen</name>
|
|
</author>
|
|
<author>
|
|
<name>Duong Q. Nguyen</name>
|
|
</author>
|
|
<author>
|
|
<name>Dung D. Le</name>
|
|
</author>
|
|
<author>
|
|
<name>Ngo Anh Vien</name>
|
|
</author>
|
|
<author>
|
|
<name>H. Nguyen-Xuan</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.39563v1</id>
|
|
<title>RESUME: Recurrent State Updates from Motion and Residual Signals for Efficient Video Language Modeling</title>
|
|
<updated>2026-09-30T12:00:41Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.39563v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.39563v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Existing video language models encode sampled RGB frames independently, so a long video must either exhaust the token budget or drop the changes between sampled frames. Codec-aware front-ends read the motion vectors and residuals that encoding produced, but in their deployed form each predictive frame is still tokenized on its own: the tokens are a function of the current primitives, not of a carried reference. We argue that a more natural function is of both---the current primitives and a carried reference. A clip and its time reversal share the same frames and differ only in the order of changes---an axis that symmetric pooling discards by construction, and that is non-empty in the frozen vision features VideoLMs use---and the codec recurrence already composes those changes in order against a reference state. We introduce RESUME, a stateful codec representation: an anchor I-frame initializes a compact latent state, each subsequent predictive frame is consumed as an update to that state, and a shared readout exposes VideoLM-compatible tokens from the accumulated state. Codec prediction is thereby kept at the representation level and handed to the language model as a trajectory, not as a set of independent token groups. At the same per-predictive-frame token budget as prior codec-aware methods, a predictive frame enters the language model as a readout of what the front-end already knows, not as an encoding of the current primitives alone. Across ten benchmarks, the gains concentrate on temporal reasoning: on all three temporal benchmarks RESUME improves over both the RGB-frame baseline LLaVA-Video-7B (by 2.8, 5.1, and 3.9 points on TempCompass, TOMATO, and MVBench) and the codec-based baseline CoPE-7B, while staying competitive on general and long-form QA. Frozen-transition tests further show anchor dependence, order sensitivity, and useful rollout behavior beyond the training horizon.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-30T12:00:41Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Can Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Xiaotian Han</name>
|
|
</author>
|
|
<author>
|
|
<name>Junyuan Shang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yuchen Ding</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhenyu Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Shuohuan Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Dianhai Yu</name>
|
|
</author>
|
|
<author>
|
|
<name>Ruirui Li</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.34314v1</id>
|
|
<title>PlaylistEval: Can Video-Language Judges Be Trusted at Day Scale and Beyond?</title>
|
|
<updated>2026-09-28T04:54:29Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.34314v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.34314v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video-language models are increasingly used as judges of video understanding, both for evaluating model outputs and for training reward models. Whether their judgments remain reliable when the evidence is buried in day-long videos has yet to be established. Existing benchmarks cannot answer this. Their videos are typically only a few minutes long, many answer pairs can be separated from the transcript alone, and collecting human judgments does not scale to ultra-long videos. We introduce PlaylistEval, an agentic framework that builds video-language judge benchmarks over 100-hour playlist collection without human annotation. It automatically generates questions with paired answers whose differences are controlled by causal degradation, so that every pair demands retrieval across the collection. The resulting benchmark contains 630 pairs across seven domains spanning both static and dynamic knowledge, and on a stratified subset of 152 pairs it agrees with human judgments 93.0% of the time (IAA 0.781). Evaluating 17 omnimodal and multimodal models from eight families reveals that frontier judges reach only 75.4% pairwise accuracy, while open-source judge models perform far behind. We further show that both retrieval and final judgment depend on using multiple modalities, and that judge accuracy degrades as the playlist set grows. We release our pipeline, benchmark, and evaluation code at https://playlisteval.github.io.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-28T04:54:29Z</published>
|
|
<arxiv:comment>53 pages, 15 figures, 23 tables. Project page: https://playlisteval.github.io</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Shayekh Bin Islam</name>
|
|
</author>
|
|
<author>
|
|
<name>Hwanjun Song</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.27470v1</id>
|
|
<title>DeltaS: Reading the Gated Linear Attention State for KV Cache Eviction in Streaming Video</title>
|
|
<updated>2026-09-23T07:35:52Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.27470v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.27470v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Recent video-language models increasingly adopt hybrid architectures that interleave linear and full attention layers for efficient long-context processing. While the recurrent state of linear attention remains fixed in size, the KV cache of full attention continues to grow with the video stream, making eviction necessary under a bounded memory budget. The key challenge in streaming is that eviction must occur before the question arrives, so what to retain has to be decided without the question. Existing eviction methods derive token scores from the KV cache itself, using position, attention, or key-value representations, and attention-based scores further require proxy queries or extra computation. Hybrid backbones offer another source of signal. In gated-delta linear attention, the recurrent state is updated by the residual between each input and what can already be retrieved from the state, so its change over a chunk of frames reflects how much new information the chunk brings. We propose DeltaS, a query-agnostic, training-free method that retains video chunks inducing larger normalized state change, or state drift. In a controlled comparison with the budget and retention policy held fixed, state drift outperforms position-, attention-, and key-value-based signals. With a signal costing only 1.9% of the forward pass, DeltaS surpasses the strongest query-agnostic bounded-memory baseline by 2.1 points on average across six long-video benchmarks and by 5.6 points on the longest benchmark. These results suggest that the two memories of hybrid architectures can work cooperatively. Code is available at https://github.com/MaumAI-Company/DeltaS.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-23T07:35:52Z</published>
|
|
<arxiv:comment>15 pages, 8 figures, 6 tables. Code: https://github.com/MaumAI-Company/DeltaS</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Taeyoun Kwon</name>
|
|
</author>
|
|
<author>
|
|
<name>Seungjin Kim</name>
|
|
</author>
|
|
<author>
|
|
<name>Hyeonyu Kim</name>
|
|
</author>
|
|
<author>
|
|
<name>Moon Hwan Kim</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.08755v1</id>
|
|
<title>Kairos: A Dataset for Fine-Grained Video-Language Modeling over Space, Time, and Dynamics</title>
|
|
<updated>2026-09-08T13:50:30Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.08755v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.08755v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Many emerging video language modeling tasks require systems to move beyond clip-level abstraction and model visual content as it unfolds over extended time horizons. However, most existing video datasets rely on coarse or sparsely aligned supervision, which compresses temporal variation and limits the ability of models to learn reusable representations of continuous visual dynamics. We introduce Kairos, a video dataset for video-language modeling with time-resolved annotations. Kairos consists of long-duration videos, ranging from ten minutes to half an hour, annotated with fine-grained temporal alignment. The annotations capture ongoing actions, entity appearances and attributes, interactions, and evolving contextual cues along the video timeline. This time-resolved structure supports fine-grained evaluation, long-range modeling and reasoning, instruction data construction, representation learning, and video generation. Kairos provides a general-purpose foundation for modeling visual experiences over time.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-08T13:50:30Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Ruibo Ming</name>
|
|
</author>
|
|
<author>
|
|
<name>Lei Sun</name>
|
|
</author>
|
|
<author>
|
|
<name>Deheng Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>He Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Jialu Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Jian Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhendong Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Mengshun Hu</name>
|
|
</author>
|
|
<author>
|
|
<name>Danda Pani Paudel</name>
|
|
</author>
|
|
<author>
|
|
<name>Luc Van Gool</name>
|
|
</author>
|
|
<author>
|
|
<name>Jinjin Gu</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.04802v3</id>
|
|
<title>Linguistic Trajectory Encoding for Efficient Long-Horizon Spatial Memory in Embodied Agents</title>
|
|
<updated>2026-09-29T08:04:58Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.04802v3" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.04802v3" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Embodied agents performing long-horizon tasks require a memory representation in which the state transitions of dynamic objects remain queryable in natural language across hours-to-days observation horizons. Existing systems either drop fine-grained motion (clip-level video-language embeddings), keep it only as raw coordinates (geometric SLAM), or organise it around immediate task context (agent working memories). None of them gives the agent a per-object timeline whose state transitions are themselves queryable in language. Our key contribution is \textbf{Linguistic Trajectory Encoding} (LTE), which compresses dynamic object motion histories via a hybrid representation combining natural language descriptions, sparse spatial anchors, and visual anchors. LTE adapts compression to motion complexity by anchoring periods without reliable observations to the last seen location, while representing motion with geometric waypoints and linguistic descriptions to preserve accuracy. To evaluate these capabilities across extended time horizons, we construct the \textbf{Spatial Memory Benchmark} (SMB) from EgoLife multi-day recordings, targeting capabilities absent in existing benchmarks: semantic trajectory retrieval and long-horizon object retrieval. On SMB, the LTE-based system achieves $45.3\%$ success in semantic trajectory retrieval and $48.7\%$ in long-horizon object retrieval, outperforming structured-memory and VLM baselines (best prior: $31.9\%$ and $34.4\%$). LTE achieves trajectory compression by factors of $8.7\times$ to $26.1\times$ with sub-second query latency on $24$\,h video. On Ego4D natural-language queries, the system reaches $28.75\%$ / $55.10\%$ R@1/R@5, $+15.80$ / $+31.30$ pts over EgoVLPv2.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-04T07:00:48Z</published>
|
|
<arxiv:comment>Accepted at NeurIPS 2026. Project page: https://sealical.github.io/st-mem/</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Tianyidan Xie</name>
|
|
</author>
|
|
<author>
|
|
<name>Shenyi Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Qiang Tang</name>
|
|
</author>
|
|
<author>
|
|
<name>Mingjie Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhicheng Qiu</name>
|
|
</author>
|
|
<author>
|
|
<name>Xuanfu Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhan Xu</name>
|
|
</author>
|
|
<author>
|
|
<name>Jian Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Lanjun Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zili Yi</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.04550v1</id>
|
|
<title>VISTA: Dense Multi-Label Classroom Coding with Vision-Language Models</title>
|
|
<updated>2026-09-03T23:13:56Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.04550v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.04550v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video-language benchmarks are usually constructed by the dataset authors without published reliability statistics, leaving the noise floor of the construct unknown. We argue that multimodal benchmarking benefits from methods taken from research communities that have already invested in strategies to ensure reliability. We illustrate the case with the Classroom Observation Protocol for Undergraduate STEM (COPUS): a 24-code multi-label observation instrument with a decade of peer-reviewed reliability literature. We recast COPUS as a video benchmark for multimodal foundation models, where it provides a dense set of structured labels (a 24-dimensional binary vector every 2 minutes across a 50-90 minute lecture), an externally validated vocabulary, and established literature that provides a per-code reliability target based on human evaluators. Annotations in our evaluation corpus are produced by a 5-person human-evaluator panel whose consensus matrix is our reference. We propose VISTA, a baseline that runs MiniCPM-V-4.5 over a dense sliding window, refines its per-window outputs with a lightweight multi-layer perceptron (MLP) head trained on top of the frozen backbone, and max-pools the resulting predictions onto the 2-minute COPUS grid. On three held-out chemistry lectures, VISTA reaches 80.1% restricted macro accuracy versus 74.9% for the zero-shot variant, with the largest residual errors on visually similar instructor codes and on rare audio-dependent codes. We characterize three systematic failure modes (audio-partial observability, fine-grained group-work discrimination, long-tail recall) and release the benchmark tooling, prompts and baseline code at https://github.com/ajfranck/VISTA.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-03T23:13:56Z</published>
|
|
<arxiv:comment>DataMFM Workshop @ Computer Vision & Pattern Recognition (CVPR) 2026</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Andrew Franck</name>
|
|
</author>
|
|
<author>
|
|
<name>Brendan Ng</name>
|
|
</author>
|
|
<author>
|
|
<name>Ben Fitzgerald</name>
|
|
</author>
|
|
<author>
|
|
<name>Zane Derrod</name>
|
|
</author>
|
|
<author>
|
|
<name>Chris Cianci</name>
|
|
</author>
|
|
<author>
|
|
<name>Chris Craney</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.03820v1</id>
|
|
<title>Select, Compress, Reinvest: A Controlled Study of Visual-Token Allocation in Long-Video MLLMs</title>
|
|
<updated>2026-09-03T13:20:48Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.03820v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.03820v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-video language models cannot look at every frame: an hour sampled once per second is 3,600 images, and a system keeps only a small fixed slice of that pool. Which frames survive that slice is usually treated as a preprocessing detail; we test whether it should be. Published selectors make the comparison hard because they change the frame scorer, the prompt boundary, the resolution policy, and the answering model all at once. We hold each fixed and vary one decision at a time: selection, spatial compression, and reinvestment of the savings, across six training-free selection rules, three long-video benchmarks, and two answering models. Selection is the largest single lever: on LongVideoBench's hour-long bin, eight query-selected frames beat sixteen uniformly spaced ones by 6.9 points, and Orthogonal Matching Pursuit, an unmodified decades-old sparse-approximation algorithm, matches or comes within a point of every purpose-built selector we compare it against, across all three benchmarks. Compression is close to free: halving each frame's spatial budget at fixed timestamps costs at most 0.44 points. Reinvestment is where that budget turns back into accuracy: spending the freed tokens on twice as many compressed frames, at a measured cost no higher than the original eight, returns a further two to three points; compression only pays off once its savings are spent this way. Along the way, an implementation bug in our own AKS baseline and a 0.07 to 3.74 point gap between two harnesses running the same published rules at the same budget show why these comparisons need to happen inside one controlled harness rather than across papers.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-03T13:20:48Z</published>
|
|
<arxiv:comment>16 pages, 6 figures. Code and data: https://github.com/codeprakhar25/omp-keyframe-sampling</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Prakhar Khatri</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2609.01004v1</id>
|
|
<title>SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models</title>
|
|
<updated>2026-09-01T09:52:10Z</updated>
|
|
<link href="https://arxiv.org/abs/2609.01004v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2609.01004v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Despite their strong multimodal understanding ability, multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences. To reduce inference costs, recent studies have explored visual token pruning through vision-centric or text-guided strategies. However, these methods often overlook high-norm outlier tokens, i.e., tokens with abnormally large feature norms, leading to suboptimal pruning decisions. In this work, we show that such high-norm outlier tokens are highly redundant in both feature and spatial dimensions, yet are often mistakenly preserved as informative cues by existing methods.
|
|
Motivated by this observation, we propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference. SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query.
|
|
Extensive experiments on twelve image-language and four video-language benchmarks demonstrate the effectiveness, efficiency, and generalizability of our framework. Notably, SinkPruner preserves 96.5% (91.8%) of the original performance of LLaVA-1.5 (Qwen2.5-VL) under an 89% token reduction. Experiments further indicate that our visual sanitizer exhibits promising transferability in enhancing the performance of existing pruning methods. Our code is available at https://github.com/LaVi-Lab/SinkPruner.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.LG" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-09-01T09:52:10Z</published>
|
|
<arxiv:comment>EMNLP 2026 (findings)</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Shiyu Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Zi-Yuan Hu</name>
|
|
</author>
|
|
<author>
|
|
<name>Shijia Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yanyang Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Yiwu Zhong</name>
|
|
</author>
|
|
<author>
|
|
<name>Liwei Wang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2608.19723v1</id>
|
|
<title>StreamSoccer: Event-Driven Memory for Streaming Soccer Commentary</title>
|
|
<updated>2026-08-20T07:19:29Z</updated>
|
|
<link href="https://arxiv.org/abs/2608.19723v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2608.19723v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Streaming video understanding requires models to causally update state as video arrives and organize growing history into semantic units that can evolve, persist, and be recalled under bounded computation and memory. This challenge is pronounced in live soccer commentary, where a system must describe completed events, summarize recent play, recall earlier events, or remain silent using only information available before each utterance. We present StreamSoccer, an event-driven system that uses event memory as its intermediate representation. A fixed-budget active memory integrates the stream; completed event states are retained locally and consolidated into retrievable historical records. A unified generator uses current, recent, and historical context to produce three commentary modes, while a rule-assisted scheduler selects a mode or silence. Unlike streaming video-language models organized around frames, visual tokens, or caches, and soccer-commentary methods based on predefined clips or output timestamps, StreamSoccer explicitly models event lifecycles. We construct a three-track streaming soccer commentary dataset and a layered evaluation protocol. At common reference anchors, StreamSoccer obtains CIDEr scores of 38.62, 23.96, and 17.39 for current-event, recent-window, and historical-memory commentary, ranking first on the current-event and historical-memory tracks and second on recent-window. Controlled ablations show that local completed events improve all tracks and that the full system performs best on all three. Across 174 raw-video runs on 58 matches, per-minute RTF p95 ranges from 0.10 to 0.22 without sustained growth with match history. These results indicate that event memory supports streaming soccer commentary across temporal scopes while controlling long-history computation.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-08-20T07:19:29Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Chenxi Shao</name>
|
|
</author>
|
|
<author>
|
|
<name>Bozhong Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Jiaxin Huang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhao Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Sunwei Zhu</name>
|
|
</author>
|
|
<author>
|
|
<name>Tianxin Hang</name>
|
|
</author>
|
|
<author>
|
|
<name>Gaoqi He</name>
|
|
</author>
|
|
<author>
|
|
<name>Yang Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Changbo Wang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2608.18386v1</id>
|
|
<title>TTSD-FAR: Test-Time Self-Distillation with Fisher-Anchored Restoration for Missing-Modality Emotion Recognition in LVLMs</title>
|
|
<updated>2026-08-18T23:40:28Z</updated>
|
|
<link href="https://arxiv.org/abs/2608.18386v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2608.18386v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Large video-language models (LVLMs) have shown remarkable performance on multimodal tasks like multimodal emotion recognition (ER) in the wild. ER is inherently multimodal, requiring a joint understanding of facial expressions, vocalizations, language, biosignals, and gestures. However, real-world deployment remains challenging: modalities may be missing or noisy at test time. Partial observations can be viewed as a distribution shift relative to the complete-modality distribution. SOTA TTA methods based on entropy minimization or perplexity reduction do not transfer to autoregressive LVLMs, while retrieval augmented generation (RAG) degrades when the observed modality is weak. Because no ground-truth supervision exists to verify individual updates, adaptation across this stream risks accumulating drift and degrading once the model departs from a reliable solution. An effective solution must therefore adapt to arbitrary missing-modality patterns and remain effective during continual adaptation. We address both jointly with Test-Time Self-Distillation (TTSD), a parameter-efficient framework in which a frozen teacher, trained on complete modalities, guides an adaptive low-rank student via self-distillation, updating only a negligible number of parameters. Stability is built into this same loop through Fisher-Anchored Restoration (FAR), which monitors Fisher information stability to detect convergence versus drift and restores the student toward the teacher's anchor when distributional shifts are identified. Our experiments on MELD, DFEW, and BAH under 0%-50% missing modalities show that this unified adaptation-restoration design consistently outperforms entropy-based adaptation, RAG, and perplexity-based generation over long adaptation horizons, where baselines without restoration progressively degrade while TTSD-FAR remains consistent.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-08-18T23:40:28Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Muhammad Haseeb Aslam</name>
|
|
</author>
|
|
<author>
|
|
<name>Alessandro Koerich</name>
|
|
</author>
|
|
<author>
|
|
<name>Marco Pedersoli</name>
|
|
</author>
|
|
<author>
|
|
<name>Ali Etemad</name>
|
|
</author>
|
|
<author>
|
|
<name>Eric Granger</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2608.04302v1</id>
|
|
<title>CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models</title>
|
|
<updated>2026-08-05T00:20:23Z</updated>
|
|
<link href="https://arxiv.org/abs/2608.04302v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2608.04302v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The evaluation suite employs an ensemble of five state-of-the-art LLM-based embedding models to increase reliability and mitigate single-model bias, and applies two complementary methodologies: (i) coarse-grained semantic matching and (ii) fine-grained semantic matching to compare model-generated descriptions against CLIP-CC-Bench references. Using this framework, we evaluate 17 state-of-the-art video-language models and report both their Borda-aggregated rankings and their average scores on CLIP-CC-Bench. We further quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability. We release standardized evaluation scripts, model outputs, and aggregation tools at https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench to support reproducibility. CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.IR" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.MM" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-08-05T00:20:23Z</published>
|
|
<arxiv:comment>Accepted and presented at EvalMG 2026, the Second Workshop on Evaluation for Multimodal Generation, co-located with ACM SIGIR 2026</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Mukhtiar Ali</name>
|
|
</author>
|
|
<author>
|
|
<name>Harsh Dubey</name>
|
|
</author>
|
|
<author>
|
|
<name>Sugam Mishra</name>
|
|
</author>
|
|
<author>
|
|
<name>Chulwoo Pack</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2607.11798v1</id>
|
|
<title>StoryTeller: Training-Free Narrative Grounding for Long-Form Audio Description</title>
|
|
<updated>2026-07-13T16:50:03Z</updated>
|
|
<link href="https://arxiv.org/abs/2607.11798v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2607.11798v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long-form audio description (AD) requires more than describing visible actions: it must preserve characters, events, relationships, and story context across scenes so that blind and low-vision (BLV) audiences can follow a film. Modern video-language models (VLMs) are effective on short clips, but they often treat each moment independently, producing descriptions that miss who characters are, why events matter, and how the current scene connects to earlier narrative context. We propose StoryTeller, a training-free framework for story-aware long-form AD. Instead of relying only on local visual cues, StoryTeller maintains a verified narrative memory that carries forward story-relevant information across scenes, enabling later descriptions to remain coherent, grounded, and contextually informative. Given only raw video and a movie title, StoryTeller can optionally retrieve public movie metadata to resolve names and story context, while accepting only facts that are supported by the video through semantic filtering and VLM verification. The method requires no subtitles, scripts, AD transcripts, aligned captions, character banks, precomputed face identities, or task-specific fine-tuning. To evaluate whether generated AD preserves narrative information, we introduce StoryAD-QA, a question-answering benchmark that tests whether a language model can answer story-context questions using only the generated descriptions. Experiments on standard AD benchmarks and diverse long-form videos show that StoryTeller consistently improves narrative coherence, factual grounding, and story comprehension over strong baselines in automatic, QA-based, and human evaluations.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-07-13T16:50:03Z</published>
|
|
<arxiv:comment>Accepted to the European Conference on Computer Vision (ECCV) 2026</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Seung Hyun Hahm</name>
|
|
</author>
|
|
<author>
|
|
<name>Minh T. Dinh</name>
|
|
</author>
|
|
<author>
|
|
<name>SouYoung Jin</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2606.31245v1</id>
|
|
<title>HyperVLP: Enhancing Hierarchical Surgical Video-Language Pre-training in Hyperbolic Space</title>
|
|
<updated>2026-06-30T07:21:06Z</updated>
|
|
<link href="https://arxiv.org/abs/2606.31245v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2606.31245v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Surgical vision-language foundation models typically adopt educational materials, such as surgical lecture videos, to transfer surgical knowledge encoded in language into visual representations. These knowledge are multi-dimensional and hierarchical: fine-grained action cues appear in narration, mid-level key steps are summarized in subsection headings, and global procedural context, such as patient history and surgical strategy, is described in abstract texts. Prior work largely collapses these heterogeneous signals into a single flat embedding space, implicitly assuming independence across hierarchy levels. However, this is suboptimal because it ignores cross-level semantic containment, e.g., actions belong to steps, steps compose phases, weakens long-range dependency modeling. To this end, we propose a hyperbolic surgical video-language pre-training framework that explicitly preserves the hierarchical structure by mitigating structural false negatives induced by procedural context and enforcing semantic consistency between parent phases and their constituent child steps. Extensive experiments on multiple surgical benchmarks show consistent gains in zero- and few-shot phase recognition across procedures and institutions.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-06-30T07:21:06Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<arxiv:journal_ref>MICCAI2026</arxiv:journal_ref>
|
|
<author>
|
|
<name>Yaojun Hu</name>
|
|
</author>
|
|
<author>
|
|
<name>Kun Yuan</name>
|
|
</author>
|
|
<author>
|
|
<name>Nassir Navab</name>
|
|
</author>
|
|
<author>
|
|
<name>Haochao Ying</name>
|
|
</author>
|
|
<author>
|
|
<name>Jian Wu</name>
|
|
</author>
|
|
<author>
|
|
<name>Nicolas Padoy</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2606.17798v1</id>
|
|
<title>LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams</title>
|
|
<updated>2026-06-16T11:18:05Z</updated>
|
|
<link href="https://arxiv.org/abs/2606.17798v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2606.17798v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Despite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory. These obstacles undermine real-time responsiveness and cause severe forgetting throughout prolonged interactions. In this work, we introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams. The design of LiveStarPro rests on three complementary components. The first component is Streaming Verification Decoding (SVeD), an inference framework that identifies the appropriate response timing through single-pass perplexity verification, thereby eliminating the dependency on explicit silence tokens. The second component is Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams. The third component is Tree-Structured Hierarchical Memory (TSHM), a recursive memory architecture that organizes evicted historical information into event chains and consequently enables efficient retrieval from effectively unbounded video streams. To facilitate a comprehensive evaluation under realistic online conditions, we further present OmniStarPro, a large-scale benchmark that spans 15 diverse real-world scenarios and that extends to hour-scale streams for the assessment of long-term recall. Extensive experiments demonstrate that LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error, while its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching. The model and the code are publicly available at https://github.com/sotayang/LiveStarPro.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-06-16T11:18:05Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Zhenyu Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Kairui Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Bing Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Shengsheng Qian</name>
|
|
</author>
|
|
<author>
|
|
<name>Changsheng Xu</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.31069v1</id>
|
|
<title>Towards Effective Long-Video Event Prediction via Multi-Level Event Semantics Mining</title>
|
|
<updated>2026-05-29T09:38:43Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.31069v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.31069v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Accurately predicting future events is fundamental to content understanding and decision-making across various domains. While prior research has primarily focused on text or short-video scenarios, long-video event prediction, characterized by vast multimodal context and more complex narratives, remains underexplored. Meanwhile, although recent Long-Video Language Models (LVLMs), built on Large Language Models (LLMs) and Vision-Language Models (VLMs), have shown promise in long-video question answering and summarization, they struggle to generalize to event prediction, as they can neither precisely extract event-related details nor perform fine-grained analysis of event development. To address this gap, we propose VISTA, a multi-level event semantics mining framework for long-video event prediction. Initially, VISTA applies a character-centric visual prompt to precisely extract event-related visual details, enhancing detail-level semantics; subsequently, it employs a knowledge-enhanced iterative retrieval strategy, guiding the LLM to progressively construct logically coherent event chains, thereby improving event-level narratives; ultimately, VISTA adopts a human-like propose-then-retrieve strategy to generate diverse future-oriented proposals and integrate multi-level clues, producing robust and accurate predictions. Extensive experiments on real-world datasets validate the effectiveness of VISTA for long-video event prediction.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-29T09:38:43Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Bo Peng</name>
|
|
</author>
|
|
<author>
|
|
<name>YuanJie Lyu</name>
|
|
</author>
|
|
<author>
|
|
<name>PengGang Qin</name>
|
|
</author>
|
|
<author>
|
|
<name>Tong Xu</name>
|
|
</author>
|
|
<arxiv:doi>10.1007/978-981-95-6950-2</arxiv:doi>
|
|
<link rel="related" href="https://doi.org/10.1007/978-981-95-6950-2" title="doi"/>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.30750v1</id>
|
|
<title>SLAP: The Semantic Least Action Principle for Variational Video-Language Modeling</title>
|
|
<updated>2026-05-29T02:28:33Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.30750v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.30750v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>In the era of Large Video-Language Models (LVLMs), the computational necessity of sparse frame sampling creates a fundamental ``temporal gap'', rendering models blind to critical causal transitions. Existing solutions relying on generative hallucination (e.g., latent diffusion) or autoregressive extrapolation often fail to maintain semantic consistency over long horizons, suffering from object vanishing and energetic instability. We propose a paradigm shift from probabilistic generation to variational mechanics with the \textbf{Semantic Least Action Principle (SLAP)}. Drawing a rigorous isomorphism between classical mechanics and semantic dynamics, we model the latent video trajectory as a path on a Riemannian manifold governed by a Semantic Lagrangian. By formulating the interpolation task as a Boundary Value Problem (BVP) solved via the discrete Euler-Lagrange equations, SLAP naturally enforces object persistence without pixel-level rendering. Extensive experiments show the effectiveness of our proposed SLAP.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-29T02:28:33Z</published>
|
|
<arxiv:comment>Accepted by ICML 2026</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Xiang Fang</name>
|
|
</author>
|
|
<author>
|
|
<name>Wanlong Fang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.27318v2</id>
|
|
<title>Q-GeoMem: Question-Guided Geometric Memory for Video Spatial Reasoning</title>
|
|
<updated>2026-07-06T17:18:57Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.27318v2" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.27318v2" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video spatial reasoning requires accumulating viewpoint-dependent evidence over time while retaining information useful to the question being asked. Existing spatial video-language models improve geometric perception and long-range context modeling, but often treat memory as a generic temporal cache, which can introduce redundant or irrelevant evidence and weaken long-horizon reasoning. We propose Q-GeoMem, a question-guided geometric memory framework for video spatial reasoning. Q-GeoMem injects camera-conditioned geometry into visual tokens and maintains two complementary memories: a Fine-Grained Context Bank for recent dense features and camera states, and a Semantic-Geometric Evidence Bank for compact long-range evidence. For each candidate frame, a calibrated Q-Former estimates question relevance, while novelty and evidence utility are recomputed with respect to the active evidence bank. The resulting relevance-novelty utility controls capacity-based replacement and serves as an attention bias during memory reading. During reasoning, both memories are read before update and adaptively fused with the current frame representation. Extensive experiments across two in-domain and five out-of-distribution benchmarks, and controlled memory analyses show that Q-GeoMem achieves state-of-the-art performance in the evaluated settings and validate the effectiveness of question-guided geometric evidence selection.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-26T17:26:29Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Xianqiang Gao</name>
|
|
</author>
|
|
<author>
|
|
<name>Qizhi Chen</name>
|
|
</author>
|
|
<author>
|
|
<name>Delin Qu</name>
|
|
</author>
|
|
<author>
|
|
<name>Haoming Song</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhigang Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Bin Zhao</name>
|
|
</author>
|
|
<author>
|
|
<name>Dong Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Xuelong Li</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.20818v1</id>
|
|
<title>OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026</title>
|
|
<updated>2026-05-20T07:14:11Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.20818v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.20818v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>In this report, we present our champion solutions for the Natural Language Queries and GoalStep tracks of the Ego4D Episodic Memory Challenge at CVPR 2026. Both tracks require accurately localizing temporal segments from long untrimmed egocentric videos. To address these tasks, we propose a reranking-based framework that effectively leverages the strong video-language reasoning capability of multimodal large language model (MLLM) while preserving the efficiency and candidate recall of conventional localization pipelines. Specifically, we first obtain a set of candidate segments from existing localization model OSGNet, and then employ MLLM to select the segment that best matches the given query, thereby refining the final prediction. Ultimately, our method achieved first place in both the Natural Language Queries and GoalStep tracks. Our code can be found at https://github.com/iLearn-Lab/CVPR25-OSGNet.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-20T07:14:11Z</published>
|
|
<arxiv:comment>Champion solution for the Natural Language Queries and GoalStep tracks of the Ego4D Challenge at the CVPR EgoVis Workshop 2026</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Yisen Feng</name>
|
|
</author>
|
|
<author>
|
|
<name>Leigang Qu</name>
|
|
</author>
|
|
<author>
|
|
<name>Haoyu Zhang</name>
|
|
</author>
|
|
<author>
|
|
<name>Qiaohui Chu</name>
|
|
</author>
|
|
<author>
|
|
<name>Meng Liu</name>
|
|
</author>
|
|
<author>
|
|
<name>Xuemeng Song</name>
|
|
</author>
|
|
<author>
|
|
<name>Weili Guan</name>
|
|
</author>
|
|
<author>
|
|
<name>Liqiang Nie</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.08158v3</id>
|
|
<title>HIMMEL: Hierarchical Interleaved Multi-stream Motion Encoding for Long Video Understanding</title>
|
|
<updated>2026-10-15T09:04:56Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.08158v3" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.08158v3" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Long video understanding with multimodal language models suffers from three compounding bottlenecks: heavy decode cost to obtain dense RGB frames, token growth with frame count, and weak motion perception under sparse keyframe sampling. Existing remedies either prune visual tokens after decoding, which leaves the expensive RGB pipeline untouched, or they discretise codec motion vectors with a generative tokeniser, which couples motion modelling to a heavy pre-training stage. In this paper we present HIMMEL, a hierarchical video-language framework that allocates semantic and motion capacity along separate paths. A small set of sparse semantic anchor frames is routed to the expensive host ViT to ground object identity and scene layout, while the far denser inter-frame intervals are encoded by a lightweight compressed-domain tri-stream adapter that distils motion evidence from motion-vector maps, residual maps, and an I-frame context branch into aligned motion tokens. These tokens are injected into the LLM via a differentiable placeholder mechanism, after a dedicated Stage 1 contrastive alignment that places the motion representation in a geometry compatible with the frozen visual backbone. We further show that an InfoNCE alignment objective beats MSE regression by +1.5 pp. On Video-MME, HIMMEL surpasses the dense 32-frame baseline by +2.3 pp (61.2 to 63.5%) while using 3.6 times fewer context tokens and running end-to-end in 1.03 s per question instead of 2.75 s. Switching to a stronger Qwen3-VL-8B host pushes HIMMEL to 64.9%, narrowing the gap to much larger proprietary systems while keeping inference on a single NVIDIA H20 GPU. Extensive ablations across stream composition, motion-encoder family, fusion mode, alignment objective, anchor count, LoRA rank, and video duration confirm that the full tri-stream contributes to the observed gains, and that the benefit grows with video length.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-04T09:35:25Z</published>
|
|
<arxiv:comment>NeurIPS 2026 Main. Code: https://github.com/Sunbeam23333/HIMMEL</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Haopeng Jin</name>
|
|
</author>
|
|
<author>
|
|
<name>Wenlong Zhao</name>
|
|
</author>
|
|
<author>
|
|
<name>Tao Yu</name>
|
|
</author>
|
|
<author>
|
|
<name>Tiankun Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Zhenyu Guan</name>
|
|
</author>
|
|
<author>
|
|
<name>ShiQuan Dong</name>
|
|
</author>
|
|
<author>
|
|
<name>Yujia Yang</name>
|
|
</author>
|
|
<author>
|
|
<name>Shani Ye</name>
|
|
</author>
|
|
<author>
|
|
<name>Chubin Chen</name>
|
|
</author>
|
|
<author>
|
|
<name>Hongzhu Yi</name>
|
|
</author>
|
|
<author>
|
|
<name>Jinwen Luo</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2605.02262v1</id>
|
|
<title>WindowQuant: Mixed-Precision KV Cache Quantization based on Window-Level Similarity for VLMs Inference Optimization</title>
|
|
<updated>2026-05-04T06:17:13Z</updated>
|
|
<link href="https://arxiv.org/abs/2605.02262v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2605.02262v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Recently, video language models (VLMs) have been applied in various fields. However, the visual token sequence of the VLM is too long, which may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in VLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This paper introduces a novel approach called WindowQuant, which employs window-adaptive mixed-precision quantization to optimize the KV cache. WindowQuant consists of two modules: window-level quantization search and window-level KV cache computation. Window-level quantization search quickly determines the optimal bit-width configuration of the KV cache windows based on the similarity scores between the corresponding visual token windows and the text prompt, maintaining the model accuracy. Furthermore, window-level KV cache computation reorders the KV cache windows before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that WindowQuant outperforms state-of-the-art VLM models and KV cache quantization methods on various datasets.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.CL" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-05-04T06:17:13Z</published>
|
|
<arxiv:comment>Accepted to ACM Transactions on Architecture and Code Optimization (ACM TACO)</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Wei Tao</name>
|
|
</author>
|
|
<author>
|
|
<name>Xiaoyang Qu</name>
|
|
</author>
|
|
<author>
|
|
<name>Peiqiang Wang</name>
|
|
</author>
|
|
<author>
|
|
<name>Guokuan Li</name>
|
|
</author>
|
|
<author>
|
|
<name>Jiguang Wan</name>
|
|
</author>
|
|
<author>
|
|
<name>Kai Lu</name>
|
|
</author>
|
|
<author>
|
|
<name>Jianzong Wang</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2604.25886v3</id>
|
|
<title>MarkIt: Training-Free Visual Markers for Precise Video Temporal Grounding</title>
|
|
<updated>2026-06-16T05:05:15Z</updated>
|
|
<link href="https://arxiv.org/abs/2604.25886v3" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2604.25886v3" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video temporal grounding (VTG) aims to localize the start and end timestamps of the event described by a given query within an untrimmed video. Despite the strong open-world video understanding and recognition ability of video language large models (Vid-LLMs), outputting precise temporal grounding information remains challenging, since explicit temporal cues are scarce in untrimmed videos, and query-relevant entities are hard to track consistently across the video timeline. In this paper, we present \MarkIt{}, a training-free framework that transforms an input video into a query-conditioned marked video, which empowers Vid-LLMs to generate more reliable temporal localization predictions. The core component of \MarkIt{} is an annotation-free query-to-mask grounding bridge (Q2M-Bridge). Given a natural-language query, it automatically derives a compact set of canonical subject tags through linguistic parsing and normalization, then maps these tags to query-conditioned instance masks using text-conditioned open-vocabulary segmentation. The bridge also embeds lightweight semantic instance markers and a persistent frame index into each frame, effectively transforming long-range temporal reasoning into explicit visual cues for Vid-LLMs. \MarkIt{} adopts an inference-time plug-and-play design, needs no modifications to Vid-LLM weights, and is fully compatible with supervised fine-tuning. Experiments conducted on multiple mainstream moment retrieval and highlight detection benchmarks demonstrate that \MarkIt {} achieves state-of-the-art results, delivering consistent temporal grounding improvements across a wide range of existing models.</summary>
|
|
<category term="cs.MM" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-04-28T17:29:19Z</published>
|
|
<arxiv:primary_category term="cs.MM"/>
|
|
<author>
|
|
<name>Pengcheng Fang</name>
|
|
</author>
|
|
<author>
|
|
<name>Yuxia Chen</name>
|
|
</author>
|
|
<author>
|
|
<name>Xiaohao Cai</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2604.14074v1</id>
|
|
<title>Training-Free Semantic Multi-Object Tracking with Vision-Language Models</title>
|
|
<updated>2026-04-15T16:44:57Z</updated>
|
|
<link href="https://arxiv.org/abs/2604.14074v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2604.14074v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Semantic Multi-Object Tracking (SMOT) extends multi-object tracking with semantic outputs such as video summaries, instance-level captions, and interaction labels, aiming to move from trajectories to human-interpretable descriptions of dynamic scenes. Existing SMOT systems are trained end-to-end, coupling progress to expensive supervision, limiting the ability to rapidly adapt to new foundation models and new interactions. We propose TF-SMOT, a training-free SMOT pipeline that composes pretrained components for detection, mask-based tracking, and video-language generation. TF-SMOT combines D-FINE and the promptable SAM2 segmentation tracker to produce temporally consistent tracklets, uses contour grounding to generate video summaries and instance captions with InternVideo2.5, and aligns extracted interaction predicates to BenSMOT WordNet synsets via gloss-based semantic retrieval with LLM disambiguation. On BenSMOT, TF-SMOT achieves state-of-the-art tracking performance within the SMOT setting and improves summary and caption quality compared to prior art. Interaction recognition, however, remains challenging under strict exact-match evaluation on the fine-grained and long-tailed WordNet label space; our analysis and ablations indicate that semantic overlap and label granularity substantially affect measured performance.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-04-15T16:44:57Z</published>
|
|
<arxiv:comment>Accepted to the 20th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2026)</arxiv:comment>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Laurence Bonat</name>
|
|
</author>
|
|
<author>
|
|
<name>Francesco Tonini</name>
|
|
</author>
|
|
<author>
|
|
<name>Elisa Ricci</name>
|
|
</author>
|
|
<author>
|
|
<name>Lorenzo Vaquero</name>
|
|
</author>
|
|
</entry>
|
|
<entry>
|
|
<id>http://arxiv.org/abs/2603.24539v1</id>
|
|
<title>CliPPER: Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition</title>
|
|
<updated>2026-03-25T17:14:36Z</updated>
|
|
<link href="https://arxiv.org/abs/2603.24539v1" rel="alternate" type="text/html"/>
|
|
<link href="https://arxiv.org/pdf/2603.24539v1" rel="related" type="application/pdf" title="pdf"/>
|
|
<summary>Video-language foundation models have proven to be highly effective in zero-shot applications across a wide range of tasks. A particularly challenging area is the intraoperative surgical procedure domain, where labeled data is scarce, and precise temporal understanding is often required for complex downstream tasks. To address this challenge, we introduce CliPPER (Contextual Video-Language Pretraining on Long-form Intraoperative Surgical Procedures for Event Recognition), a novel video-language pretraining framework trained on surgical lecture videos. Our method is designed for fine-grained temporal video-text recognition and introduces several novel pretraining strategies to improve multimodal alignment in long-form surgical videos. Specifically, we propose Contextual Video-Text Contrastive Learning (VTC_CTX) and Clip Order Prediction (COP) pretraining objectives, both of which leverage temporal and contextual dependencies to enhance local video understanding. In addition, we incorporate a Cycle-Consistency Alignment over video-text matches within the same surgical video to enforce bidirectional consistency and improve overall representation coherence. Moreover, we introduce a more refined alignment loss, Frame-Text Matching (FTM), to improve the alignment between video frames and text. As a result, our model establishes a new state-of-the-art across multiple public surgical benchmarks, including zero-shot recognition of phases, steps, instruments, and triplets. The source code and pretraining captions can be found at https://github.com/CAMMA-public/CliPPER.</summary>
|
|
<category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
|
|
<category term="cs.AI" scheme="http://arxiv.org/schemas/atom"/>
|
|
<published>2026-03-25T17:14:36Z</published>
|
|
<arxiv:primary_category term="cs.CV"/>
|
|
<author>
|
|
<name>Florian Stilz</name>
|
|
</author>
|
|
<author>
|
|
<name>Vinkle Srivastav</name>
|
|
</author>
|
|
<author>
|
|
<name>Nassir Navab</name>
|
|
</author>
|
|
<author>
|
|
<name>Nicolas Padoy</name>
|
|
</author>
|
|
</entry>
|
|
</feed>
|