i-modal extraction, we introduce CLIP-Variance guided keyframe selection that combines query-semantic relevance with visual informativeness, ensuring selected frames are both relevant and information-rich. Given a user query, EC-RAG performs event-level localization to identify the most relevant events along with their temporal context, and gathers supporting evidence from associated modalities for answer generation.
We evaluate EC-RAG on Video-MME (Fu et al., 2025) , MLVU (Zhou et al., 2024) , and LongVideoBench (Wu et al., 2024) benchmarks. By applying EC-RAG to five different open-source LVLMs, we achieve an average performance improvement of 4% on Video-MME. Furthermore, compared with existing frame-level retrieval methods such as Video-RAG and TV-RAG, EC-RAG consistently demonstrates superior performance, validating the effectiveness of our event-chain-based approach. Our contributions are summarized as follows:
•
We propose EC-RAG, a novel framework that explicitly models temporal event chains for long video understanding. Unlike existing methods operating at frame or snippet level, EC-RAG captures inter-event relationships and enables coherent reasoning over extended temporal spans.
•
We introduce CLIP-Variance guided keyframe selection that jointly considers semantic relevance and visual informativeness, improving the quality of multi-modal information extraction for event construction.
•
We design a query-guided event localization mechanism that identifies relevant events within the temporal chain and aggregates multi-modal evidence, providing structured context for LVLM answer generation.
2. Related Work
2.1. Video Understanding with Large Language Models
The emergence of powerful LLMs has spurred efforts to develop comprehensive video-language systems. Early works like Video-ChatGPT (Maaz et al., 2024) and VideoChat (Li et al., 2025) adopt frame-wise feature extraction followed by temporal aggregation. Video-LLaVA (Lin et al., 2024a) introduces a unified projection to bridge visual and textual modalities, while LLaVA-NeXT-Video (Zhang et al., 2024d) further refines this through video-specific adaptation. These models perform well on short videos but struggle with longer content due to frame sampling constraints.
To address this limitation, recent methods explore expanding the context window for extended sequences. LongVA (Zhang et al., 2024b) and Long-LLaVA (Song et al., 2024) leverage pre-training on lengthy documents to develop transferable sequence modeling capabilities. INTP (Shang et al., 2024) reorganizes visual tokens to accommodate more information within fixed context limits. However, densely sampling frames quickly becomes computationally prohibitive, and the high redundancy in video data often limits the benefits of processing more frames.
2.2. LLM-based Agents for Video Understanding
An alternative paradigm employs LLMs as controllers to orchestrate specialized tools for video comprehension (Gupta and Kembhavi, 2023; Min et al., 2024; Surís et al., 2023; Wang et al., 2024; Zhang et al., 2024a). Representative methods such as VideoAgent (Fan et al., 2024) and MM-VID (Lin et al., 2023) enable dynamic interaction with video content through iterative retrieval and reasoning. While flexible, these approaches introduce substantial latency from multiple tool calls and often rely on proprietary APIs, limiting broader adoption in open-source settings.
2.3. Retrieval-Augmented Generation for Video
RAG-based methods retrieve relevant information from pre-constructed databases to augment video understanding. Video-RAG (Luo et al., 2024) extracts multimodal cues including speech transcripts and detected objects to build searchable knowledge bases. TV-RAG (Cao et al., 2025) enhances this with temporal-aware retrieval, while AdaVideoRAG (Xue et al., 2025) employs adaptive routing for queries of varying complexity. Current approaches, however, retrieve content at the frame or segment granularity without capturing the te