et al., 2024) extracts multimodal cues including speech transcripts and detected objects to build searchable knowledge bases. TV-RAG (Cao et al., 2025) enhances this with temporal-aware retrieval, while AdaVideoRAG (Xue et al., 2025) employs adaptive routing for queries of varying complexity. Current approaches, however, retrieve content at the frame or segment granularity without capturing the temporal progression of events, limiting performance on questions requiring understanding of how actions unfold across time.
Figure 2. Overview of the EC-RAG framework. The pipeline comprises four stages: (i) Query Decoupling parses the user question into modality-specific retrieval requests for ASR, OCR, and DET; (ii) CLIP-Variance Guided Multimodal Extraction selects informative keyframes through weighted scoring of visual similarity and frame variance, then extracts speech transcripts, on-screen text, and scene graphs via open-vocabulary object detection; (iii) Temporal Event Chain Construction segments the video into 30-second units and builds a chain of semantically coherent events, where each event integrates its associated multimodal evidence with explicit temporal relations; (iv) Evidence-augmented Multimodal Reasoning performs query-guided event localization to retrieve relevant events, then feeds the event chain summary, located events, and raw evidence to the LVLM for answer generation. By organizing video content as structured event sequences, EC-RAG captures temporal dynamics and inter-event dependencies for complex video understanding.
3. Method
We propose EC-RAG, a training-free retrieval-augmented framework for large video-language models that organizes video content as structured event chains. As illustrated in Figure 2, our pipeline comprises four stages: (i) Query Decoupling: The user query is decomposed into modality-specific retrieval requests for ASR, OCR, and DET. (ii) CLIP-Variance Guided Multimodal Extraction: Informative keyframes are selected through joint CLIP similarity and variance weighting, followed by parallel extraction of speech, text, and object information. (iii) Temporal Event Chain Construction: The video is segmented into temporal units and organized as a chain of semantically coherent events with explicit inter-event relations. (iv) Evidence-augmented Multimodal Reasoning: Query-guided localization identifies relevant events, and the LVLM generates answers grounded in the event chain context and multimodal evidence.
Problem Formulation. Given a video VV and a user query QQ, a frame sampler extracts NN frames 𝐅={F1,F2,…,FN}\mathbf{F}=\{F_{1},F_{2},\ldots,F_{N}\}. These frames are encoded by a visual encoder (e.g., CLIP-L (Radford et al., 2021)) to obtain visual features 𝐅v\mathbf{F}_{v}. The standard LVLM inference can be formulated as:
(1)
O=LVLM​(𝐅v,Q)O=\text{LVLM}(\mathbf{F}_{v},Q)
However, this direct approach struggles with long videos where critical information may be sparse across extended durations. EC-RAG addresses this limitation by constructing an event chain (EC) ℰ\mathcal{E} that captures the temporal structure of video content, and performing query-guided event localization to retrieve relevant context before final reasoning:
(2)
O=LVLM​(𝐅v,Q,ℰ,𝐀)O=\text{LVLM}(\mathbf{F}_{v},Q,\mathcal{E},\mathbf{A})
where ℰ\mathcal{E} denotes the temporal event chain and 𝐀\mathbf{A} represents the retrieved multimodal evidence. This formulation enables the model to reason over both visual frames and structured textual context derived from the video.
3.1. Query Decoupling
Before processing the video content, we first analyze the user query to understand what types of information are needed to answer it. Motivated by recent methods (Lin et al., 2024a; Luo et al., 2024), this query decoupling step separates the retrieval planning from the actual video processing, allowing the system to focus computational resources on relevant modalities. Upon receiving a user query, EC-RAG decouples it into structured