criptions (e.g., “left side of frame”, “upper right corner”), making positional information accessible to the language model; (2) Object Count AcntA_{\text{cnt}} aggregates detection results to provide accurate counts for each object category, addressing the counting limitations of LVLMs; (3) Object Relation ArelA_{\text{rel}} computes and describes relative spatial relationships between detected objects (e.g., “person A is to the left of person B”). The complete scene graph representation combines these components:
(11)
Tdet=SceneGraph​(Aloc,Acnt,Arel)T_{\text{det}}=\text{SceneGraph}(A_{\text{loc}},A_{\text{cnt}},A_{\text{rel}})
3.3. Temporal Event Chain Construction
Unlike frame-level retrieval methods that treat video as a collection of independent images, EC-RAG explicitly models the temporal structure of video content by constructing an event chain. This representation captures how events unfold over time and enables reasoning about temporal relationships.
Video Segmentation. Long videos contain extended sequences of content that cannot be processed as a single unit. We partition the video into non-overlapping temporal segments of fixed duration Δ\Delta:
(12)
𝒮={Si=[ti,ti+Δ)∣i=0,1,…,⌊L/Δ⌋}\mathcal{S}=\{S_{i}=[t_{i},t_{i}+\Delta)\mid i=0,1,\ldots,\lfloor L/\Delta\rfloor\}
where LL is the total video duration. Each segment SiS_{i} serves as the basic unit for event extraction and is associated with the multimodal information (ASR, OCR, DET) extracted from frames within its time range. This segmentation balances granularity and efficiency: segments should be short enough to capture coherent events while being long enough to contain meaningful content.
Event Extraction. Raw multimodal data from each segment needs to be synthesized into a coherent semantic representation. For each segment SiS_{i}, we aggregate the corresponding ASR, OCR, and DET information and prompt the LVLM to extract a semantic event description that summarizes what happens in that temporal window:
(13)
ei=LVLM​(Pe,Tasr(i),Tocr(i),Tdet(i))e_{i}=\text{LVLM}(P_{e},T_{\text{asr}}^{(i)},T_{\text{ocr}}^{(i)},T_{\text{det}}^{(i)})
where PeP_{e} is the event extraction prompt that instructs the model to integrate information across modalities. The prompt guides the LVLM to focus on the main action or occurrence while considering all available evidence. Each extracted event eie_{i} comprises three components: a description that concisely summarizes what happens in this segment by integrating speech, text, and visual information; a type that classifies the event as action, dialogue, scene transition, or informational content; and a list of participants identifying key entities (people, objects) involved in the event.
Event Chain Assembly. Individual events are connected to form a coherent narrative structure. The extracted events are organized into a temporal event chain ℰ={e0,e1,…,en}\mathcal{E}=\{e_{0},e_{1},\ldots,e_{n}\} with explicit temporal ordering that reflects the video’s chronological structure. We establish predecessor-successor relations between adjacent events:
(14)
ei→temporalei+1,∀i∈[0,n−1]e_{i}\xrightarrow{\text{temporal}}e_{i+1},\quad\forall i\in[0,n-1]
The complete event chain provides a structured overview of the video’s narrative progression, capturing how events unfold and connect across time. This representation enables the model to understand not just individual moments but also the broader context and temporal dynamics of the video content.
3.4. Evidence-augmented Multimodal Reasoning
With the event chain constructed, EC-RAG performs query-guided retrieval to identify relevant events and gather supporting evidence for answer generation.
Query-guided Event Localization. Not all events in a video are relevant to a given query. We identify the most relevant events through semantic similarity matching between the query and event descriptions. Each event description is encoded using Contriever and compared against the query embedding using cosin