etrieval to identify relevant events and gather supporting evidence for answer generation.
Query-guided Event Localization. Not all events in a video are relevant to a given query. We identify the most relevant events through semantic similarity matching between the query and event descriptions. Each event description is encoded using Contriever and compared against the query embedding using cosine similarity:
(15)
simi=cos(Contriever(Q),Contriever(ei.desc))\text{sim}_{i}=\cos\bigl(\text{Contriever}(Q),\;\text{Contriever}(e_{i}.\text{desc})\bigr)
This text-to-text matching is more appropriate than image-text matching (e.g., CLIP) since both the query and event descriptions are textual. Events exceeding a relevance threshold τloc\tau_{\text{loc}} are selected as located events:
(16)
ℰloc={ei∣simi>τloc}\mathcal{E}_{\text{loc}}=\{e_{i}\mid\text{sim}_{i}>\tau_{\text{loc}}\}
The located events represent the temporal regions most likely to contain information needed to answer the query.
Evidence Retrieval. While event descriptions provide high-level summaries, detailed evidence from the original multimodal sources may be necessary for accurate reasoning. For each located event ei∈ℰloce_{i}\in\mathcal{E}_{\text{loc}}, we retrieve the associated multimodal evidence from the constructed databases. Using the retrieval requests 𝐑\mathbf{R} generated during query decoupling, we query the ASR and OCR databases to obtain relevant text segments:
(17)
Aasr=Retrieve​(DBasr,Rasr,τ),Aocr=Retrieve​(DBocr,Rocr,τ)A_{\text{asr}}=\text{Retrieve}(\text{DB}_{\text{asr}},R_{\text{asr}},\tau),\quad A_{\text{ocr}}=\text{Retrieve}(\text{DB}_{\text{ocr}},R_{\text{ocr}},\tau)
The DET evidence is collected from the scene graphs of keyframes falling within the located events’ time ranges:
(18)
Adet={Tdet(t)∣Ft∈𝐅key,t∈ei.span,ei∈ℰloc}A_{\text{det}}=\{T_{\text{det}}^{(t)}\mid F_{t}\in\mathbf{F}_{\text{key}},\;t\in e_{i}.\text{span},\;e_{i}\in\mathcal{E}_{\text{loc}}\}
where the retrieval function returns all database entries with similarity scores exceeding threshold τ\tau. The DET evidence AdetA_{\text{det}} is directly obtained from the scene graph descriptions of keyframes within the located events’ time ranges, providing detailed object-level information for visual grounding.
Context Construction and Answer Generation. The final reasoning step integrates multiple levels of context to provide the LVLM with comprehensive information. The context combines three complementary components: (1) Event Chain Summary provides a global overview of all events in the video, giving the model awareness of the full narrative arc even for content not directly relevant to the query; (2) Located Events contain detailed descriptions of query-relevant events along with their temporal predecessors, enabling understanding of causal relationships and temporal context; (3) Multimodal Evidence includes the raw ASR, OCR, and DET information retrieved for the located events, providing specific details that support accurate reasoning. The complete context is assembled as:
(19)
C=Concat​(Summary​(ℰ),Detail​(ℰloc),Aasr,Aocr,Adet)C=\text{Concat}(\text{Summary}(\mathcal{E}),\text{Detail}(\mathcal{E}_{\text{loc}}),A_{\text{asr}},A_{\text{ocr}},A_{\text{det}})
Finally, the LVLM generates the answer by jointly reasoning over the video frames, user query, and constructed context:
(20)
O=LVLM​(𝐅v,Q,C)O=\text{LVLM}(\mathbf{F}_{v},Q,C)
Table 1. Performance on the Video-MME (Fu et al., 2025) benchmark. #Text indicates the volume of retrieved textual context. EC-RAG is integrated into four open-source LVLMs spanning 8–32 input frames.
Model
#Text
LLM Params
Frames
Short
Medium
Long
Overall
Gain
Proprietary LVLMs
GPT-4o (OpenAI, 2024)
–
–
384
80.0
70.3
65.3
71.9
–
Gemini-1.5-Pro (Team et al., 2024)
–
–
0.5 fps
81.7
74.3
67.4
75.0
–
Open-Source LVLMs
Video-LLaVA (Lin et al., 2024a)
–
7B
8
44.6
38.3
35.8
39.6
–
Video-LLaVA + EC-RAG
2.0K
7B
8
50.1
44.6
43.4
46.0
+6.4
LLaVA-NeXT-Video (Zhang et al., 2024