ments, whereas EC-RAG localizes the query-relevant event through its event chain and aggregates ASR and DET evidence to arrive at the correct answer.A qualitative case study showing how EC-RAG correctly answers a temporal reasoning question by localizing the relevant event and aggregating multimodal evidence.
To quantify each component’s contribution, we ablate EC-RAG on Video-MME with LLaVA-Video-7B at 32 frames (Table 4).
Among all modules, ASR has the greatest impact: removing it causes the most significant performance degradation, particularly on medium and long videos, confirming that speech transcription is the most informative auxiliary modality for video QA.
The event chain module ranks second in importance, as disabling temporal localization leads to less targeted evidence retrieval.
CLIP-variance guided keyframe selection, OCR, and DET each provide meaningful but comparatively smaller gains, reflecting their complementary roles in selecting informative frames, capturing on-screen text, and grounding physical entities respectively.
Table 4. Component ablation of EC-RAG on Video-MME with LLaVA-Video-7B.
ASR
OCR
DET
EC
CV
Short
Medium
Long
Overall
×\times
✓
✓
✓
✓
74.8
60.3
54.7
63.3
✓
×\times
✓
✓
✓
76.5
62.8
59.9
66.4
✓
✓
×\times
✓
✓
77.1
62.9
59.2
66.4
✓
✓
✓
×\times
✓
77.0
61.6
58.4
65.7
✓
✓
✓
✓
×\times
76.4
63.5
58.9
66.3
✓
✓
✓
✓
✓
77.8
64.3
60.4
67.5
4.4. Qualitative Analysis
Figure 4 presents a representative case from Video-MME in which the baseline LLaVA-Video fails but EC-RAG succeeds.
The question asks what happened in the car after the heroine returned from shopping.
With 32 uniformly sampled frames, the baseline captures a frame showing the baby in a cheerful state and, lacking any temporal context to disambiguate it from other segments, incorrectly selects option C (“Baby happy”).
EC-RAG addresses this through its pipeline: the query decoupling stage extracts modality-specific retrieval cues (e.g., ASR keywords “car, shopping, unhappy” and DET targets “baby, car seat”).
The event chain then localizes the relevant temporal segment, whose ASR transcript explicitly states “He was just not happy in the car.”
Simultaneously, CLIP-variance guided keyframe selection identifies a visually informative frame that is forwarded to the object detector, revealing a baby being fed with a recently-cried expression.
Equipped with this temporally grounded multimodal evidence, the model correctly selects option D (“Baby unhappy”).
Figure 3 further illustrates the effect of EC-RAG through Grad-CAM heatmaps and t-SNE projections.
The Grad-CAM visualizations compare the attention distribution of the baseline and EC-RAG on both a query-relevant and a query-irrelevant frame.
With the supplementary textual evidence provided by EC-RAG, the model’s attention concentrates more tightly on the semantically relevant regions (e.g., the baby and car seat), whereas the baseline distributes attention more diffusely.
The t-SNE plot projects the query, vision, and text features into a shared 2D space, showing that EC-RAG’s auxiliary tokens pull the vision and query representations closer together, resulting in more aligned cross-modal features that facilitate accurate reasoning.
5. Conclusion
We presented EC-RAG, a training-free retrieval-augmented generation framework that structures long-video understanding around temporal event chains. By decomposing videos into semantically coherent events through the fusion of ASR, OCR, and open-vocabulary detection, and localizing query-relevant segments before injecting multimodal evidence into the LVLM prompt, EC-RAG consistently improves diverse backbones across three benchmarks—Video-MME, MLVU, and LongVideoBench—outperforming existing RAG-based methods under every tested setting. Notably, with the 72B LLaVA-Video backbone, EC-RAG surpasses both GPT-4o and Gemini-1.5-Pro on Video-MME using a fully open-source pipeline, demonstrating that structured event-level retrieval enables open-source models to ri