 we first analyze the user query to understand what types of information are needed to answer it. Motivated by recent methods (Lin et al., 2024a; Luo et al., 2024), this query decoupling step separates the retrieval planning from the actual video processing, allowing the system to focus computational resources on relevant modalities. Upon receiving a user query, EC-RAG decouples it into structured retrieval requests without accessing video frames. The LVLM analyzes the query to determine what information sources are needed:
(3)
𝐑=LVLM​(Pd,Q)={Rasr,Rocr,Rdet}\mathbf{R}=\text{LVLM}(P_{d},Q)=\{R_{\text{asr}},R_{\text{ocr}},R_{\text{det}}\}
where PdP_{d} is the decoupling prompt enhanced with few-shot examples that demonstrate how to extract retrieval targets from various question types. Each request may be NULL if the corresponding modality is unnecessary for the given query. Specifically, RasrR_{\text{asr}} specifies key phrases to retrieve from speech transcripts, which is particularly useful for questions about dialogue content or narration. RocrR_{\text{ocr}} defines keywords to search in on-screen text, targeting questions about titles, labels, or displayed information. RdetR_{\text{det}} indicates physical objects to detect in video frames, essential for visual grounding questions.
The decoupling prompt guides the LVLM to identify concrete retrieval targets through structured examples. For detection requests, we enforce that RdetR_{\text{det}} contains only physical entities rather than abstract concepts, ensuring compatibility with vision-based detection models. This constraint prevents the system from attempting to detect non-visual concepts like emotions or intentions.
3.2. CLIP-Variance Guided Multimodal Extraction
This stage selects informative keyframes and extracts multimodal information from the video in parallel.
Keyframe Selection. A critical challenge in video understanding is selecting which frames to process for detailed analysis. Uniform sampling may miss important moments, while processing all frames is computationally prohibitive. Rather than uniformly sampling frames or relying solely on semantic similarity, inspired by query-guided frame selection in prior work (Cao et al., 2025; Luo et al., 2024), we propose a CLIP-variance (CV) guided selection mechanism that jointly considers query relevance and visual informativeness. The intuition is that informative frames should both relate to the query content and contain rich visual details rather than static or redundant imagery. For each frame FtF_{t}, we compute a weighted score:
(4)
st=simCLIP​(Rdet,Ft)⋅σts_{t}=\text{sim}_{\text{CLIP}}(R_{\text{det}},F_{t})\cdot\sigma_{t}
where simCLIP​(⋅)\text{sim}_{\text{CLIP}}(\cdot) denotes the CLIP similarity between the detection query and frame content. This term ensures that selected frames are semantically relevant to the objects or entities mentioned in the query. The second factor σt\sigma_{t} is the normalized visual variance of frame FtF_{t}:
(5)
σt=Var​(Ft)maxj⁡Var​(Fj)\sigma_{t}=\frac{\text{Var}(F_{t})}{\max_{j}\text{Var}(F_{j})}
The variance term captures the visual complexity of each frame, computed from the pixel intensity distribution. Frames with higher variance typically contain more visual structure and details, while low-variance frames often correspond to static backgrounds, black screens, or uniform scenes that provide limited information. By multiplying these two factors, we favor frames that are both query-relevant and visually informative. Keyframes are selected by thresholding on the weighted scores:
(6)
𝐅key={Ft∣st≥τs}\mathbf{F}_{\text{key}}=\{F_{t}\mid s_{t}\geq\tau_{s}\}
where τs\tau_{s} is a score threshold, with constraints on minimum and maximum frame counts to ensure adequate coverage across the video duration.
ASR Extraction. Audio information provides crucial context that is often unavailable through visual analysis alone, including dialogue, narration, and background audio cues. Following