key={Ft∣st≥τs}\mathbf{F}_{\text{key}}=\{F_{t}\mid s_{t}\geq\tau_{s}\}
where τs\tau_{s} is a score threshold, with constraints on minimum and maximum frame counts to ensure adequate coverage across the video duration.
ASR Extraction. Audio information provides crucial context that is often unavailable through visual analysis alone, including dialogue, narration, and background audio cues. Following recent retrieval-augmented video systems (Cao et al., 2025; Luo et al., 2024), we extract the audio track from the video and transcribe it using Whisper (Radford et al., 2023), a robust speech recognition model capable of handling diverse audio conditions. The resulting speech transcripts TasrT_{\text{asr}} are segmented by timestamps to preserve temporal alignment with video frames. These segments are then encoded using Contriever (Izacard et al., 2021), a dense retrieval encoder trained for semantic similarity matching, producing embeddings that capture the semantic content of each transcript segment. The embeddings are indexed in a FAISS (Johnson et al., 2019) database for efficient similarity search:
(7)
DBasr←FAISS𝐄asr=Contriever​(Tasr)\text{DB}_{\text{asr}}\xleftarrow{\text{FAISS}}\mathbf{E}_{\text{asr}}=\text{Contriever}(T_{\text{asr}})
OCR Extraction. On-screen text frequently contains valuable information such as titles, subtitles, labels, and other textual elements that complement visual and audio content. Current LVLMs often struggle to accurately recognize text within video frames, particularly for small or stylized fonts. To address this limitation, we employ EasyOCR (JaidedAI, 2023), a dedicated text recognition model, to extract on-screen text from each sampled frame. Since OCR models may produce noisy outputs, we apply a confidence threshold to filter low-quality detections:
(8)
Tocr={t∣t=EasyOCR(F),conf(t)>τocr}T_{\text{ocr}}=\{t\mid t=\text{EasyOCR}(F),\text{conf}(t)>\tau_{\text{ocr}}\}
This filtering step removes unreliable text detections that could introduce noise into the retrieval process. The filtered texts are encoded and stored in a separate FAISS index following the same procedure as ASR:
(9)
DBocr←FAISS𝐄ocr=Contriever​(Tocr)\text{DB}_{\text{ocr}}\xleftarrow{\text{FAISS}}\mathbf{E}_{\text{ocr}}=\text{Contriever}(T_{\text{ocr}})
Object Detection with Scene Graph. While LVLMs demonstrate strong capabilities in object recognition, they continue to face challenges in precise object counting, localization, and understanding spatial relationships between objects. These limitations often lead to hallucinations when answering questions that require fine-grained visual understanding. To provide more accurate object-level information, we employ APE (Shen et al., 2024), an open-vocabulary object detection model that accepts textual prompts specifying which objects to detect. For the selected keyframes 𝐅key\mathbf{F}_{\text{key}}, APE localizes relevant objects based on the detection request:
(10)
Dt=APE​(Ft,Rdet),Ft∈𝐅keyD_{t}=\text{APE}(F_{t},R_{\text{det}}),\quad F_{t}\in\mathbf{F}_{\text{key}}
The raw detection outputs consisting of object categories and bounding box coordinates are not directly interpretable by language models. To bridge this gap, inspired by scene graph representations in visual reasoning (Cao et al., 2025; Luo et al., 2024), we transform the detection results into semantic scene graph descriptions expressed in natural language. For each keyframe, we generate three complementary types of information: (1) Object Location AlocA_{\text{loc}} converts bounding box coordinates into natural language spatial descriptions (e.g., “left side of frame”, “upper right corner”), making positional information accessible to the language model; (2) Object Count AcntA_{\text{cnt}} aggregates detection results to provide accurate counts for each object category, addressing the counting limitations of LVLMs; (3) Object Relation ArelA_{\text{rel}} computes and describes relative spatial relationships between detected 