 sub-VLLM, qiq_{i} is the chunk-specific summarization query, and kik_{i} denotes optional multimodal information extracted
through external tools. Experiments focus on a single recursive level in which the Root-LLM delegates chunk-level summarization to VideoLLM sub-calls. The framework generalizes to deeper recursive hierarchies over temporal sub-events.
Adaptive Multimodal Tool Calling
To complement recursive calls to VideoLLMs, RVLSM augments chunk-level analysis through external tools specialized for video understanding.
The REPL environment provides access to auxiliary multimodal analysis functions that extract semantic signals from each video chunk.
Specifically, RVLSM can invoke object detection, action recognition, automatic speech recognition (ASR), and Optical Character Recognition (OCR). The extracted knowledge is injected into the VideoLLM sub-model prompt alongside the video chunk, providing structured perceptual grounding that enhances the quality of the generated chunk-level summary.
Algorithm 2 AdaptiveToolSelector
1:
Previous chunk summary si−1s_{i-1}
2:
Video Query function rir_{i}
3:
𝒞←{actions,speech,text,objects}\mathcal{C}\leftarrow\{\text{actions},\text{speech},\text{text},\text{objects}\} ⊳\triangleright signal categories
4:
Rc←R_{c}\leftarrow predefined keyword root set, for each c∈𝒞c\in\mathcal{C}
5:
SingleModalityQuery←{\mathrm{SingleModalityQuery}\leftarrow\{
6:
actions→ActionQuery​(action recognition),\text{actions}\to\mathrm{ActionQuery}\ \text{(action recognition)},
7:
speech→ASRQuery​(automatic speech recognition),\text{speech}\to\mathrm{ASRQuery}\ \text{(automatic speech recognition)},
8:
text→OCRQuery​(optical character recognition),\text{text}\to\mathrm{OCRQuery}\ \text{(optical character recognition)},
9:
objects→ObjectQuery​(object detection)\text{objects}\to\mathrm{ObjectQuery}\ \text{(object detection)}
10:
}\}
11:
for c∈𝒞c\in\mathcal{C} do
12:
raw​[c]←|{r∈Rc:r⊆si−1}|\text{raw}[c]\leftarrow|\{r\in R_{c}:r\subseteq s_{i-1}\}| ⊳\triangleright keyword matches
13:
scores​[c]←raw​[c]/|Rc|\text{scores}[c]\leftarrow\text{raw}[c]\,/\,|R_{c}|
14:
total←∑c∈𝒞scores​[c]\text{total}\leftarrow\sum_{c\in\mathcal{C}}\text{scores}[c]
15:
top←arg⁡maxc∈𝒞​scores​[c]\text{top}\leftarrow\arg\max_{c\in\mathcal{C}}\text{scores}[c]
16:
if total=0\text{total}=0 or scores​[top]<0.05\text{scores}[\text{top}]<0.05 then
17:
return VideoQuery\mathrm{VideoQuery} ⊳\triangleright no specific category
18:
else if scores​[top]<0.6⋅total\text{scores}[\text{top}]<0.6\cdot\text{total} then
19:
return EnrichedVideoQuery\mathrm{EnrichedVideoQuery} ⊳\triangleright no dominant category
20:
else
21:
return SingleModalityQuery⁡(top)\mathrm{SingleModalityQuery}(\text{top}) ⊳\triangleright dominant category
Unlike static multimodal augmentation pipelines, RVLSM performs heuristic adaptive tool selection. The first chunk is analyzed using full multimodal extraction in order to establish an initial semantic context. For subsequent chunks the Root-LLM selects the appropriate analysis configuration based on the previous chunk summary:
ri=ℛ⁡(si−1),r_{i}=\mathcal{R}(s_{i-1}),
where ℛ\mathcal{R} denotes the routing mechanism.
The procedure described in Algorithm 2 focuses on the AdaptiveToolSelector. Each perceptual signal is associated with one of four categories (actions, speech,text, and objects) corresponding respectively to action recognition, automatic speech recognition, optical character recognition, and object detection. Every category is paired with a predefined set of representative keywords (its root set), against which the previous chunk summary si−1s_{i-1} is matched. A normalized keyword-matching score is then computed for each category by dividing the raw number of matches by the size of the corresponding keyword set, so that categories with larger keyword sets are not favored.
The selector defaults to VideoQuery when no signal is detected, to EnrichedVideoQuery when no category dominates, and to the corresponding SingleModa