l
Given a base VideoLLM MM with limited temporal context capacity, a Recursive Video Language Summarization Model (RVLSM) is an inference-time scaffold around MM that treats the input video as part of an external environment while preserving the ability to densely process its temporal structure through recursive sub-calls. Given an input video
V={f1,f2,…,fT},V=\{f_{1},f_{2},\dots,f_{T}\},
where fif_{i} denotes the it​hi_{th} video frame and its associated audio stream, RVLSM interacts with a persistent REPL environment EE to generate an abstractive summary SS.
According to the RLM paradigm, instead of directly loading the entire video into the context window of a VideoLLM, RVLSM stores the video as an external object inside the REPL environment. The Root-LLM recursively decomposes the input video into semantically coherent temporal regions and delegates chunk-level summarization to sub-LLM calls.
Algorithm 1 describes the overall execution process. Given a video VV,
RVLSM initializes a REPL environment containing the input video together with functions
for scene-aware chunking, recursive VideoLLM querying, and multimodal tool execution.
The Root-LLM iteratively dispatches sub-calls to VideoLLM sub-models, collecting chunk-level summaries, and finally aggregates them into a single coherent abstractive summary.
Algorithm 1 Recursive Video Language Summarization Model
1:
Input video VV, query qq
2:
Final summary SS
3:
E←InitREPL⁡(V)E\leftarrow\mathrm{InitREPL}(V)
4:
E←AddFunctions⁡(E)E\leftarrow\mathrm{AddFunctions}(E)
5:
(c1,…,cN)←ChunkVideoByScenes⁡(V)(c_{1},\ldots,c_{N})\leftarrow\mathrm{ChunkVideoByScenes}(V)
6:
𝒮←[]\mathcal{S}\leftarrow[\,]
7:
for i=1,…,Ni=1,\ldots,N do
8:
if i=1i=1 then
9:
ri←EnrichedVideoQuery⁡(ci,q)r_{i}\leftarrow\mathrm{EnrichedVideoQuery(c_{i},q)}
10:
else
11:
ri←AdaptiveToolSelector⁡(si−1)r_{i}\leftarrow\mathrm{AdaptiveToolSelector}(s_{i-1})
12:
si←ri​(ci,q)s_{i}\leftarrow r_{i}(c_{i},q)
13:
𝒮←𝒮|[si]\mathcal{S}\leftarrow\mathcal{S}\,\|\,[s_{i}]
14:
S←RootLLM⁡(𝒮,q)S\leftarrow\mathrm{RootLLM}(\mathcal{S},q)
15:
return SS
RVLSM introduces two key modifications tailored to long-video understanding: scene-aware Semantic Temporal Decomposition (STD) and Adaptive Multimodal Tool Calling (AMTC).
Semantic Temporal Decomposition
Unlike standard fixed-duration chunking, the STD module performs an adaptive scene-based segmentation of the video.
Given an input video VV, the chunking module produces a sequence of chunks:
V={c1,c2,…,cN}V=\{c_{1},c_{2},\dots,c_{N}\}
where each chunk cic_{i} corresponds to a scene-consistent temporal segment. A minimum chunk duration of 30 seconds is enforced, ensuring that no segment is too short to carry meaningful semantic content. All chunks are
temporally disjoint, chronologically ordered, and exhaustive over the input video.
REPL Environment and Recursive Video Queries
RVLSM operates inside a persistent Python REPL environment containing the input video and a collection of executable functions. The environment exposes functions for retrieving video metadata, performing scene-aware semantic temporal decomposition, querying VideoLLMs, and extracting multimodal signals from video chunks. The Root-LLM interacts with the environment iteratively by generating executable Python code. Intermediate summaries and tool outputs are stored as variables inside the REPL, allowing subsequent recursive calls to access previously computed information.
For each chunk cic_{i}, RVLSM invokes a sub-VLLM:
si=ℳs​u​b​(ci,qi,ki),s_{i}=\mathcal{M}_{sub}(c_{i},q_{i},k_{i}),
where ℳs​u​b\mathcal{M}_{sub} denotes the sub-VLLM, qiq_{i} is the chunk-specific summarization query, and kik_{i} denotes optional multimodal information extracted
through external tools. Experiments focus on a single recursive level in which the Root-LLM delegates chunk-level summarization to VideoLLM sub-calls. The framework generalizes to deeper recursive hierarchies over temporal sub-events.
Adaptive Multimodal Tool Calling
To compl