Memory Internalization via Test-Time Training for Long-Video Understanding

Video question answering requires models to ground language in visual entities, actions, temporal order, and causal relations. Early benchmarks progressively expanded this reasoning demand from short animated clips to more complex web videos and causal-temporal questions(Tapaswi et al., 2016; Lei et al., 2018; Xiao et al., 2021). Modern video-language models connect a visual encoder to an LLM through learned projection or cross-modal modules and use instruction tuning to support question answering (Maaz et al., 2024; Zhang et al., 2023; Lin et al., 2024; Jin et al., 2024). They answer by integrating sampled frames or visual tokens into the language-model context, making the quality and cost of reasoning depend on the visual information available for each question.

Long-video question answering further requires locating and combining evidence scattered over minutes or hours. Recent benchmarks evaluate long-range retrieval, temporal reasoning, and event comprehension in this setting (Mangalam et al., 2023; Fu et al., 2025; Wang et al., 2025a; Wu et al., 2024). Existing methods extend multimodal context capacity (Chen et al., 2025; Zhang et al., 2024), compress observations into sparse memories (Song et al., 2024; He et al., 2024), reduce temporal and knowledge redundancy (Wang et al., 2024; Ren et al., 2024), or retrieve relevant evidence from external multimodal indices (Ren et al., 2026). Although these approaches improve access to long-range evidence, answering each question still requires video tokens, cached representations, or retrieval. They therefore optimize access to video evidence rather than internalizing a video’s knowledge for subsequent text-only question answering.

A complementary direction internalizes a video’s knowledge into video-specific parameters, enabling video-free, text-only question answering after one-time per-video adaptation. NKR (Guan et al., 2026) synthesizes multi-scale descriptions and question-answer pairs and trains a video-specific adapter (Hu et al., 2021), demonstrating the feasibility of parameterized video memory. However, its internalization process primarily relies on synthesized text and adopts a fixed adaptation configuration, without explicitly determining where video knowledge should be written within the backbone or how visual evidence should shape the resulting memory.

Test-time training adapts models on test inputs with auxiliary self-supervised objectives to improve robustness under distribution shifts (Sun et al., 2020; Liu et al., 2021; Gandelsman et al., 2022; Zhang et al., 2022). Related fully test-time adaptation methods also update models directly from unlabeled test data, for example through entropy minimization (Wang et al., 2020; Wang et al., 2022; Niu et al., 2022; Niu et al., 2023). Unlike these methods, MIT³ treats each long video as a test-time memory-writing instance, using the video once to construct adaptive parameters that answer subsequent text-only questions. This formulation introduces two video-memory-specific requirements beyond standard test-time training: determining where video knowledge should be written and how visual evidence should shape the resulting memory. MIT³ addresses them through causal tracing for video-specific adaptation locations and visual memory distillation for video-induced representation changes.

Let $V$ be a long video, $Q$ a question about its content, and $A$ the corresponding answer. A conventional video-language model $\mathcal{M}$ performs video-conditioned question answering as

requiring $V$, or representations derived from it, whenever a question is asked. We instead write the knowledge of $V$ into compact adaptive parameters $\theta_{V}$ and answer subsequent questions from text:

We instantiate $\theta_{V}$ as LoRA parameters (Hu et al., 2021) inserted into the attention sublayers of selected model layers, where each model layer corresponds to one Transformer block. The backbone remains frozen, and each video is represented by its own lightweight adapter.

MIT³ internalizes each video through three stages, shown in Fig. 2. First, a VLM constructs compact multi-scale descriptions and local and global question-answer pairs. Second, delta-injection causal tracing estimates each model layer’s causal memory utility from a small set of temporally grounded local QA pairs and selects top-$K$ layers under a fixed adaptation budget. Third, LoRA modules in selected layers are optimized with textual supervision and visual memory distillation, which trains the adapter to reproduce normalized video-induced representation changes. These stages determine what knowledge to write, where to write it, and how visual evidence shapes the memory.

The first stage constructs compact, video-specific supervision that preserves both fine-grained evidence and longer temporal structure. Following NKR (Guan et al., 2026), we adopt a multi-scale temporal partitioning paradigm. Specifically, our implementation segments $V$ into a temporal pyramid whose window durations double from 5 seconds (i.e., 5, 10, 20, 40, $\ldots$ seconds) up to the full video length. Short-scale windows are non-overlapping, while longer-scale windows overlap to retain events crossing segment boundaries. For each window, a VLM generates complementary low-, mid-, and high-level descriptions covering visible details, actions and transitions, and the overall content. For windows up to 40 seconds, overlapping information across the three levels is fused into one coherent, redundancy-reduced description; for windows of 80 seconds or longer, the three descriptions are retained separately. Further details on overlap and related configurations are provided in Appendix A.

Using the generated descriptions, we construct local and global QA supervision without another video pass. Local QA pairs are generated from individual window descriptions and cover entities, actions, attributes, temporal order, and clip-level reasoning. Each local question includes the timestamp or temporal range of its source window, allowing the adapter to encode temporal position and order while identifying the supporting video segment for visual memory distillation in Section 3.4. At inference, questions retain the native format of each evaluation benchmark. To capture video-wide knowledge, we merge recurring entities and selected long-window descriptions into a global memory that summarizes the event timeline, state changes, and major stages of the video. This global memory conditions global-QA generation and is also included as one next-token prediction example. Each generation round selects a different group of focus clips and produces $N$ timestamp-free global QA pairs, cycling through the groups to distribute supervision across the video. These global questions target cross-clip retrieval and reasoning. The resulting descriptions and QA pairs form the video-specific training set:

During adapter optimization, $\mathcal{D}_{\mathrm{desc}}$ provides next-token prediction supervision, while $\mathcal{D}_{\mathrm{localQA}}$ and $\mathcal{D}_{\mathrm{globalQA}}$ provide fine-tuning examples. Temporally grounded local QA pairs associate each example with its supporting video segment for visual memory distillation, while description and global-QA examples remain text-only. Our description and entity construction follow the multi-level generation design of NKR (Guan et al., 2026), while local-QA generation adapts its evidence format to open-ended answering and the global-memory and full-video QA components provide video-wide supervision introduced by MIT³. Appendix B provides the corresponding construction details.

The second stage localizes where to write knowledge from $V$ in the frozen backbone. Adapting every model layer distributes memory parameters throughout the backbone, while a fixed subset assumes shared storage locations across videos. MIT³ instead uses delta-injection causal tracing to measure each model layer’s video-specific causal memory utility. For a candidate layer $l$, we ask: if the hidden-state change induced by the video is injected during question-only inference, how strongly can the remaining computation convert it into a higher likelihood for the paired answer? The resulting answer gain ranks model layers by utility as video-specific memory-writing locations.

Video-induced representations. For each calibration triple $(V_{i},Q_{i},A_{i})$, we run the frozen model with the relevant video segment and question, then with the question alone. Let $q_{i}$ denote the aligned positions of identical question tokens in both forward passes. Let $H^{VQ}_{q_{i},l}$ and $H^{Q}_{q_{i},l}$ denote their hidden states at model layer $l$ under video-conditioned and question-only inference, respectively. We define the video-induced representation change on these aligned question tokens as

This delta captures how observing the relevant video segment changes the same question representations and provides the visual signal shared by causal localization and visual memory distillation.

Hidden-state intervention. We next measure how effectively the computation after model layer $l$ can use the video-induced delta. For an intervention strength $\lambda\in[0,1]$, we construct

where small $\lambda$ values test whether the remaining computation can exploit a limited video-induced change, while $\lambda=1$ injects the complete question-token delta at layer $l$. We replace the question-token states at layer $l$ with $\widetilde{H}_{q_{i},l}(\lambda)$ and continue question-only inference through subsequent model layers. Let $s_{i}^{Q}$ and $s_{i,l}(\lambda)$ denote the answer log-likelihood before and after intervention:

The intervention gain is $G_{i,l}(\lambda)=s_{i,l}(\lambda)-s_{i}^{Q}.$ A larger gain indicates that computation after layer $l$ more effectively converts the injected video-induced change into evidence for the paired answer.

Budget-averaged causal memory utility. The intervention curve $G_{i,l}(\lambda)$ characterizes the causal memory utility of model layer $l$ as increasing amounts of the video-induced delta are injected. A compact adapter benefits from layers where even a limited representation change produces answer-relevant gains, rather than only becoming useful near full intervention. We therefore introduce $b\in[0,1]$ as the maximum intervention strength under consideration. For each $b$, the prefix integral $\int_{0}^{b}G_{i,l}(\lambda)\,d\lambda$ measures the cumulative answer gain available within that intervention budget. Averaging these prefix gains over all budgets gives

The equivalent weighted form emphasizes gains obtained from smaller interventions while retaining the utility accumulated across the full intervention trajectory. A high $R_{i,l}$ therefore identifies a layer that responds to limited video-induced changes and continues to provide answer-relevant utility as more visual information is injected.

For each video $V$, let $\mathcal{C}_{V}$ denote a calibration set of temporally grounded triples $(V_{i},Q_{i},A_{i})$ sampled across temporal scales. Averaging the example-level scores gives the video-level causal memory utility of model layer $l$:

Under the fixed adaptation budget $K$, we select the layers with the highest video-level utilities:

The utility scores are computed once with the frozen model and without gradients before adapter optimization. LoRA modules are inserted only into the attention sublayers of model layers in $\mathcal{S}_{V}$, while the remaining layers and all backbone parameters stay frozen. This converts adapter placement from a fixed architectural configuration into a video-specific memory-writing decision.

The third stage writes visual evidence into the selected memory locations by distilling how video conditioning redirects question representations. Synthesized descriptions provide explicit semantic supervision, while video-conditioned hidden states supply complementary signals about appearance, spatial configuration, and transient events. For each temporally grounded local QA example, the frozen model processes the relevant video segment and question as teacher, while the adapter-enabled student receives the same question without video input. Visual memory distillation trains the normalized adapter-induced changes to reproduce the normalized video-induced representation changes at the selected model layers $\mathcal{S}_{V}$. This transfers the internal computational effect of the relevant video segment into text-only memory without retaining frames or visual tokens.

Teacher and student deltas. At each selected model layer $l\in\mathcal{S}_{V}$, the frozen teacher provides the video-induced target defined in Equation 4:

The text-only student processes $Q_{i}$ with the video-specific adapter enabled. Let $H_{q_{i},l}^{\mathrm{LoRA}}$ denote its hidden states at the same aligned question-token positions. The adapter-induced change is

Both deltas are measured relative to the same question-only state $H_{q_{i},l}^{Q}$. This shared reference isolates the transformations contributed by the video and the adapter, making their representation changes directly comparable.

Representation alignment. We align the teacher and student deltas using cosine distance. For model layer $l\in\mathcal{S}_{V}$, the token-averaged loss is

where $\epsilon$ provides numerical stability. The per-example visual loss averages this reproduction objective over the selected model layers:

This loss is evaluated for the temporally grounded local QA examples used for visual distillation, while description and global-QA examples provide text-only NTP and SFT supervision.

For each video, we freeze the base VLM and optimize only LoRA parameters in attention sublayers of the causally selected model layers $\mathcal{S}_{V}$. Let $\mathcal{D}_{\mathrm{visual}}\subseteq\mathcal{D}_{\mathrm{localQA}}$ denote the temporally grounded local QA examples used for visual memory distillation. Their sample-averaged visual loss is

The complete test-time memory optimization objective is

where each term is averaged over its corresponding training subset and $\alpha$ controls the visual-distillation weight. After this one-time optimization, the resulting adapter $\theta_{V}$ serves as the video-specific memory. At inference, the frozen backbone receives only the question $Q$ with $\theta_{V}$ enabled, requiring no video input, visual tokens, or retrieval. The same adapter can answer subsequent questions about $V$ without reprocessing the video.

Table 1 compares MIT³ with published long-video systems, direct video-conditioned inference, and the matched NKR-style baseline. Under identical video-specific supervision, MIT³ consistently outperforms the NKR-style baseline across both backbone sizes and all four evaluation settings, demonstrating the effectiveness of video-specific layer selection and visual memory distillation for parameterized video memory. Despite performing video-free, text-only inference after one-time internalization, MIT³ remains close to direct video-conditioned inference on the full benchmarks, while requiring no video frames, visual tokens, retrieval, or video-dependent KV cache at query time. More importantly, with the 32B backbone, MIT³ reaches 50.1 on LVBench $>1$h and 55.9 on LongVideoBench-Long, surpassing direct video-conditioned inference on both long-video subsets. Together with the small variation between the full LVBench and its $>1$h subset, these results highlight the effectiveness and stability of MIT³ for long-video understanding. This makes MIT³ particularly suitable for scenarios that repeatedly revisit long videos, such as archived surveillance review, recorded meetings, and lecture or instructional video analysis. We further analyze the resource advantages of MIT³ over alternative inference paradigms in Appendix F.

We analyze the layer-wise causal utilities of Qwen3-VL-8B on LVBench to examine whether video-specific localization selects arbitrary layers or reveals a consistent organization of video memory. For each video, we normalize $R_{l}(V)$ across layers and identify the top-18 layers selected for adaptation. Figure 3(a) shows the resulting video-layer heatmap, while Figure 3(b) reports the selection frequency of each layer across videos.

The selected layers show substantial cross-video agreement. A compact group of layers is repeatedly assigned high causal memory utility, forming a shared memory-writing core that accounts for 65% of all top-18 selections. At the same time, a smaller set of selected layers varies across videos, introducing content-dependent refinements around this common structure. These results suggest that causal localization captures both a consistent cross-video organization of memory and video-specific adaptation needs, rather than reducing to either a fixed layer subset or arbitrary per-video selection. The corresponding analysis for Qwen3-VL-32B is provided in Appendix E.