 on synthesized text and adopts a fixed adaptation configuration, without explicitly determining where video knowledge should be written within the backbone or how visual evidence should shape the resulting memory.
2.3 Test-Time Training
Test-time training adapts models on test inputs with auxiliary self-supervised objectives to improve robustness under distribution shifts (Sun et al., 2020; Liu et al., 2021; Gandelsman et al., 2022; Zhang et al., 2022).
Related fully test-time adaptation methods also update models directly from unlabeled test data, for example through entropy minimization (Wang et al., 2020; Wang et al., 2022; Niu et al., 2022; Niu et al., 2023).
Unlike these methods, MIT3 treats each long video as a test-time memory-writing instance, using the video once to construct adaptive parameters that answer subsequent text-only questions.
This formulation introduces two video-memory-specific requirements beyond standard test-time training: determining where video knowledge should be written and how visual evidence should shape the resulting memory.
MIT3 addresses them through causal tracing for video-specific adaptation locations and visual memory distillation for video-induced representation changes.
3 Method
3.1 Overview
Let VV be a long video, QQ a question about its content, and AA the corresponding answer.
A conventional video-language model ℳ\mathcal{M} performs video-conditioned question answering as
A∼pℳ​(A∣V,Q),A\sim p_{\mathcal{M}}(A\mid V,Q),
(1)
requiring VV, or representations derived from it, whenever a question is asked.
We instead write the knowledge of VV into compact adaptive parameters θV\theta_{V} and answer subsequent questions from text:
A∼pℳθV​(A∣Q).A\sim p_{\mathcal{M}_{\theta_{V}}}(A\mid Q).
(2)
We instantiate θV\theta_{V} as LoRA parameters (Hu et al., 2021) inserted into the attention sublayers of selected model layers, where each model layer corresponds to one Transformer block.
The backbone remains frozen, and each video is represented by its own lightweight adapter.
MIT3 internalizes each video through three stages, shown in Fig. 2.
First, a VLM constructs compact multi-scale descriptions and local and global question-answer pairs.
Second, delta-injection causal tracing estimates each model layer’s causal memory utility from a small set of temporally grounded local QA pairs and selects top-KK layers under a fixed adaptation budget.
Third, LoRA modules in selected layers are optimized with textual supervision and visual memory distillation, which trains the adapter to reproduce normalized video-induced representation changes.
These stages determine what knowledge to write, where to write it, and how visual evidence shapes the memory.
Figure 2: Overview of MIT3.
(a) A VLM constructs multi-scale descriptions, temporally grounded local QA, and global QA to cover video knowledge.
(b) Delta-injection causal tracing estimates each model layer’s causal memory utility and selects top-KK layers under a fixed adaptation budget.
(c) Visual memory distillation optimizes LoRA modules in selected layers to reproduce normalized video-induced representation changes, transferring visual evidence into text-only memory.
3.2 Video-Specific Training Data Generation
The first stage constructs compact, video-specific supervision that preserves both fine-grained evidence and longer temporal structure.
Following NKR (Guan et al., 2026), we adopt a multi-scale temporal partitioning paradigm. Specifically, our implementation segments VV into a temporal pyramid whose window durations double from 5 seconds (i.e., 5, 10, 20, 40, …\ldots seconds) up to the full video length.
Short-scale windows are non-overlapping, while longer-scale windows overlap to retain events crossing segment boundaries.
For each window, a VLM generates complementary low-, mid-, and high-level descriptions covering visible details, actions and transitions, and the overall content.
For windows up to 40 seconds, overlapping information across the 