nto the language-model context (Maaz et al., 2024; Zhang et al., 2023).
Sparse context can omit critical evidence, whereas dense context repeatedly incurs visual-token and memory costs, making efficient knowledge reuse for subsequent questions a central challenge for long-video QA.
Existing approaches improve the video-conditioned question-answering path through frame selection, token compression, sparse memories, redundancy reduction, and longer multimodal contexts (Song et al., 2024; Chen et al., 2025; Wang et al., 2024).
Retrieval-based methods complement these architectures by preprocessing videos into searchable memories and locating relevant clips for each question (Ren et al., 2026).
These strategies make long inputs more tractable, but their inference path remains video-dependent: each question still requires visual encoding, cached representations, retrieval, or inspection of relevant clips.
When one video must support many questions, as in lecture archives, meetings, surveillance review, or personalized video assistants, repeatedly reconstructing much of the same visual context becomes an avoidable systems bottleneck.
Video-free inference offers a complementary scaling axis for this setting: it writes a video’s knowledge into compact parameters once and answers subsequent questions from text alone.
Neural Knowledge Representation (NKR) (Guan et al., 2026) instantiates this idea by generating descriptions and question-answer pairs from a video and fitting a video-specific adapter (Hu et al., 2021).
Once fitted, the adapter acts as compact parametric memory for text-only question answering, a formulation contrasted with conventional video-dependent inference in Fig. 1.
Effective internalization therefore poses two knowledge-writing questions.
First, where should a particular video’s knowledge be written? Transformer layers play distinct computational roles, whereas existing adaptation configurations are typically fixed across videos.
Second, how should memory capture the effect of visual evidence? Synthetic descriptions provide factual supervision, while video-conditioned representations directly signal how seeing the video changes internal model computation.
To address these two questions, we propose Memory Internalization via Test-Time Training (MIT3), a test-time training framework for video-specific adaptive memory.
To determine where to write, MIT3 uses causal tracing to extract the hidden-state change induced by video input, inject this change at each model layer, and score the resulting improvement in answer likelihood.
Layers with the highest causal memory utility are selected as video-specific adaptive locations, where the frozen backbone most effectively converts video evidence into answer-relevant computation.
To determine how to write, visual memory distillation uses a frozen video-conditioned teacher to supply the representation change induced by the relevant clip, and trains the text-only adapter to reproduce that change from the question.
With compact multi-scale supervision, these mechanisms store video knowledge at effective computational sites while preserving the internal effect of visual evidence.
Figure 1: Comparison of video-conditioned question answering, NKR, and MIT3. Conventional video-language models encode or retrieve visual evidence per question. NKR trains a video-specific LoRA adapter on synthesized text using a fixed all-layer configuration. MIT3 instead selects video-specific model layers through causal tracing and distills the visual-evidence-induced representation change into these memory-writing locations, enabling video-free question answering.
To evaluate the effectiveness of MIT3, we conduct experiments on LVBench (Wang et al., 2025a), LongVideoBench (Wu et al., 2024), and Video-MME (Fu et al., 2025), spanning long-video and general video question answering across diverse durations.
On LVBench, MIT3 improves the matched NKR-style baseline by 6.8 points with Qwen3-VL-8B and by 5.7 