nt video segment and question, then with the question alone.
Let qiq_{i} denote the aligned positions of identical question tokens in both forward passes.
Let Hqi,lV​QH^{VQ}_{q_{i},l} and Hqi,lQH^{Q}_{q_{i},l} denote their hidden states at model layer ll under video-conditioned and question-only inference, respectively.
We define the video-induced representation change on these aligned question tokens as
Δ​Hqi,lV=Hqi,lV​Q−Hqi,lQ.\Delta H_{q_{i},l}^{V}=H^{VQ}_{q_{i},l}-H^{Q}_{q_{i},l}.
(4)
This delta captures how observing the relevant video segment changes the same question representations and provides the visual signal shared by causal localization and visual memory distillation.
Hidden-state intervention.
We next measure how effectively the computation after model layer ll can use the video-induced delta.
For an intervention strength λ∈[0,1]\lambda\in[0,1], we construct
H~qi,l​(λ)=Hqi,lQ+λ​Δ​Hqi,lV,λ∈[0,1],\widetilde{H}_{q_{i},l}(\lambda)=H^{Q}_{q_{i},l}+\lambda\Delta H_{q_{i},l}^{V},\qquad\lambda\in[0,1],
(5)
where small λ\lambda values test whether the remaining computation can exploit a limited video-induced change, while λ=1\lambda=1 injects the complete question-token delta at layer ll.
We replace the question-token states at layer ll with H~qi,l​(λ)\widetilde{H}_{q_{i},l}(\lambda) and continue question-only inference through subsequent model layers.
Let siQs_{i}^{Q} and si,l​(λ)s_{i,l}(\lambda) denote the answer log-likelihood before and after intervention:
siQ=log⁡pℳ​(Ai∣Qi),si,l​(λ)=log⁡pℳ​(Ai∣H~qi,l​(λ)).s_{i}^{Q}=\log p_{\mathcal{M}}(A_{i}\mid Q_{i}),\quad\quad\quad s_{i,l}(\lambda)=\log p_{\mathcal{M}}(A_{i}\mid\widetilde{H}_{q_{i},l}(\lambda)).
(6)
The intervention gain is
Gi,l​(λ)=si,l​(λ)−siQ.G_{i,l}(\lambda)=s_{i,l}(\lambda)-s_{i}^{Q}.
A larger gain indicates that computation after layer ll more effectively converts the injected video-induced change into evidence for the paired answer.
Budget-averaged causal memory utility.
The intervention curve Gi,l​(λ)G_{i,l}(\lambda) characterizes the causal memory utility of model layer ll as increasing amounts of the video-induced delta are injected.
A compact adapter benefits from layers where even a limited representation change produces answer-relevant gains, rather than only becoming useful near full intervention.
We therefore introduce b∈[0,1]b\in[0,1] as the maximum intervention strength under consideration.
For each bb, the prefix integral ∫0bGi,l​(λ)​𝑑λ\int_{0}^{b}G_{i,l}(\lambda)\,d\lambda measures the cumulative answer gain available within that intervention budget.
Averaging these prefix gains over all budgets gives
Ri,l=∫01∫0bGi,l​(λ)​dλ​db=∫01(1−λ)​Gi,l​(λ)​dλ.\displaystyle R_{i,l}=\int_{0}^{1}\int_{0}^{b}G_{i,l}(\lambda)\,d\lambda\,db=\int_{0}^{1}(1-\lambda)G_{i,l}(\lambda)\,d\lambda.
(7)
The equivalent weighted form emphasizes gains obtained from smaller interventions while retaining the utility accumulated across the full intervention trajectory.
A high Ri,lR_{i,l} therefore identifies a layer that responds to limited video-induced changes and continues to provide answer-relevant utility as more visual information is injected.
For each video VV, let 𝒞V\mathcal{C}_{V} denote a calibration set of temporally grounded triples (Vi,Qi,Ai)(V_{i},Q_{i},A_{i}) sampled across temporal scales.
Averaging the example-level scores gives the video-level causal memory utility of model layer ll:
Rl​(V)=1|𝒞V|​∑i∈𝒞VRi,l.R_{l}(V)=\frac{1}{|\mathcal{C}_{V}|}\sum_{i\in\mathcal{C}_{V}}R_{i,l}.
(8)
Under the fixed adaptation budget KK, we select the layers with the highest video-level utilities:
𝒮V=TopKlRl​(V).\mathcal{S}_{V}=\operatorname*{TopK}_{l}\;R_{l}(V).
(9)
The utility scores are computed once with the frozen model and without gradients before adapter optimization.
LoRA modules are inserted only into the attention sublayers of model layers in 𝒮V\mathcal{S}_{V}, while the remaining layers and all backbone parameters stay frozen.
This converts adapter placement from a fixe