n Causal2Needles
The videos in Causal2Needles are collected from the SyMoN (Sun et al., 2022) and YMS (Dogan et al., 2018) datasets, comprising movie recap videos and their corresponding narrations (98 movies from SyMoN and 94 from YMS). After integration, the videos in Causal2Needles have an average duration of 438 seconds, with 58.6 clips and 1,267.2 words per movie. The movie themes in our dataset are highly diverse. To quantify this diversity, we first use ChatGPT-4o to assign multi-labels to each narrative based on 11 predefined thematic categories, and then compute the frequency of each predicted label. The distribution is shown in Fig. 6 (a). As detailed in Sec. 3, we extract 747 cause-effect pairs from the narrations and construct 2-needle questions accordingly. We further analyze the distribution of their distances, defined as the number of clips between the cause and effect events, shown in Fig. 6 (b).
Appendix B Event Graph Merging
To obtain comprehensive and long-range causal relationships, we merge the extracted local and global event graphs. The merging process simply takes the union of the causal relations from different event graphs. For example, suppose the output of Graph A is [(1, 5), (3, 10), (21, 27)]. Here, (1, 5) denotes a causal relation from event 1 to event 5. Further suppose the output of Graph B is [(1, 5), (3, 10), (6, 11)]. The merged result is the union [(1, 5), (3, 10), (6, 11), (21, 27)].
Appendix C Evaluation Details
Computing Resources.
We conduct experiments using four NVIDIA GeForce RTX A6000 GPUs for all open-source models. For proprietary models, we obtain results through their publicly available APIs. The inference batch size is set to 1.
Details of VLMs.
Current video understanding VLMs primarily rely on converting videos into sequences of images for processing. For example, LLaVA-Next-Video (Zhang et al., 2024b) samples 32 frames from a video, encodes them into visual tokens using a Vision Transformer (ViT) (Dosovitskiy et al., 2020), and feeds these tokens, along with textual instructions, into Vicuna-1.5 (Zheng et al., 2023) to generate responses. Interleaved VLMs (Chen et al., 2024; Li et al., 2024b) extend input formats to support text-image sequences.
Proprietary models such as Gemini-Pro (Team et al., 2024), GPT-4o (OpenAI, 2024a), and Claude-3.5 (Anthropic, 2024) benefit from substantially greater computational resources, enabling them to process millions of image tokens—achieving dense visual understanding at up to 2 frames per second. These models also support interleaved text-image inputs, offering stronger temporal and contextual reasoning capabilities over video content.
Next, we show the input content and format we used when evaluating different models.
Proprietary LLMs.
For GPT-4o, GPT-4o-mini, Gemini-1.5-Pro-002, and Claude-Sonnet, the input consists of:
(1) A series of movie scene images, including the scene containing the correct answer. Each scene is represented by five video frames stitched together vertically.
(2) The task instruction, which specifies the task type—either visual grounding or image description. In the visual grounding task, the model is instructed to identify the correct scene number, whereas in the image description task, the model is asked to select the correct option.
(3) The movie’s narration background.
(4) The question. If the task is image description, the input will also include textual answer options.
Thus, each input is structured as shown in the list below:
<image> <image> <image> …
<text: Test Instruction>
<text: Movie Narration>
<text: Questions>
Open-source VLMs.
For LLaVA-Next-Video-7B, LLaVA-Next-Video-34B, LLaVA-OneVision-7B, LongVU-7B, and LongVA-7B, the input formats are the same as those used for the proprietary models. However, these models utilize different chat templates during generation.
We follow the examples presented on each model’s GitHub webpage or Hugging Face repository.
LLaVA-Next-Video-7B: https://huggingface.co/llav