>
Open-source VLMs.
For LLaVA-Next-Video-7B, LLaVA-Next-Video-34B, LLaVA-OneVision-7B, LongVU-7B, and LongVA-7B, the input formats are the same as those used for the proprietary models. However, these models utilize different chat templates during generation.
We follow the examples presented on each model’s GitHub webpage or Hugging Face repository.
LLaVA-Next-Video-7B: https://huggingface.co/llava-hf/LLaVA-NeXT-Video-7B-hf
LLaVA-Next-Video-34B: https://huggingface.co/llava-hf/LLaVA-NeXT-Video-34B-hf
LLaVA-OV-7B: https://huggingface.co/llava-hf/llava-onevision-qwen2-7b-ov-hf
LongVU-7B: https://huggingface.co/Vision-CAIR/LongVU_Qwen2_7B
LongVA-7B: https://github.com/EvolvingLMMs-Lab/LongVA
The remaining three are VLMs pretrained on interleaved image-text data. To match their input format, we insert scene numbers between video segments. However, we do not insert any text between scenes, as this could provide shortcuts by overly aligning with the narration, rather than requiring true visual understanding.
scene1<image>scene2 <image>scene3<image>
<text: Test Prompt>
<text: Movie Narration>
<text: Questions>
The chat template is collected from:
InternVL2: https://huggingface.co/OpenGVLab/InternVL2-8B
Aria-28B: https://huggingface.co/rhymes-ai/Aria
Qwen2.5VL-7B: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
Random Chance Calculation.
For a 1-needle question, the model is required to select the correct scene from a sequence of given scenes (including extra padding scenes). Assuming the total number of input scenes is NN, the random chance probability is 1/N1/N (reported as a percentage). For a visual grounding 2-needle question, where the model must make two separate selections, the random chance of answering both questions correctly is 1/N21/N^{2}.
Since the distance between the cause and effect scenes varies, the total number of input scenes NN differs for each question. Therefore, we compute the average input length N¯\bar{N} over each type of questions and determine the random chance probability. For 1-needle questions, the random chance probability is 9.8%. For visual grounding 2-needle questions, the probability of correctly identifying a single part is 9.8%, while the probability of getting both correct is only 1.0%. For image description 2-needle questions with four options, the random chance is 25.0%.
Appendix D Additional Experiments
In this section, we provide a deeper analysis of several aspects: (1) the influence of the distance between cause and effect events; (2) textual grounding versus visual grounding; (3) the uniqueness of answers to Part 2 questions; (4) existence of counterfactual answers in ID 2-needle questions; and (5) non-stacked versus stacked image input settings.
The Influence of Needle Distance.
Figure 7: The 2-needle visual grounding performance of various models on questions with different needle distances.
Fig. 4 (b) in Sec. 4.2 shows that the 2-needle visual grounding performance decreases as the distance between the cause and effect events increases, based on evaluations with two proprietary models. Here, we provide additional results in Fig. 7. These results indicate that needle distance significantly impacts model performance in multi-needle questions.
Textual vs. Visual Grounding.
To better understand the mechanisms underlying the models’ performance on VG 2-needle questions, we conduct additional experiments to analyze visual and textual grounding. We introduce a new task called Sentence Grounding (SG), which requires the model to select the sentence that contains the answer from the story narration text, which are part of the input of Causal2Needles. We compare the SG performance of GPT-4o on Part 1 and Part 2 questions, and summarize the results in Fig. 8. We make the following observations.
Figure 8: Performance of ChatGPT-4o on the sentence grounding (SG) task and the visual grounding (VG) task on the 2-needle questions. In the SG tasks, the sentences are always arranged in their natural