.5-pro (gemini-1.5-pro-002, Sep 2024) (Team et al., 2024), and Claude-3.5-sonnet (claude-3-5-sonnet-20241022) (Anthropic, 2024). Open-source models include LLaVA-Next-Video-7B (Zhang et al., 2024b), LLaVA-Next-Video-34B (Zhang et al., 2024b), LLaVA-OneVision-7B (Li et al., 2024a), LongVU-7B (Shen et al., 2024), Aria-28B (Li et al., 2024b), Qwen2.5VL-7B/32B (Bai et al., 2025), LongVA-7B (Zhang et al., 2024a) and InternVL2-8B (Chen et al., 2024). To establish a human baseline, we employ two annotators to answer 316 randomly selected questions (including causal 1-needle, 2-needle visual grounding and image description). Details about the input content, the chat template of VLMs can be found in Appendix Sec. C.
Most models we tested do not natively support inputs containing multiple video clips. As a workaround, we uniformly sample five frames from each clip and stack them vertically as one input image. This allows us to stay within the image input limit, which is 32 images for most models. Compared to other possible techniques we experimented with, this trick yields the best performance (see Appendix Sec. D).
To prevent the models from using sentence indices as shortcuts, we input the full narration along with only a subset of the corresponding video clips. The narration and video clips are provided separately. We avoid aligning video clips with sentences in the input to prevent the models from relying on text to locate video clips. We input all video clips from the cause clip to the effect clip, as well as a total of five clip before and after this span; the numbers of clip before and after are random. This setup prevents the cause clip from always being the first clip and the effect clip always the last, eliminating
shortcut learning based on location.
4.2 Main Results
We present the performance of various VLMs on the noncausal 1-needle questions, causal 1-needle questions, Visual Grounding (VG) 2-needle questions, and Image Description (ID) 2-needle questions in Tab. 3. We report the accuracy on each type of questions. For VG 2-needle questions, we separately compute the accuracy for Part 1, Part 2, and both parts answered correctly. In addition, we evaluate each model using both the original (forward) and reversed video clip order, and report the average of the two orders as the final result on VG 2-needle questions. This design is motivated by positional bias of the models, discussed in Sec. 4.3.
Figure 3: Comparison of performance of models on Part 2 of VG 2-needle questions (average order) and causal 1-needle questions. This is a fair comparison since both ask to retrieve the cause event. The dashed diagonal line represents equal performance across the two question types. Most points fall below the line, indicating that models are poorer at VG 2-needle questions than causal 1-needle questions. The size of the dots indicates the model size. The stars indicate proprietary models.
Figure 4: Performance on VG 2-needle questions as the distance between the two needle grows. We report the average-order performance for models and the forward-order performance for human annotators. The model performance declines whereas human performance stays mostly unchanged.
Causal Questions Are More Challenging Than Noncausal Questions.
From Tab. 3, we observe that most models perform substantially better on noncausal 1-needle questions than on causal ones. For instance, ChatGPT-4o and Qwen2.5-VL-32B achieve results that are 17.6% and 19.0% higher, respectively. This gap suggests that causal reasoning still poses a significant hurdle for current state-of-the-art models on long-video understanding.
Two-needle Questions Are More Challenging Than One-needle.
We compare the performance on
Part 2 of VG 2-needle questions (average order) and causal 1-needle questions, since both ask for the retrieval of the cause event and constitute a fair comparison. Most models perform worse on the 2-needle questions than the 1-needle questions. Fig. 3 visualizes