-of-the-art models on long-video understanding.
Two-needle Questions Are More Challenging Than One-needle.
We compare the performance on
Part 2 of VG 2-needle questions (average order) and causal 1-needle questions, since both ask for the retrieval of the cause event and constitute a fair comparison. Most models perform worse on the 2-needle questions than the 1-needle questions. Fig. 3 visualizes this pattern.
This highlights the deficiency of using only 1-needle questions to evaluate long-video understanding.
Open-Source Models Exhibit Weaker World Modeling Ability.
According to Tab. 3, open-source models generally perform worse than proprietary models. Specifically, proprietary models surpass all open-source models on both 1-needle and ID 2-needle questions. For VG 2-needle questions (Avg), all open-source models, except for Qwen2.5VL-7B and Aria-28B, perform at or below random levels, falling behind proprietary models. This gap may be due to insufficient human-behavior training data during the pretraining of open-source VLMs.
Performance Decreases with Increasing Needle Distance.
In Fig. 4 (a) and (b), we present the VG 2-needle question performance (average order) for ChatGPT-4o and Gemini-1.5-pro. Performance is measured on questions with varying distances between the cause and effect events. The results show a clear decline as distance increases, especially when the evaluation aggregates answers from both parts. This phenomenon indicates that the distance between multiple needles has a significant impact when joint understanding of these needles is required. More details are provided in Appendix Sec.D.
A possible confounder is the strength of causal relations11
1
We thank an anonymous reviewer for suggesting this.. That is, long distances between needles may be correlated with weak causal relationships. It may be the weak causality, rather than long distance, that causes performance decrease. To verify this conjecture, we test if humans can identify the cause clip over long distances. Two annotators answered 113 visual grounding 2-needle questions and the results with different needle distances are detailed in Fig. 4 (c).
We can observe that for 24 samples with a needle distance between 6 and 8 clips, the human performance is 87.5% (Part 1), 87.5% (Part 2) and 87.5% (Both). This is even better than the performance on 97 samples with a distance between 3 and 5 clips: 83.5% (Part 1), 86.6% (Part 2) and 77.3% (Both). These results demonstrate that humans can easily undestand the causal relations. Thus, we argue that the model performance degradation mainly stems from the long distances between needles.
4.3 Pathological Behaviors of VLMs
Positional Bias.
As shown in Tab. 3, the accuracy of models responses varies depending on the positions of cause and effect clips. In VG 2-needle questions, Part 1 is grounded in the effect video clip, while Part 2 is grounded in the cause clip. In the forward-ordered clip sequence, the cause clip appears earlier, and all proprietary models show higher accuracy in Part 2 compared to Part 1; for instance, the accuracy of ChatGPT-4o exhibits an accuracy increase of 22.5%. This result appears paradoxical, since from an information-processing perspective answering the Part 2 question requires an understanding of the video clip required for the Part 1 question.
Why are models more accurate when locating the cause clip required by Part 2 than Part 1?
To answer this question, we reverse the order of the input video clips (the Reverse column of Tab. 3). In this setting, Part 1 achieves much higher accuracy than Part 2. This suggests that models pay more attention to clips appearing earlier; therefore, visual grounding for the cause clips is easier under normal playback, while reverse playback facilitates the visual grounding of the effect clips. To mitigate the influence of position bias, we evaluate model accuracy of VG 2-needle questions using the average of forward and reverse clip ordering.
Fig