 than Part 2. This suggests that models pay more attention to clips appearing earlier; therefore, visual grounding for the cause clips is easier under normal playback, while reverse playback facilitates the visual grounding of the effect clips. To mitigate the influence of position bias, we evaluate model accuracy of VG 2-needle questions using the average of forward and reverse clip ordering.
Figure 5: The answer distribution of various models in the forward evaluation of visual grounding 2-needle questions. GT denotes ground truth. None means no clip number is output. Predictions of opensource models are heavily concentrated in a few numbers, exhibiting significant bias.
Static Output Bias.
In Tab. 3, we observe that some open-source VLMs achieve extremely low scores, sometimes zero, on VG 2-needle questions. When examined closely, we observe that some models tend to output the same prediction regardless of the question content. We call this phenomenon the static output bias.
In Fig. 5, we compare the output distributions of well-performing and poorly-performing models on VG 2-needle questions.
For example, the ground truth of Part 1 is mainly in clip 5 and beyond, as the effect clip appears after the cause. ChatGPT-4o shows a balanced distribution, aligning with its strong performance in Tab. 3. In contrast, the three open-source models predominantly select clip 1 as the answer, demonstrating severe output bias.
We experiment with different prompting strategies as attempts to fix this bias, but the models consistently produce fixed responses, indicating an inherent flaw in these models. The test prompts and output patterns are in Appendix Sec. F.
Table 4: Results of models on different input settings. Numbers in parentheses indicate the absolute increase over random chance.
1-N: 1-needle, 2-N: 2-needle, V: Video, N: Narration, Q: Question. When V is removed in causal 1-N, performance decreases to near chance level, demonstrating minimal textual bias. When V+N are removed in ID 2-N, performances stay above chance level but are still far from saturation.
Models
Causal 1-N Questions
ID 2-N Questions
V+N+Q
N+Q
V+N+Q
Q only
Proprietary Models
Gemini-1.5-pro
35.6 (+25.7)
12.3 (+2.4)
60.9 (+35.9)
39.0 (+14.0)
Claude-3-5-sonnet
26.5 (+16.7)
10.7 (+0.9)
60.5 (+35.5)
29.2 (+4.2)
ChatGPT-4o
39.2 (+29.4)
3.8 (-6.0)
59.2 (+34.2)
47.5 (+22.5)
Open-source Models
LongVA-7B
14.7 (+4.9)
11.2 (+1.4)
49.7 (+24.7)
38.4 (+13.4)
InternVL2-8B
7.4 (-2.4)
11.7 (+1.9)
40.2 (+15.2)
25.9 (+0.9)
LLaVA-Next-Video-7B
17.2 (+7.4)
0.3 (-9.5)
27.3 (+2.3)
26.9 (+1.9)
4.4 Analysis of Dataset Bias
Textual Bias.
Textual bias refers to the extent that the dataset allows models to derive the correct answer from the accompanying text alone.
To verify if our visual grounding test format effectively avoids textual bias, we test various models on causal 1-needle questions without providing the video clips. The results are presented in the first two columns of Tab. 4. After removing video input, model performance drops significantly to around random chance (9.8%) or lower. For instance, the performance of ChatGPT-4o drops from 39.2% to 3.8%. These results indicate that our setting effectively mitigates textual bias, ensuring a reliable evaluation of multimodal understanding abilities.
Knowledge Leakage from Pretraining.
To prevent performance underestimation caused by the out-of-distribution VG question format, we introduce an image description (ID) format. However, an VLM may have memorized details of movies from pretraining, which can be used to answer these questions. To assess the impact of prior knowledge, we test different models using question-only inputs. The results are shown in the last two columns of Tab. 4. We find that ChatGPT-4o, Gemini-1.5-pro, and LongVA achieve high performance on ID questions with question-only inputs, with +22.5%, +14.0%, and +13.4% over random chance, respectively. This indicates that they can call upon prior knowledge to answer the