Two Causally Related Needles in a Video Haystack
Report GitHub Issue
×
Title:
Content selection saved. Describe the issue below:
Description:
Submit without GitHub
Submit in GitHub
arXiv is now an independent nonprofit!
Learn more
×
Back to arXiv
Why HTML?
Report Issue
Back to Abstract
Download PDF
Abstract
1 Introduction
2 Related Work
2.1 Long Video Understanding Benchmarks
2.2 World Model
3 Causal2Needles: Dataset Construction
3.1 Causal Relationships Extraction
3.2 Generation of One-Needle Questions
3.3 Generation of Two-Needle Questions
Rephrasing the Cause and Effect Sentences.
Generation of Visual Grounding Questions.
Generation of Image Description Questions.
3.4 Quality Evaluation of Generated Questions
4 Evaluation of VLMs
4.1 Evaluation Setup
4.2 Main Results
Causal Questions Are More Challenging Than Noncausal Questions.
Two-needle Questions Are More Challenging Than One-needle.
Open-Source Models Exhibit Weaker World Modeling Ability.
Performance Decreases with Increasing Needle Distance.
4.3 Pathological Behaviors of VLMs
Positional Bias.
Static Output Bias.
4.4 Analysis of Dataset Bias
Textual Bias.
Knowledge Leakage from Pretraining.
References
A Dataset Statistics
B Event Graph Merging
C Evaluation Details
Computing Resources.
Details of VLMs.
Proprietary LLMs.
Open-source VLMs.
Random Chance Calculation.
D Additional Experiments
The Influence of Needle Distance.
Textual vs. Visual Grounding.
Answer Uniqueness of Part 2 Questions.
Existence of Counterfactual Answers in ID 2-Needle Questions.
Non-stacked vs. Stacked Image Input.
E Limitations
F Generation and Evaluation Prompts
G Examples
License: CC BY-NC-SA 4.0
arXiv:2505.19853v3 [cs.CV] 06 Nov 2025
Two Causally Related Needles in a Video Haystack
Miaoyu Li
††thanks: Equal contribution
Affiliation: College of Computing and Data Science
Qin Chao11footnotemark:
1
Affiliation: Nanyang Technological University, Singapore
Boyang Li
Affiliation: {miaoyu.li, chao0009, boyang.li}@ntu.edu.sg
Abstract
Properly evaluating the ability of Video-Language Models (VLMs) to understand long videos remains a challenge. We propose a long-context video understanding benchmark, Causal2Needles, that assesses two crucial abilities insufficiently addressed by existing benchmarks: (1) extracting information from two separate locations (two needles) in a long video and understanding them jointly, and (2) modeling the world in terms of cause and effect in human behaviors. Causal2Needles evaluates these abilities using noncausal one-needle, causal one-needle, and causal two-needle questions. The most complex question type, causal two-needle questions, require extracting information from both the cause and effect events from a long video and the associated narration text. To prevent textual bias, we introduce two complementary question formats: locating the video clip containing the answer, and verbal description of a visual detail from that video clip. Our experiments reveal that models excelling on existing benchmarks struggle with causal 2-needle questions, and the model performance is negatively correlated with the distance between the two needles. These findings highlight critical limitations in current VLMs. The project page is available at: https://limiaoyu.github.io/Causal2Needles
1 Introduction
On many popular benchmarks (Hendrycks et al., 2020; Cobbe et al., 2021; Srivastava et al., 2022; Chen et al., 2021), recent AI systems achieved performance comparable to humans. However, real-world observations suggest that there is still a significant gap between AI and human capabilities (West et al., 2023; Frieder et al., 2023; Amirizaniani et al., 2024; Chinchure et al., 2025). This apparent contradiction suggests that (1) there is much overfitting to popular benchmarks such as GSM8K (Mirzadeh et al., 2024) through mechanisms like data contamination (Singh et al., 2024), and (2) existing benchmarks may not fully reflect differences between machine intelligence and human intelligence (D