t al., 2023; Frieder et al., 2023; Amirizaniani et al., 2024; Chinchure et al., 2025). This apparent contradiction suggests that (1) there is much overfitting to popular benchmarks such as GSM8K (Mirzadeh et al., 2024) through mechanisms like data contamination (Singh et al., 2024), and (2) existing benchmarks may not fully reflect differences between machine intelligence and human intelligence (Dziri et al., 2023; Wu et al., 2024). As a result, the development of sophisticated benchmarks that critically evaluate model capabilities has become a high priority for AI research.
Within the context of long video understanding, we investigate two crucial limitations of existing benchmarks for Video-Language Models (VLMs). First, benchmarks evaluating information extraction from a single location do not fully reflect the long video understanding ability of VLMs. A popular type of evaluation for long-context models adopts the “needle in a haystack” problem formulation (Kamradt, 2023; Wang et al., 2024b), where the “needle” represents the information to be extracted from the long context. Some long video benchmarks feature 1-needle questions that require information from a single location in the video. However, research in NLP indicates that model performances on 1-needle questions are often much higher than performances on questions requiring the extraction and understanding multiple needles, revealing the limitations of 1-needle questions for assessing long-context understanding (Li et al., 2023; Vodrahalli et al., 2024; Yang, 2024; Levy et al., 2024). However, multiple-needle questions remain rare in the multimodal setting.
Second, existing benchmarks offer incomplete evaluation of whether VLMs possess an internal world model, which captures the underlying mechanisms governing object dynamics and human behaviors and enables predictions of future events from an intervention (Forrester, 1971; Ha and Schmidhuber, 2018). Existing evaluations of world models (Guan et al., 2024a; Motamed et al., 2025; Kang et al., 2024) focus solely on object motion prediction, neglecting human behaviors and event causality (Sun et al., 2024).
To address these limitations, we propose a long-context video understanding benchmark, Causal2Needles, comprising 2,606 1-needle questions and 1,494 2-needle questions. Out of these, 902 1-needle questions and all 2-needle questions involve causal, world-model reasoning. Causal2Needles evaluates two key abilities of VLMs: (1) extracting relevant information from two locations in long videos and jointly reasoning about them, and (2) modeling the world in terms of causes and effects of human behaviors.
The causal 2-needle questions are constructed from a pair of cause and effect events. Each question should require the VLM to first retrieve the effect event and then the cause event. To formulate the question, we identify a bridge entity, which is an entity or a piece of information shared by the cause event and the effect event. Part one of the question asks the VLM to resolve the bridge entity by retrieving the effect event. Part two of the question requires the retrieval of the cause of the effect event. As an example, Fig. 1 shows a cause event and an effect event that share a bridge entity, “Superman’s death,” ambiguously referred to as “tragedy.” To answer question part one, the model must resolve the content of the tragedy by retrieving the effect event, which reveals the tragedy is Superman’s death. Only after that, the model can answer question part two by retrieving the video clip showing the cause of Superman’s death. If the bridge entity were explicitly stated, the question would become “how did Superman die, leading to his memorial service in Metropolis”, which can be answered by retrieving the cause event directly. In that case, the 2-needle question would degenerate to a 1-needle question. This novel problem formulation allows Causal2Needles to assess the two-needle ability and the causal reasoning abili