could be induced by factors like biology (e.g., heavy sweating causes thirst), psychology (e.g., hearing compliments makes people feel good), and social norms (e.g., providing good service leads to tipping). Although causal questions sporadically appear in video understanding datasets, such as the episodic reasoning questions in MVBench, it is usually difficult to isolate them and quantitatively measure this factor independently. In contrast, Causal2Needles is dedicated to evaluating reasoning over cause and effect in the context of human behaviors, which allows precise diagnosis of this capability and fills a gap in benchmarks of world models.
3 Causal2Needles: Dataset Construction
Causal2Needles is built on two video-language datasets, YMS (Dogan et al., 2018) and SyMoN (Sun et al., 2022). With a total of 192 fully annotated movie recap videos, the datasets offer a rich collection of human-behavior events from diverse movie genres. Each event consists of a narration sentence and its corresponding video clip.
Details such as the distribution of movie themes and the temporal distance between causal events can be found in Appendix Sec. A.
To facilitate the evaluation of models of diverse capabilities, Causal2Needles consists of both 1-needle and 2-needle questions, as well as causal and noncausal questions, which have different difficulty levels.
The generation of both 1-needle and 2-needle questions depend on pairs of cause and effect sentences, extracted in a causal relationship extraction step (Sec. 3.1).
We describe the generation of 1-needle questions in Sec. 3.2 and the generation of 2-needle questions in Sec. 3.3.
3.1 Causal Relationships Extraction
We employ a Large Language Model (LLM) to extract causal relationships from narration by combining global and local event graphs, based on an event graph extraction method (Sun et al., 2024). Each event (a narration sentence) is represented as a node, and causal relationships as directed edges. The global graph is extracted from the complete narration of a video. The graph captures long-range causal relationships but may be incomplete, because LLM often overlook sentences in the middle section of a long context (Liu et al., 2024). To address this, we introduce a sliding-window approach that extracts local graphs from 15-sentence segments with a 5-sentence stride. These local graphs capture more detailed causal relationships with a shorter range. We merge the global and local graphs to obtain comprehensive and long-range causal relationships. To avoid superficial causal relationships resulting from temporal adjacency, we retain only those where the cause and effect are separated by at least three events. In Causal2Needles, the distance between cause and effect events ranges from 3 to 21 events. Appendix Sec. A and B contain more details. The prompt used is shown in Appendix Fig. 9.
3.2 Generation of One-Needle Questions
With the extracted causal relationships, we use an LLM to generate simple 1-needle questions. To facilitate reader understanding, we illustrate the process using the cause and effect events shown in Fig. 2.
We first prompt the LLM to generate noncausal one-needle questions, which ask for a detail in either the cause or the effect event. For the cause event in Fig. 2, the generated question is: “Who does the creature stab and kill?” We can similarly generate another question for the effect event.
The causal one-needle questions differ from the noncausal question by requiring understanding of the causal relation between the two events. We prompt the LLM to generate a question asking for the cause event of a specific effect. The generated question for the example in Fig. 2 is: “What happened to Superman, leading to his memorial service in Metropolis?”
Finally, to mitigate potential textual bias from the input narration, we combine these questions with the visual grounding instruction: “Select the scene that contains the answer to the question.” The prompt for 1-ne