h. If the bridge entity were explicitly stated, the question would become “how did Superman die, leading to his memorial service in Metropolis”, which can be answered by retrieving the cause event directly. In that case, the 2-needle question would degenerate to a 1-needle question. This novel problem formulation allows Causal2Needles to assess the two-needle ability and the causal reasoning ability together.
Figure 1: The logical solution process for the 2-needle questions of Causal2Needles. Each step involves an operation, with the input shown above the step and the output below the step. The question purposely refer to the bridge entity, “Superman’s death,” ambiguously as “tragedy.” As a result, one must first resolve the bridge entity using Part 1 before answering Part 2. This question design mandates joint understanding of both the cause and effect events. Note that the steps are necessary only in an information-processing sense. A VLM may adopt different steps.
Figure 2: The evaluation framework of Causal2Needles. To help models understand the storyline, we also feed the full textual narration into the model. Four types of questions are designed for each pair of causally related events.
Causal2Needles is constructed on top of movie summary videos, containing video clips from the movies and narration text. However, this may create a shortcut that VLMs can exploit. The video clips could be difficult to understand without the narration text, but if we feed the narration text as input, the VLM may answer the question from the text directly without using the video. This is the infamous phenomenon of textual bias (Ko et al., 2023; Cores et al., 2024; Xiao et al., 2024).
As mitigation, we introduce two complementary question formats. The first format, called visual grounding, requires the model to select the video clip corresponding to the event it needs to retrieve. The second format, called image description, requires the model to answer multiple-choice questions about the appearance of the retrieved video clip. Fig. 2 shows some examples. Visual grounding questions necessitate the understanding of video clips, but they may underestimate model performance as the format may be out-of-distribution (OOD) to most VLMs. In contrast, image description questions circumvent the OOD issue, but the VLM may benefit from knowledge of the movie learned from pretraining data, such as the color of Lois Lane’s outfit, leading to overestimation of model abilities. Causal2Needles adopts the visual grounding format for all 1-needle questions and splits the 2-needle questions between the two formats (747 each).
Experiments reveal several important findings. First, causal questions appear substantially more difficult than noncausal questions. Second, models that perform well on 1-needle questions exhibit remarkable performance drops on 2-needle questions. Finally, the distance between the two needles is negatively correlated with model performance. Taken together, these results demonstrate that joint understanding of and causal reasoning over two separate video locations remain important weaknesses of existing VLMs. We summarize the contributions of this paper as follows:
∙\bullet We propose the Causal2Needles benchmark that contains causal 1-needle and 2-needle questions, generated from story videos. We devise a question creation strategy utilizing bridge entities to force the video-language models to jointly understand two video clips arbitrarily located in the context window.
∙\bullet We reveal significant weaknesses in current VLMs in the joint understanding and retrieval of two separate video clips, even though their 1-needle performances are high. The more distant the two clips are, the worse the models perform.
2 Related Work
2.1 Long Video Understanding Benchmarks
Retrieval of information from a specific location in a long video (usually called “needle in a haystack”) is widely regarded as a key capability in long-video understanding (Kam