 to ChatGPT-4.1. Then, it is asked to select the cause sentence that contains the answer to the Part 2 question. The results show that out of the 9999 such cases, ChatGPT-4.1 selects the wrong cause sentence in only one instance. This indicates that the answer to Part 2 question is mostly unique.
Table 5: Comparison of results between stacked image inputs and non-stacked image inputs.
Models
Input
Setting
VG 2-Needle Questions
Forward
Reverse
Avg
Part 1
Part 2
Both
Part 1
Part 2
Both
Part 1
Part 2
Both
ChatGPT-4o
non-stacked
5.0
29.4
1.4
29.7
4.1
0.9
17.4
16.8
1.2
stacked
16.7
39.2
9.4
45.4
21.2
13.4
31.1
30.2
11.4
Gemini-1.5-pro
non-stacked
5.4
37.3
2.9
26.3
17.5
6.1
15.8
27.4
4.5
stacked
21.0
40.0
10.2
35.7
21.4
8.4
28.4
30.7
9.3
Qwen2.5VL-7B
non-stacked
2.1
11.0
0.0
13.8
4.4
0.4
8.0
7.7
0.2
stacked
27.6
17.7
5.0
11.2
18.9
1.9
19.4
18.3
3.4
Existence of Counterfactual Answers in ID 2-Needle Questions.
A counterfactual answer refers to an answer that is supported apparently by the input video and can be ruled out only by causal reasoning. If the video contains only the correct answer and no counterfactual answers, a neural network can easily rule out wrong answers without locating the required video clip.
For example, in Fig. 2, the image description question is “What tragedy was commemorated by the memorial service in Metropolis? When the tragedy that causes the memorial service occurs, what color were the energy blasts emanating from the creature? A. Yellow B. Blue C. Red D. White.” The correct answer is “Yellow”, which appears in the cause clip. “Blue” is a competing answer as a blue light is shown in another battle clip at 2’37 of the video. Due to the presence of both answers in the input video, to correctly answer the question, the model must identify the cause clip through causal reasoning.
We adopt two question design methods to ensure the existence of counterfactual answers in image description questions. First, we are deliberately ambiguous about the reference in the question wording, as discussed in Sec. 3.3. Second, we encourage the LLM to search for visually similar but potentially confusing content from adjacent clips, using the instructions in Appendix Fig. 14.
To demonstrate the existence of counterfactual answers, we conduct an experiment with the following setting: (1) We remove the cause clip from the input and any information that helps to locate the cause clip from the question, allowing the model to answer the question using other clips. For example, the aforementioned question becomes “ What color were the energy blasts emanating from the creature?”; (2) We replace the ground truth answer with “None is correct,” so if there are no counterfactual answers, the model should choose this option. In experiments with ChatGPT-4o, the model selects “None is correct” for only 17.8% of questions. This suggests that most image description questions in Causal2Needles have counterfactual answers.
Non-stacked vs. Stacked Image Input.
We explain the rationale for the design choice of inputting each video clip by stacking the video frames into a single image. We represent a video clip using five uniformly sampled frames. After that, we stack these frames from top to bottom as a single image as input to the VLMs. There are two reasons for the stacked approach. First, most open-source models have a limited window size and can only process 32 frames at a time. Therefore, without stacking, processing multiple video clips would quickly exceed the window size. Second, for proprietary models that can accept more frames as input, stacking the frames can help the model better capture and comprehend the story within a video clip. As verification, we conduct additional experiments on VG 2-needle questions using ChatGPT-4o, Gemini-1.5-pro and Qwen2.5VL-7B with non-stacked frames as input. For a fair comparison, we instruct the model to output the frame number containing the answer and calculate the clip number by dividing the frame