he correspondence between the questions and the cause-effect event pairs. For Factor 4, due to the difficulty in writing unreadable questions, we do not construct any random baseline for readability.
We utilize two state-of-the-art models as the evaluation LLMs, ChatGPT-4.1 and Gemini-2.0-flash (neither is involved in the question generation process), and recruit five human annotators. The models evaluate both 1-needle and 2-needle questions, while the annotators evaluate 136 visual grounding 2-needle questions. For human evaluation, we adopt majority voting among the five annotators for Yes/No evaluations, and use the average for numerical evaluations.
Tab. 2 shows the evaluation results. The generated questions receive near-perfect scores on all metrics. Reassuringly, all random baselines score near the lowest possible, which is 0% for shared existence of bridge entities and correctness of vague references, and 1 for factual correctness of questions. These results indicate that the automatically generated questions have high quality and the evaluation processes are valid. The prompts for question evaluation are in Appendix Sec. F.
Table 2: Evaluation results of generated questions. VG and ID refer to visual grounding and image description, respectively. 1-N and 2-N denote 1-needle and 2-needle questions. Numbers in parentheses indicate the performance of random baselines.
Models
Shared Existence
of Bridge Entities
Correctness of
Vague References
Factual Correctness of Questions
Readability of Questions
Noncausal
1-N
Causal
1-N
VG 2-N
ID 2-N
Noncausal
1-N
Causal
1-N
VG 2-N
ID 2-N
ChatGPT-4.1
95.6% (0.3%)
91.0% (3.6%)
4.71 (1.10)
4.62 (1.12)
4.99 (1.10)
4.74 (1.13)
4.91
4.85
4.83
4.67
Gemini-2.0-flash
95.0% (3.8%)
98.7% (2.5%)
4.75 (1.05)
4.66 (1.02)
4.96 (1.01)
4.83 (1.00)
4.75
4.18
4.69
4.25
Human
82.4%
98.5%
-
-
4.50
-
-
-
4.80
-
Table 3: Quantitative results (accuracy, %) of VLMs on our benchmark. “Forward” refers to inputting video clips in chronological order, while “Reverse” uses reverse order. “Avg” denotes results averaged over both orders. Best scores are in bold.
Models
Noncausal 1-N
Questions
Causal 1-N
Questions
VG 2-N Questions
ID 2-N
Questions
Forward
Reverse
Avg
Part 1
Part 2
Both
Part 1
Part 2
Both
Part 1
Part 2
Both
Human
–
78.2
83.7
85.9
79.3
-
-
88.2
Proprietary Models
ChatGPT-4o
56.8
39.2
16.7
39.2
9.4
45.4
21.2
13.4
31.1
30.2
11.4
59.2
Gemini-1.5-pro
55.4
35.6
21.0
40.0
10.2
35.7
21.4
8.4
28.4
30.7
9.3
60.9
ChatGPT-4o-mini
39.9
33.4
17.4
22.9
5.0
32.4
11.9
5.2
24.9
17.4
5.1
52.3
Claude-3.5-sonnet
37.6
26.5
16.6
22.4
4.8
19.3
13.9
2.9
17.9
18.1
3.9
60.5
Open-source Models
Qwen2.5VL-32B
30.7
11.7
26.3
17.7
5.4
10.3
20.4
1.9
18.3
19.0
3.6
53.5
Qwen2.5VL-7B
17.5
13.6
27.6
17.7
5.0
11.2
18.9
1.9
19.4
18.3
3.4
43.2
LLaVA-Next-Video-34B
12.4
12.3
0.8
17.4
0.0
11.8
0.9
0.0
6.3
9.2
0.0
48.6
LLaVA-OneVision-7B
12.3
18.0
4.6
14.7
0.0
17.0
5.6
0.1
10.8
10.2
0.1
28.3
InternVL2-8B
11.6
7.4
14.5
8.3
1.2
9.5
9.1
0.5
12.0
8.7
0.9
40.2
LLaVA-Next-Video-7B
11.7
17.2
0.0
4.4
0.0
15.9
0.0
0.0
8.0
2.2
0.0
27.3
LongVA-7B
9.2
14.7
2.8
5.0
0.0
10.3
0.7
0.0
6.6
2.8
0.0
49.7
Aria-28B
7.0
12.1
19.0
14.8
0.6
18.7
18.1
0.1
18.9
16.5
0.4
43.0
LongVU-7B
3.3
12.2
3.2
1.3
0.3
4.4
2.1
0.5
3.8
1.7
0.4
34.2
Random Chance
9.8
9.8
9.8
9.8
1.0
9.8
9.8
1.0
9.8
9.8
1.0
25.0
4 Evaluation of VLMs
4.1 Evaluation Setup
We evaluate a total of 13 advanced VLMs, consisting of 4 proprietary models and 9 open-source models. Proprietary models include ChatGPT-4o (gpt-4o-2024-08-06) (OpenAI, 2024a), ChatGPT-4o-mini (gpt-4o-mini-2024-07-18) (OpenAI, 2024b), Gemini-1.5-pro (gemini-1.5-pro-002, Sep 2024) (Team et al., 2024), and Claude-3.5-sonnet (claude-3-5-sonnet-20241022) (Anthropic, 2024). Open-source models include LLaVA-Next-Video-7B (Zhang et al., 2024b), LLaVA-Next-Video-34B (Zhang et al., 2024b), LLaVA-OneVision-7B (Li et al., 2024a), LongVU-7B (Shen et al., 2024), Aria-28B (Li et al., 2024b), Qwen2.5VL-7B/32B (Bai et al., 2025), LongVA-7B (Zhang et a