ing the frames can help the model better capture and comprehend the story within a video clip. As verification, we conduct additional experiments on VG 2-needle questions using ChatGPT-4o, Gemini-1.5-pro and Qwen2.5VL-7B with non-stacked frames as input. For a fair comparison, we instruct the model to output the frame number containing the answer and calculate the clip number by dividing the frame number by five, as every five frames belong to a single clip. The results are detailed in Tab. 5. We can observe that stacked frames improve performance significantly, indicating that the stacked method enhances visual understanding.
Appendix E Limitations
While our benchmark addresses several limitations of prior work, such as the lack of multi-needle evaluation on long-context video understanding and the understanding of human behaviors, there remain areas for future improvement.
First, our videos are primarily sourced from movie clips, which provide a rich set of event causal relations. Compared to real life, movies tend to over-emphasize uncommon events, such as serendipity, intricate conspiracy, or dramatic conflicts. These events could serve as effective out-of-distribution tests for models primarily trained on common events such as ego-centric videos. However, they should not be taken as necessarily representative of real-life events. Second, our current evaluation focuses on VLMs. Many of them, particularly open-source ones, do not yet support audio inputs. Accordingly, we exclude audio from our inputs, but future extensions of the benchmark will incorporate audio to enable evaluation of more comprehensive multimodal understanding.
Appendix F Generation and Evaluation Prompts
We present prompting templates for causal-relationship extraction (Fig. 9), 1-needle question generation (Fig. 10), visual-grounding 2-needle question generation (Fig. 12), and image-description 2-needle question generation (Fig. 14). Visual-grounding 2-needle questions and 1-needle questions are generated with GPT-o1 (openai-o1-preview-2024-09-12), with sentence rephrasing performed by Gemini-1.5-Pro-002 as part of the visual-grounding 2-needle pipeline; image-description 2-needle questions are generated with Gemini-1.5-Pro-002.
In Fig. 15, we continue using the same model to generate answer choices. At this stage, to increase the difficulty of the options, we input the cause clip as well as the two adjacent frames and prompt the model to consider generating more challenging options based on objects appearing in the remaining scenes.
Causal Relationships Extraction (GPT-o1-preview-2024-09-12)
Here is a list of nodes (events) from a story event graph. We want you to fill
in the edges of the event graph with causal connections between nodes. An event
graph contains nodes and edges. Each node represents an event, and each edge
represents the causal connection between two events.
Example Input:
Node 0: When Dan goes to school in the morning, he has to take the bus.
Node 1: One day Dan was running late, and missed the bus to school.
Node 2: Dan called his friend Pete, and asked for a ride to school.
Node 3: Pete gave Dan a ride to school, but Dan was late for his first class.
Node 4: Luckily Dan wasn’t late for any of his other classes that day.
Example Output:
Edge 0: (Node 0 -> Node 1)
Edge 1: (Node 1 -> Node 2)
Edge 2: (Node 2 -> Node 3)
Edge 3: (Node 1 -> Node 3)
Edge 4: (Node 3 -> Node 4)
(continue with another five demonstrations)
Now, it is your turn to construct the event graph for the following event list.
Event List:
Node 0: <S1>
Node 1: <S2>
Node 2: <S3>
Node 3: <S4>
Node 4: <S5>
Output:
Figure 9: The prompt for causal relationships extraction.
1-Needle Question Generation (GPT-o1-preview-2024-09-12)
Caulsal 1-Needle Question
Given a pair of sentences with a causal relationship, create a question about the “Effect" sentence such that the answer to this question
is found exclusively in the “Cause" sentence. Ensure that the answer does not appear wi