stion asking for the cause event of a specific effect. The generated question for the example in Fig. 2 is: “What happened to Superman, leading to his memorial service in Metropolis?”
Finally, to mitigate potential textual bias from the input narration, we combine these questions with the visual grounding instruction: “Select the scene that contains the answer to the question.” The prompt for 1-needle question generation is shown in Appendix Fig. 10.
3.3 Generation of Two-Needle Questions
Generating 2-needle questions involves two steps after obtaining causal relationships: (1) rephrasing the cause and effect sentences to establish the bridge entity between them, and (2) generating a question that requires joint understanding of both cause and effect events.
Rephrasing the Cause and Effect Sentences.
The generation of 2-needle questions is based on a bridge entity that connects cause and effect events. The bridge entity serves two critical purposes: it provides a hint that drives the model to locate the cause event, so we do not provide the cause event directly; it forces the model to resolve the content of the vaguely phrased bridge entity by locating the effect event, before it can locate the cause event.
However, the bridge entity may not be explicitly mentioned in the original cause and effect sentences. Therefore, we prompt a VLM to rephrase the cause and effect sentences to explicitly establish the bridge entity. For example, the cause sentence, “As it dies, the creature stabs and kills the weakened Superman with one of its bone protrusions,” and the effect sentence, “A memorial is held for Superman in Metropolis,” are rephrased into sentences in Fig. 2. To maintain semantic consistency between the rephrased sentences and their corresponding clips, we also input the video clips into the VLM as a constraint.
Generation of Visual Grounding Questions.
After obtaining causal sentence pairs with a clear bridge entity, we utilize an LLM to generate two parts of a visual grounding 2-needle question. We use the causal relationship in Fig. 2 to demonstrate this process.
First, we prompt the LLM to generate a Part 1 question that requires retrieving the effect event to identify the bridge entity. Specifically, we ask it to extract the bridge entity that establishes the causal relationship, such as “Superman’s death.” Next, we instruct the LLM to rephrase this bridge entity as a vague reference, such as “tragedy.” After that, using this vague reference, we prompt the LLM to create a question that uses the effect sentence as context and the bridge entity as the answer. In this case, the final output is: “What tragedy was commemorated by the memorial service in Metropolis?”
Further, we prompt the LLM to generate a Part 2 question, which requires retrieving the cause event. In the example, with the vague reference “tragedy”, we instruct the LLM to generate a question whose answer appears exclusively in the cause sentence. The output is: “How did such a tragedy occur?” It is important to keep the bridge entity vague in this question, so that the question does not give the effect event away. Instead, the model must retrieve the video clip of the effect event and resolve the bridge entity reference.
Finally, combining the two question parts with a task instruction, we complete a visual grounding 2-needle question: “What tragedy was commemorated by the memorial service in Metropolis, and how did such a tragedy occur? Select the scenes that contain the answers to the question.” To answer it, the model must jointly understand cause and effect events and ground the answers on video clips. The prompts are in Appendix Fig. 12.
Generation of Image Description Questions.
The visual grounding question format may be out-of-distribution for some models. As a result, it may underestimate their performance. To fix this, we also generate questions in a complementary format: multiple-choice image description questions. These questions share the same Pa