 text, which are part of the input of Causal2Needles. We compare the SG performance of GPT-4o on Part 1 and Part 2 questions, and summarize the results in Fig. 8. We make the following observations.
Figure 8: Performance of ChatGPT-4o on the sentence grounding (SG) task and the visual grounding (VG) task on the 2-needle questions. In the SG tasks, the sentences are always arranged in their natural order. In the VG task, we take the average of the forward and reverse orderings.
First, the results show that the Part 2 question relies on joint understanding of both the cause and the effect sentences. We compute the conditional probability that GPT-4o answers SG Part 2 correctly given that it answers SG Part 1 correctly, P⁡(SG-Part2=correct∣SG-Part1=correct)=82.1%\mathrm{P}(\text{SG-Part2}={\text{correct}}\mid\text{SG-Part1}={\text{correct}})=82.1\%. In contrast, the conditional probability that GPT-4o answers SG Part 2 correctly given that it answers SG Part 1 incorrectly, P⁡(SG-Part2=correct∣SG-Part1=incorrect)=39.6%\mathrm{P}(\text{SG-Part2}={\text{correct}}\mid\text{SG-Part1}={\text{incorrect}})=39.6\%. The large gap between the two conditional probabilities demonstrates that performance on SG Part 2 significantly depends on SG Part 2. This suggests the existence of a sequential dependency between the two parts. The finding demonstrates that, at least in the textual modality, the two-needle questions benefit from the joint understanding of both needles.
Second, the dependence of VG performance on SG performance is limited. We compute the conditional probability that GPT-4o answers VG Part 2 correctly given that it answers SG Part 2 correctly, P⁡(VG-Part2 = correct∣SG-Part2=correct)=33.5%\mathrm{P}(\text{VG-Part2 = \text{correct}}\mid\text{SG-Part2}={\text{correct}})=33.5\%. In contrast, the conditional probability that the model answers VG Part 2 correctly given that it answers SG Part 2 incorrectly is P⁡(VG-Part2=correct∣SG-Part2=incorrect)=25.4%\mathrm{P}(\text{VG-Part2}={\text{correct}}\mid\text{SG-Part2}={\text{incorrect}})=25.4\%. The difference between the two probabilities is small. This shows the correctness of SG Part 2 only makes minor contribution to VG Part 2. That is, even if the model is wrong on SG Part 2, it still has a decent chance of being correct on VG Part 2, relative to the alternative situation that the model is correct on SG Part 2. Hypothetically, a main reason for this phenomenon is that the model utilizes a separate mechanism for visual reasoning, which operates more or less independently of reasoning over the textual modality.
Lastly, on SG Part 2, GPT-4o achieves 63.7% accuracy, much higher than the 30.3% on VG Part 2, suggesting the model is better at textual than visual reasoning. This large gap indicates substantial room for improvement in visual comprehension of VLMs.
Answer Uniqueness of Part 2 Questions.
We conduct an experiment to verify if Part 2 questions have unique answers. In causal relationship extraction, an event graph may contain multiple nodes pointing to the same node, indicating that a single event can have multiple causes. This may introduce multiple ground truth answers to Part 2 questions, as they require retrieving the cause of an event.
Our experiment is designed as follows: among the 747747 causal relationships of 2-needle questions, we identify 9999 instances where the effect sentence is not uniquely paired with a single cause sentence. For each effect sentence in them, we feed the corresponding Part 2 question and cause sentences (on average, 2.022.02 candidate causes per effect) to ChatGPT-4.1. Then, it is asked to select the cause sentence that contains the answer to the Part 2 question. The results show that out of the 9999 such cases, ChatGPT-4.1 selects the wrong cause sentence in only one instance. This indicates that the answer to Part 2 question is mostly unique.
Table 5: Comparison of results between stacked image inputs and non-stacked image inputs.
Models
Input