Table 3: Cross-modality ablation on the image-essential and image-supportive questions. With
evidence: question + evidence facts + evidence images, no haystack (n = 634). W/o evidence:
question + evidence facts, no evidence images.
        Model            Input                 Overall    IE     MSR     TR     KU       ∆
                         With evidence image    93.13    94.31 100.00 96.91 75.86 —
        GPT-5.4
                         W/o evidence image      1.74     0.41  0.00  5.15 0.00 −91.39
                         With evidence image    89.42    89.02   90.21   96.19 82.24 —
        Gemini-3.1-Pro
                         W/o evidence image      1.89     0.00   0.00    6.19 0.00 −87.53



Remaining questions therefore require using the evidence image, which is a prerequisite for the
multimodal analyses below.

Human review. Three rounds of human review operate as a cumulative quality gate on the filtered
questions. Round 1 verifies that the evidence image carries answer-critical information; for AR
questions, which are constructed by removing the evidence facts, this round instead confirms that the
removed facts was answer-critical. Round 2 checks that evidence facts are naturally embedded and
recoverable from their sessions, and that each session reads as a plausible user–assistant exchange
with natural dialogue flow; sessions that fail the naturalness criterion are subsequently refined by
humans. Round 3 reviews the haystack sessions to assess both the quality of the images and the
naturalness of the dialogues. Together, the automated filters and three-round review reduce the initial
pool of 20k candidates to the final 789 questions. The filler sessions drawn from ShareGPT and
UltraChat (§3.2) further ground the surrounding context in authentic user–AI conversational patterns.
Annotation guidelines and inter-annotator agreement appear in Appendix E.1.

3.4   Cross-modality Validation

The construction pipeline targets image-necessary questions: across M EM L ENS, 65.7% of questions
are image-essential (the answer is unrecoverable without the evidence image), 14.7% are image-
supportive (the image confirms or disambiguates a textual fact), and 19.6% are text-sufficient (all AR
questions plus a subset of MSR items that test cross-session reasoning over purely textual evidence).
We verify this cross-modal dependency with two empirical checks (Table 3). An answerability test
supplies each image-essential and image-supportive question (n = 634) with its full evidence (textual
facts and evidence images) and confirms that the questions are answerable: GPT-5.4 reaches 93.13%
overall and Gemini-3.1-Pro 89.42%. A multimodal ablation then removes all evidence images:
overall accuracy collapses below 2% for both models. Two frontier proprietary LVLMs converge on
near-identical collapses, showing our questions are highly multimodal.


4     Evaluation and Analysis

4.1   Experimental Setup

We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS. The LVLMs span
closed-source systems including GPT-5.4 [2], Claude Sonnet 4.5 [7], and Gemini-3.1-Pro [67],
alongside major open-source families such as Kimi-K2.5 [68], Qwen3.5 [69], GLM-4.6V [70],
and Gemma3 [71]; the complete list is provided in Appendix B. The memory-augmented agents
comprise three multimodal pipelines (M3-Agent [35], M2A [34], and M3C [37]) and four text-only
pipelines (Mem0 [9], MemOS [29], MemAgent-7B [30], and Memory-T1 [28]), with backbone
and adapter details reported in Appendix G. Since the four text-only memory agents do not accept
visual inputs, we follow standard practice and replace the image inputs with captions generated by
BLIP-2 [72]; the remaining input adaptations are documented in Appendix B. For comparability,
the LVLMs are evaluated at three context lengths (32K, 64K, and 128K), as many of them do not
natively support 256K, whereas the agents are evaluated across all four context lengths (32K, 64K,
128K, and 256K). We report LLM-as-Judge accuracy [73] using Qwen3-VL-235B-A22B-Instruct
as the judge, cross-validated by GPT-5.4-mini re-judging (κ = 0.93) and a three-annotator human
consensus (Appendix E.2).


                                                  6
