                                                             Multimodal Session Simulation                                                                                                   Conversation History Assembly
   1 Topic Ontology                             2 Web Image                                    3      Two-agent                           4 Multimodal Session
                                                                                                                                                                                             M M: Multimodal Session
                                                     Retrieval + Filtering                            Dialogue
                                                                                                                                                                                             E E: Evidence Session
                                                                                                   VLM                   VLM                                                                 F F: Filler Session (Text-only)
                                                                                                   User                Assistant               ···
                                                                                                                                                                                     ···     Assembled Multimodal Conversation History
                                                                                                                                               ···
                                                                                                                                                                                               11/02       M       ···
                                                                                                                                                                                     ···
           Topic Pool
                                                                                                                                                                                               12/02        F      ···

                                     ✓ image selected before dialogue → structural image-text alignment                                                                                           。         E      ···
                                                                                                                                                                                                  。         F      ···
                                     Question-Answer Pair Construction                                                                  Evidence Session Construction                             。
                                                                                                                                                                                                  。        M       ···
  1 Background Generation                 2     Visual Anchor Retrieval             3     Entity Ambiguity                                                     VLM / Agent
                                                                                                                                                                                                  。         E      ···
      Sampled Topic Ontology                                              replace name with generic placeholder                                                                                   。
                                     Anchor Entity     Golden             BEFORE (named entity)           AFTER (generic placeholder)
                                                      Gate Bridge                                                                                : What time did I meet Emma                                F      ···
                                                                               Emma and I                   Emma and I                       at the Golden Gate Bridge last
                                                                               finally met                  finally met                      Saturday?                                         14/04        E      ···
                                                                               crossed the                  crossed the
                                                                               Golden Gate                  bridge <image>                           : 6:42 AM
                                                                               Bridge together              together last                                                                        Fillers (text-only sessions)
      Emma and I finally met                                                   last Saturday ···            Saturday ···                             Atomic Facts (evidence units)
      crossed the Golden Gate
      Bridge together last                                                                                                                           F1                        F2
                                                                                                                                            My sister Emma              My sister and I
      Saturday, three years after
                                                                                                                                            and I crossed the           started crossing
                                                                                                                                                                                                      From ShareGPT / UltraChat
      we first planned to. We met                                                                                                           bridge in <image>
                                                                               Golden Gate Bridge             the bridge <image>                                        the bridge at 6:42
      on the the south side of the                                                                                                          last Saturday               AM that morning.
      bridge at at 6:42 AM to                                                                                                               morning.
      catch the sunrise, and I
      wore Dad’s red windbreaker
                                                                             Cross with    Saturday       Cross with      Saturday                                                                   Fixed text/image ratio
      from his 1998 photo.
                                                                             Emma          6:42 AM        Emma            6:42 AM           indirectly mention the fact in conversation                 Context Budgets
      Halfway across, near the
                                                                                                                                                     S1                        S2
      second tower, Marcus
      proposed to Emma. We                                                      text-only path (unsolvable)
      celebrated afterward at Pier                                                                                                                                                             32K         64K           128K     256K
                                     Anchor Image
      39.
                                                                          ✗ Requires image evidence


                                                                    Figure 1: M EM L ENS construction pipeline.


               • Temporal Reasoning (TR) assesses joint reasoning over temporal references, including both
                 natural-language expressions and session timestamps, together with visual content: duration
                 comparison compares two intervals derived from textual or visual cues, and temporal
                 grounding either orders events chronologically or extracts the specific date of an event.
                 Beyond entity abstraction, TR also replaces temporal expressions with visual artifacts such
                 as clock faces and calendar pages.
               • Knowledge Update (KU) tests the ability to track an evolving user attribute across a chain of
                 four successive updates [57] (e.g., “I used to like apples” → “now I prefer kiwi”), requiring
                 the model to reason from the final state of the chain rather than earlier superseded ones.
               • Answer Refusal (AR) removes all supporting evidence from an otherwise answerable ques-
                 tion, so it can no longer be answered from the remaining context; a correct model must
                 decline rather than hallucinate [58]. AR serves as a calibration check for hallucination
                 detection, not a core memory retrieval task.

A single cross-modal principle unifies the four answerable types: the evidence image carries infor-
mation that the text deliberately withholds, so correct answers require joint visual–textual reasoning
(§3.2); §3.4 validates this dependency empirically. The formal problem specification appears in
Appendix C.1. Appendix I grounds this taxonomy in established memory research, and Appendix C.3
provides the complete subtype breakdown with representative examples.

3.2        Data Curation

The construction proceeds in four components (Figure 1): (i) first, multimodal session simulation
generates topic-grounded multimodal dialogue sessions that form the conversation background;
(ii) next, question construction produces evaluation questions whose answers require visual content
through entity abstraction; (iii) evidence session construction wraps each evidence fact into a complete
session that matches its topical and stylistic profile; and (iv) conversation history assembly interleaves
evidence, haystack, and text-only filler sessions in timestamp order and compiles each question into
four standardized input lengths.

Multimodal session construction. Each session begins with topic sampling from a hierarchical
ontology. For each topic, we generate an image query to retrieve a batch of candidate images from the
web1 . We then filter these candidates for visual relevance. From this filtered set, we select the final
   1 https://github.com/hellock/icrawler; sourcing, licensing, privacy, and release detailed in Appendix C.7.




                                                                                                                            4
