                                                  Mem-Gallery: Benchmarking Multimodal Long-Term Conversational
                                                                  Memory for MLLM Agents

                                                    Yuanchen Bei1 , Tianxin Wei1 , Xuying Ning1 , Yanjun Zhao1 , Zhining Liu1 ,
                                                    Xiao Lin1 , Yada Zhu2 , Hendrik Hamann3,4 , Jingrui He1 , Hanghang Tong1
                                                1
                                                  University of Illinois Urbana-Champaign 2 MIT-IBM Watson AI Lab, IBM Research
                                                              3
                                                                Stony Brook University 4 Brookhaven National Laboratory

                                                                                                                                                                                                                            Timeline
                                                                                                                                             …           Multimodal Multi-Session                …
                                                                                                                     2025-03-02                                                                                 2025-06-22
                                                                                                                                                               Conversation
                                                                  Abstract                                          Session-1
                                                                                                                                                              After Multi-Session
                                                                                                                                                                Conversations…
                                                                                                                                                                                                                Session-n
                                                                                                                                                                                                      We finally had Luna’s second
                                                                                                          ... I met the sweetest Cocker
                                                                                                                  Spaniel today! …                                                                   session today! … She was more
                                                                                                                                                                                                        cooperative this time. …
                                                                                                         Attached Image
                                                Long-term memory is a critical capability for                                                                                                        Attached Image

                                                                                                                                                                                                      Typical Springer energy! …
arXiv:2601.03515v1 [cs.CL] 7 Jan 2026




                                                                                                             That’s beautiful. They’re                          Example Topics:
                                                                                                                                                                                                      Familiarity often stabilizes
                                                multimodal large language model (MLLM)                    incredibly empathetic dogs. …
                                                                                                                                                                Dog, Behavior,
                                                                                                                                                                                                        behavioral responses. …

                                                                                                                                                                Academic, Life
                                                agents, particularly in conversational settings                   Multi-Round Dialogues           …                                                          Multi-Round Dialogues       …

                                                where information accumulates and evolves                 …His name is Milo. I think I’m
                                                                                                           falling in love with spaniels…
                                                                                                                                                                                                      I want to merge Milo’s video
                                                                                                                                                                                                         too. It adds emotional
                                                                                                                                                                                                         expression variety. …
                                                over time. However, existing benchmarks ei-                 … Once someone meets a
                                                                                                                                                                                                     Attached Image
                                                                                                           Cocker Spaniel, it’s hard not
                                                                                                                                                                                                      Smart idea! Cocker data will
                                                ther evaluate multi-session memory in text-                     to be enchanted!
                                                                                                                          Attached Image
                                                                                                                                                                                                         enrich the affective
                                                                                                                                                                                                       dimension of your study.

                                                only conversations or assess multimodal un-
                                                                                                                                                        Multi-Dimension Evaluation
                                                derstanding within localized contexts, failing
                                                                                                                                                      Memory Extraction & Adaptation
                                                to evaluate how multimodal memory is pre-                       Factual Retrieval                            Visual-centric Search                          Test-Time Learning

                                                served, organized, and evolved across long-                What candidate names                             Find the two images related
                                                                                                                                                                                                        What breed is the dog
                                                                                                           have been listed when                             to Lucy‘s pet. Return the
                                                                                                                                                                                                        in the attached image?
                                                term conversational trajectories. Thus, we intro-           naming Lena's dog?                                      image_id(s).



                                                duce Mem-Gallery1 , a new benchmark for eval-             … Coral... Sunny… or Lumi …

                                                                                                                                     …
                                                                                                                                                                                             …            Cocker Spaniel:

                                                                                                                                                                                                          Others:
                                                                                                                                                                                                                                         …

                                                                                                                                                                                                                                         …
                                                                                                                                                                                             …
                                                uating multimodal long-term conversational
                                                                                                                                                             Memory Reasoning
                                                memory in MLLM agents. Mem-Gallery fea-
                                                                                                            Temporal Reasoning                             Visual-centric Reasoning                        Multi-entity Reasoning
                                                tures high-quality multi-session conversations
                                                                                                          When did Lena and Lumi meet                     The dog in the picture more                How did Luna and Rocky respond
                                                grounded in both visual and textual informa-               the Papillon at the park?                         similar to who’s pet?                   differently when a clipboard was
                                                                                                                                                                                                     dropped during the lab session?


                                                tion, with long interaction horizons and rich            Date: 2024-09-28
                                                                                                         I took Lumi to the riverside park
                                                                                                                                                              Lena’s
                                                                                                                                                                       …
                                                                                                                                                                           Amy’s
                                                                                                                                                                                         …
                                                                                                                                                                                                     … When … a clipboard, he flinched
                                                                                                                                                                                                     but then wagged his tail and …
                                                                                                                                                                                                                                             …
                                                                                                         today. we met … little dog.
                                                                                                                                                              Lucy’s       Other’s                   … Luna, on the other hand, barked
                                                multimodal dependencies. Building on this                                             …                                …             …               once and …                              …


                                                dataset, we propose a systematic evaluation                                                           Memory Knowledge Management

                                                framework that assesses key memory capabil-                    Knowledge Resolution                              Conflict Detection                           Answer Refusal


                                                ities along three functional dimensions: mem-           After Lena’s correction, which
                                                                                                        dog breed does her sister
                                                                                                                                                           Conflict with the memory or not:
                                                                                                                                                           Sussex Spaniel has white-and-
                                                                                                                                                                                                         What is the name of Lena's
                                                                                                                                                                                                         friend Amy's cat?
                                                                                                        actually like?                                     liver coat pattern and alert eyes?
                                                ory extraction and test-time adaptation, mem-              My sister loves Standard Poodles.                     ... Sussex spaniel…
                                                                                                                                                                                                            Amy has a Cairn Terrier… she
                                                                                                                                                                                                            told me she named her Cairn
                                                                                                           I got it wrong earlier. Her favorite                  … teddy bear-short fur…                    Terrier ‘Pebble’…           …
                                                ory reasoning, and memory knowledge man-                   breed is as the image. …
                                                                                                           … the Standard Schnauzer …                                        …
                                                                                                                                                                                                              Not mentioned

                                                agement. Extensive benchmarking across thir-
                                                teen memory systems reveals several key find-
                                                                                                    Figure 1: The conceptual illustration of Mem-Gallery.
                                                ings, highlighting the necessity of explicit mul-
                                                timodal information retention and memory or-
                                                                                                    a primary medium through which agents acquire,
                                                ganization, the persistent limitations in memory
                                                reasoning and knowledge management, as well         refine, and utilize memory, making conversational
                                                as the efficiency bottleneck of current models.     settings a natural and critical testbed for evaluating
                                                                                                    long-term memory capabilities (Wu et al., 2025).
                                        1       Introduction                                           Human memory in conversation is inherently
                                                                                                    multimodal and evolving (Ardesch et al., 2019;
                                        The rapid progress of Multimodal Large Language
                                                                                                    Luppi et al., 2022). An effective agent therefore re-
                                        Models (MLLMs) has enabled the development of
                                                                                                    quires the ability to not only recall past information
                                        agents that can perceive, reason, and interact with
                                                                                                    but also integrate visual and textual cues, reason
                                        the world through both language and vision (Wu
                                                                                                    across events, and update outdated knowledge as
                                        et al., 2023; Zhang et al., 2024a). A fundamental
                                                                                                    conversations progress (Hu et al., 2025b; Bo et al.,
                                        capability for such agents is long-term memory:
                                                                                                    2025). However, endowing MLLM agents with
                                        store, retrieve, and update information accumulated
                                                                                                    such long-term multimodal conversational memory
                                        over extended interactions (Zhang et al., 2025b). In
                                                                                                    largely remains an open challenge, and its system-
                                        particular, multi-session conversations constitute
                                                                                                    atic evaluation is still underexplored.
                                            1
                                                   https://github.com/YuanchenBei/Mem-Gallery          Despite growing interest in agentic memory, ex-
isting benchmarks reflect a fundamental mismatch       visual information in memory is beneficial. (2)
with the real-world conversational memory for          Memory Organization Importance: highlight-
MLLM agents. Current conversation benchmarks           ing the necessity of principled multimodal mem-
tend to fall into two disjoint categories. On one      ory organization and maintenance. (3) Memory
hand, text-only conversational memory bench-           Reasoning and Knowledge Management Limita-
marks evaluate memory over multi-session dia-          tions: existing multimodal memory models strug-
logues but discard visual modality (Wu et al., 2025;   gle in reasoning-intensive settings, as well as in
Hu et al., 2025a). On the other hand, localized        handling knowledge updates and conflicts. (4) Ef-
multimodal context understanding benchmarks            ficiency Bottleneck: multimodal memory overall
introduce the visual modality but typically focus      introduces larger storage and retrieval overhead
on short-horizon understanding within only one         that may hinder practical deployment.
session, failing to assess cross-session information      Our contributions are summarized as follows:
evolution and management (Liu et al., 2024; Xue
                                                       • New Scenario & Dataset: We formulate multi-
et al., 2025). This limitation makes them unsuit-
                                                         modal long-term conversational memory as an
able for memory evaluation. Consequently, current
                                                         evolving system that spans multiple sessions,
conversational benchmarks still fall short in evalu-
                                                         modalities, and memory functions, and build a
ating how agents organize, maintain, and retrieve
                                                         customized conversational dataset.
multimodal memory over extended conversational
timelines, where visual and linguistic information     • Evaluation Framework: We propose a new eval-
interact dynamically.                                    uation framework that systematically assesses
   To address this gap, we propose Mem-Gallery,          multimodal long-term conversational memory
a benchmark designed to evaluate multimodal long-        across memory extraction & adaptation, reason-
term conversational memory in MLLM agents sys-           ing, and knowledge management.
tematically, as described in Figure 1. Mem-Gallery     • Benchmark Takeaways: Through extensive
firstly introduces a new dataset of multi-session        benchmarking, we reveal key advantages and
conversations grounded in both images and text,          limitations of existing memory designs in multi-
reflecting daily and domain-specific knowledge.          modal long-term conversations, providing action-
Each conversation spans a long interaction horizon,      able insights for future research.
where information is incrementally introduced, ref-
erenced, and updated. Based on this dataset, we        2     Related Works
structure evaluation tasks into three key functional
dimensions of memory. These three dimensions           2.1    Multi-Round Conversational Benchmark
correspond to the core stages of long-term conver-     A number of dialogue benchmarks have been pro-
sational memory in real-world agents: acquiring us-    posed in recent years that can be used to evaluate
able memory, reasoning over evolving multimodal        memory capabilities, such as LoCoMo (Maharana
evidence, and regulating memory under dynamic          et al., 2024), LongMemEval (Wu et al., 2025), and
and potentially inconsistent states. Specifically,     MemoryAgentBench (Hu et al., 2025a). However,
Mem-Gallery evaluates: (1) Memory Extraction           most of these benchmarks are text-only and do not
and Adaptation, which includes multimodal fac-         provide an evaluation of multimodal capabilities.
tual retrieval, visual-centric search, and test-time   Although several localized multimodal dialogue
learning over long multimodal histories. (2) Mem-      benchmarks have been introduced recently, like
ory Reasoning, which evaluates how agents con-         MMDU (Liu et al., 2024) and MMRC (Xue et al.,
duct reasoning over multimodal memory clues,           2025), they are single-session only and lack the
including temporal reasoning, visual-centric rea-      multi-session conversation structure. Therefore,
soning, and multi-entity reasoning. (3) Memory         they are used to assess multi-round context under-
Knowledge Management, which examines the               standing abilities rather than long-term memory.
ability to resolve knowledge contradictions, detect       We compare Mem-Gallery with representative
conflicts, and appropriately refuse to answer when     related works in Table 1. Overall, prior benchmarks
information is missing, outdated, or inconsistent.     exhibit a structural misalignment with multimodal
   Benchmarking across thirteen memory mecha-          long-term memory evaluation, either overlooking
nisms reveals several findings. (1) Multimodal         visual information, lacking multi-session structure,
Information Effectiveness: explicitly preserving       or failing to support the assessment of multimodal
Table 1: Comparison between Mem-Gallery with representative conversational benchmarks. ✓: Satisfies; ✗: Does
             ✗ : Text modality only.
not satisfy; ✓
                                                          Conversational Characteristics                      Extract.&Adapt.                   Reasoning                Management
 Benchmark
                                               A. Round       A. Img.   Multi-Sess.       MM Info.            FR   VS        TTL           TR          VR     MR        KR       CD      AR
 DuLeMon (Xu et al., 2022)                        8.16          —             ✓                  ✗            ✗
                                                                                                              ✓     ✗           ✗          ✗           ✗        ✗        ✗       ✗        ✗
 DialogBench (Ou et al., 2024)                    7.48          —             ✓                  ✗            ✗
                                                                                                              ✓     ✗           ✗          ✗           ✗        ✗
                                                                                                                                                                ✓        ✗
                                                                                                                                                                         ✓       ✗        ✗
 MemoryBank (Zhong et al., 2024)                  3.77          —             ✓                  ✗            ✗
                                                                                                              ✓     ✗           ✗          ✗
                                                                                                                                           ✓           ✗        ✗        ✗       ✗        ✗
 MMDU (Liu et al., 2024)                          14.95        3.83           ✗                  ✓            ✓     ✗           ✗          ✗           ✓        ✓        ✗       ✗        ✗
 LoCoMo (Maharana et al., 2024)                   10.81        3.35           ✓                  ✓            ✓     ✗           ✗          ✗
                                                                                                                                           ✓           ✗        ✓        ✗       ✗        ✗
                                                                                                                                                                                          ✓
 LOCCO (Jia et al., 2025)                         4.77          —             ✓                  ✗            ✗
                                                                                                              ✓     ✗           ✗          ✗           ✗        ✗        ✗       ✗        ✗
 LongMemEval (Wu et al., 2025)                    5.19          —             ✓                  ✗            ✗
                                                                                                              ✓     ✗           ✗          ✗
                                                                                                                                           ✓           ✗        ✗
                                                                                                                                                                ✓        ✗
                                                                                                                                                                         ✓       ✗        ✗
                                                                                                                                                                                          ✓
 MemoryAgentBench (Hu et al., 2025a)              9.55          —             ✓                  ✗            ✗
                                                                                                              ✓     ✗           ✗
                                                                                                                                ✓          ✗
                                                                                                                                           ✓           ✗        ✗
                                                                                                                                                                ✓        ✗
                                                                                                                                                                         ✓       ✗        ✗
 MMRC (Xue et al., 2025)                          12.90        2.90           ✗                  ✓            ✓     ✗           ✗          ✓           ✓        ✓        ✗
                                                                                                                                                                         ✓       ✗        ✓
 Mem-Gallery (Ours)                               16.51        4.18           ✓                  ✓            ✓     ✓          ✓           ✓           ✓        ✓        ✓       ✓        ✓
* A. Round: average user-assistant dialogue round numbers per session. A. Img.: average image numbers per session. Multi-Sess. & MM Info.: whether the dataset
has basic characteristics of multi-session and multimodal information. This property serves as a fundamental prerequisite for multimodal long-term conversational
memory. For the three evaluation dimensions: (i) Memory extraction & adaptation includes factual retrieval (FR), visual-centric search (VS), and test-time learning
(TTL) subtasks. (ii) Memory reasoning includes temporal reasoning (TR), visual-centric reasoning (VR), and multi-entity reasoning (MR) subtasks. (iii) Memory
knowledge management includes knowledge resolution (KR), conflict detection (CD), and answer refusal (AR) subtasks.

                                                                                                                        w/o Visual Info.           with Visual Info.
memory functionalities. Among them, LoCoMo                                                                     LoCoMo                                            Mem-Gallery (Ours)
                                                                                                                                                                                      +22.47%
is one of the few that incorporates visual infor-                                        0.36                  -2.11%                           0.60
                                                                                         0.30                                                   0.54                   +15.42%
mation and a multi-session structure. However, it                                                                            +1.81%
                                                                                         0.24                                                   0.48
                                                                                    F1




                                                                                                                                           F1
supports a very restricted multimodal memory func-                                       0.18                                                   0.42
tionality evaluation, which is thus always used for                                               -0.63%                                                +5.36%
                                                                                         0.12                                                   0.36
text-only memory evaluation (Xu et al., 2025; Fang                                       0.06
                                                                                                Full Memory    MemGPT         A-Mem
                                                                                                                                                0.30
                                                                                                                                                       Full Memory     MemGPT         A-Mem
et al., 2025). Furthermore, Figure 2 shows that in-
corporating visual information in LoCoMo yields                                     Figure 2: Effectiveness analysis of visual information
marginal or inconsistent gains, indicating that its                                 with representative memory models. Compared with
evaluation tasks can largely be solved without vi-                                  LoCoMo (Maharana et al., 2024), visual information
                                                                                    plays a much more critical role in Mem-Gallery.
sual clues and thus lack sufficient capacity to assess
multimodal long-term memory. This highlights the
                                                                                    M3-Agent (Long et al., 2025). Although recent
need for a new benchmark that systematically in-
                                                                                    M3-Bench (Long et al., 2025) has taken a step
tegrates task-critical multimodal information into
                                                                                    toward evaluating multimodal long-term memory
multi-session conversations, enabling a comprehen-
                                                                                    with long-video QA, its setup differs fundamen-
sive evaluation of multimodal memory capabilities.
                                                                                    tally from conversational memory with multi-round
                                                                                    interactions. Key challenges to long-term conver-
2.2      Long-Term Agent Memory                                                     sational memory, such as multimodal information
Real-world tasks typically require agents to inter-                                 being incrementally introduced, referenced, and
act with their environments in a multi-round and                                    regulated across sessions for personalized assis-
dynamic manner, e.g., multi-round dialogues in                                      tants, remain largely underexplored.
conversational agents, making long-term memory
an important capability for agents (Zhang et al.,                                   3           Mem-Gallery Benchmark
2025b; Wei et al., 2025). Prior works have ex-                                      As shown in Figure 3, we describe Mem-Gallery
plored different aspects of memory design for such                                  from three aspects: (1) the benchmark construction,
agents. For example, Generative agents (Park et al.,                                (2) the unified conversational environment, and (3)
2023) introduced the concept of memory flow for                                     the evaluation framework and task taxonomies.
social event simulation. A-Mem (Xu et al., 2025)
and MemoryOS (Kang et al., 2025) designed agen-                                     3.1           Benchmark Construction
tic memory construction and maintenance mecha-                                      The statistics of the new dataset can be found in
nisms. However, existing methods primarily focus                                    Table 2. Detailed dataset construction and statistics
on the textual modality, while real-world mem-                                      can be found in Appendix A.2.
ory often requires the joint integration of mul-
timodal information. Consequently, multimodal                                       3.1.1 Conversation Data
long-term memory has recently garnered growing                                      To support realistic long-term memory evaluation,
interest, such as ViLoMem (Bo et al., 2025) and                                     the conversation data is organized as coherent
                                                                                                                                     Conversational
                                            Benchmark Construction                                                                                                                  Evaluation Framework
                                                                                                                                Environment for Memory
                                                Conversation Generation                                                      Average 198.1 Rounds Per Conversation
                                                                                                                             Average 12.0 Sessions Per Conversation
             Data Sourcing                               Data Synthesis                     Quality Assurance
           (1) Base Data Curation
                                                                                           (1) LLM Checking & Editing           Multi-Session Conversation
                                                                                                                                                                                  Memory Retrieval & Answer
                                                                                                                                                                                Generation for Evaluation Tasks
                                                                Conv. Generation (with
         (2) Background Generation                               New-Created Story)                                                                            Multi-Modal
                            New Story                                                     (2) Human Checking & Editing                                              …
     Role Profile            Creation             MLLM-based
                                                    Creator                                                                                                    Multi-Round
                           Relevant Topic
                                                               Conv. Organization (with
                             Clustering
                                                                  Clustered Dialogs)                                                                                 …
                                                                                                                                                                                   Comprehensive Evaluation
       Topics

                                            Evaluation Data Generation                                                                                               …

                                                                                                                                                              Multi-Session
                        Evaluation Taxonomy                                Data Synthesis Quality Assurance
                Memory Extraction & Adaptation                                                                                                                                  Performance Retrieval Efficiency
                                                                                                                                                                                  Metrics    Metrics    Metrics
                                                                                                                                                                                   F1, EM,   Recall, Organization
                                                                                                                                                                                  BLEU-1,   Precision, & Retrieval
    Factual Retrieval     Visual-centric Search    Test-Time Learning
                          Memory Reasoning                                Conversations QA Generation   LLM
                                                                                                                                  …                          10+ Conversation
                                                                                                                                                              Memory Agents
                                                                                                                                                                                 LLM-Judge     Hit        Time

                                                                                           (LLM)      Checking
                                                                                                                            MLLM Backbone                   Memory
   Temporal Reasoning Visual-centric Reasoning Multi-entity Reasoning                                                                                   Organization and
                Memory Knowledge Management                                                                                                              Maintenance                 Benchmark Evaluation
                                                                                                         Human                                                                            Analysis
                                                                            Evaluation    QA Generation Checking
                                                                            Taxonomy        (Human)                                       Stored Memory
   Knowledge Resolution    Conflict Detection      Answer Refusal




                                                      Figure 3: The overall pipeline of our proposed Mem-Gallery.

    Table 2: Statistics of the Mem-Gallery dataset.                                                                      erate the text-part multi-session dialogues condi-
 Mem-Gallery                        Aspect                                                    Statistics                 tioned on these specifications. Annotators then
                                    Sessions                                                      240                    insert appropriate images at suitable positions to
 Conversation Data                  Dialogue Rounds                                              3,962                   ensure multimodal dependency. On the other hand,
                                    Included Images                                              1,003
                                                                                                                         we perform conversation organization via topic-
                                    QA Pairs with Annotated Clues                                1,711
 Evaluation Data                                                                                                         based clustering, where existing single-session mul-
                                    Included Images                                               487
                                                                                                                         timodal dialogues in MMRC (Xue et al., 2025) are
                                                                                                                         grouped into candidate multi-session conversations.
multi-session interactions, with consistent user per-                                                                    Specifically, LLMs are used to extract representa-
sonas and tightly coupled visual & textual content.                                                                      tive topic keywords from single-session dialogues,
   Data Sourcing. We first curate base materi-                                                                           followed by clustering under constraints on topical
als from open-source resources, including images                                                                         relevance, fluency, and length. The clustered con-
and parts of the textual content (details in Ap-                                                                         versations are then reordered and refined to form
pendix A.2.2). Specifically, we prioritize pub-                                                                          coherent long-term interaction sequences. Details
licly available data that cover diverse everyday and                                                                     and the example can be found in Appendix A.2.3.
domain-specific scenarios, with sufficient visual de-                                                                       Quality Assurance. We apply a two-stage qual-
tail and semantic richness to support long-horizon                                                                       ity assurance process. In the first stage, advanced
multimodal grounding, rather than isolated or short-                                                                     LLMs automatically check and revise conversa-
context understanding. On top of the curated base                                                                        tions for coherence, fluency, and factual consis-
data, we further generate structured conversational                                                                      tency. In the second stage, human annotators care-
backgrounds, such as user role profiles and conver-                                                                      fully review each conversation and further refine
sation topics. These backgrounds serve as control-                                                                       the content through manual editing. This ensures
lable anchors for conversation synthesis, ensuring                                                                       the final conversation data maintains high multi-
topic diversity and long-range coherence.                                                                                modal quality and realistic conversation dynamics.
   Data Synthesis. Based on the sourced mate-
rials, we synthesize multi-session conversations                                                                         3.1.2 Evaluation Data
through two complementary strategies, each pro-                                                                          Based on the curated conversations, we systemat-
ducing valid multi-session conversations that are                                                                        ically generate evaluation data. It consists of QA
later unified into the final dataset. On one hand,                                                                       pairs based on a well-predefined evaluation taxon-
we adopt conversation generation with newly cre-                                                                         omy (details in Section 3.3) for each conversation.
ated stories as conversation backgrounds, where                                                                             Data Synthesis. Evaluation QA pairs are also
human annotators design the story outline and                                                                            constructed through two complementary ways,
inter-session transition logic. Advanced LLMs gen-                                                                       which jointly contribute to the final evaluation set.
On one hand, LLMs are prompted with selected
conversation histories and task descriptions to gen-
                                                                               Factual Retrieval      Multi-entity
erate QA candidates. On the other hand, human an-                                   12.8%              Reasoning
                                                                                                        12.0%
notators construct QA pairs by reviewing each con-
versation and designing targeted questions. These                                                                     Visual-centric
                                                                                                        Memory          Reasoning
two ways jointly ensure both coverage, difficulty,                                                     Reasoning          10.2%
                                                                                                        29.4%
and diversity of evaluation instances. In addition,           Visual-centric   Memory
                                                                 Search        Extract.
                                                                  17.9%                       Total:
for each QA pair, we explicitly annotate evidence                              & Adapt.
                                                                                            1,711 QA                        Temporal
                                                                                50.4%                                       Reasoning
clues for the correct answer that specify dialogue                                            Pairs
                                                                                                       Memory                 7.2%
                                                                                                      Knowledge
turns in which the relevant information is refer-                                                    Management
                                                                                                        20.2%
enced for the answer. These annotated clues facil-                                                                     Answer
                                                                                                                       Refusal
itate fine-grained analysis of memory behaviors,                                                                       10.8%
                                                                         Test-Time Learning                      Conflict
e.g., retrieval details (analyzed in Appendix A.6.2),                          19.7%                            Detection
                                                                                                   Knowledge      4.7%
beyond final task performance. Data synthesis de-                                                  Resolution
                                                                                                     4.7%
tails can be found in Appendix A.2.4.
   Quality Assurance. Generated QA pairs also
                                                        Figure 4: Taxonomy and distribution of evaluation tasks.
undergo a two-stage verification. The same as con-
versation generation, LLMs are first used to check      retrieval and answer generation are then performed
answer correctness and question clarity. This veri-     based on the accumulated memory. The problem
fication step is followed by careful human revision.    definition can be found in Appendix A.1.
3.2   Conversational Environment for Memory             3.3      Evaluation Framework
Following previous works (Maharana et al., 2024),       As shown in Figure 4, we design three task fam-
memory models are evaluated based on the input          ilies to systematically evaluate multimodal long-
conversation. In our setting, memory agents must        term memory as an evolving capability in agentic
go beyond storing textual information in existing       systems over extended conversations. Specifically,
benchmarks and store and associate visual content       memory extraction and adaptation assess whether
with textual content. For textual memory, image         agents can retrieve relevant multimodal informa-
captions are provided to preserve the visual content.   tion and adapt it as memory accumulates. Memory
   Conversation Structure. As shown in Figure 8,        reasoning examines whether agents can integrate
each conversation spans multiple sessions with tem-     and reason over memory, accounting for temporal
poral gaps. Within each session, agents engage in       dependencies, visual evidence, and multiple en-
multi-round multimodal dialogues, while across          tities. Memory knowledge management focuses
sessions, information may be updated or contra-         on how agents regulate long-term memory under
dicted. This design prevents reliance on short-term     dynamic states, like handling inconsistencies, re-
context and explicitly requires memory to persist       solving conflicts, and refusing to answer when in-
and evolve across long-term conversational bound-       formation is outdated, contradictory, or incomplete.
aries. Visual content in the conversation is not
limited to single-round perception but may be ref-      3.3.1 Memory Extraction & Adaptation
erenced and integrated with textual context across      At a fundamental level, memory agents must pos-
sessions. Consequently, agents must integrate mul-      sess the capability of memory extraction and adap-
timodal clues distributed over extended timelines,      tation, ensuring that stored information can be ef-
rather than treating images as isolated observations.   fectively utilized (Hu et al., 2025a). Based on the
   Benchmarking Memory Methods. We bench-               properties, we design three subtask categories. (1)
mark thirteen representative memory models under        Factual Retrieval: evaluate the ability to accurately
a unified setting. As conversations progress, agents    recall factual details (e.g., user preferences and past
accumulate an expanding memory. To ensure a fair        events) from multimodal interaction histories. (2)
and controlled comparison across diverse designs,       Visual-centric Search: evaluate whether the model
all evaluated models follow a unified memory ac-        can identify or retrieve specific visual instances
cumulation granularity and protocol (Kang et al.,       from memory (e.g., shared images). (3) Test-Time
2025), where information is incrementally stored        Learning: measure the ability to adapt its memory
along with the conversational timeline. Memory          to unseen multimodal examples at inference time.
3.3.2 Memory Reasoning                                 et al., 2025), as well as representative closed-source
In addition to memory extraction, agents should be     MLLMs, namely GPT-4.1-Nano (Achiam et al.,
equipped with reasoning capabilities over the mem-     2023) and Gemini-2.5-Flash-Lite (Comanici et al.,
ory to address complex tasks (Ke et al., 2025). We     2025), as backbone models. We adopt Qwen2.5-
design three subtasks accordingly. (1) Temporal        VL-7B-Instruct as our default MLLM backbone.
Reasoning: assess whether the model can synthe-
                                                       Memory Models. We comprehensively include
size and reason over temporally dependent ques-
                                                       thirteen representative approaches, including eight
tions from multimodal memory. (2) Visual-centric
                                                       textual memory methods and five multimodal mem-
Reasoning: test the model’s capacity to retrieve
                                                       ory methods. Specifically, the textual mem-
and utilize visual information as cues for reasoning
                                                       ory includes Full Memory (Text), First-in-first-
on multimodal memories. (3) Multi-entity Reason-
                                                       out (FIFO), NaiveRAG, Generative Agents (Park
ing: reasoning with multiple entities from memory,
                                                       et al., 2023), Reflexion (Shinn et al., 2023),
where each can be textual or visual.
                                                       MemGPT (Packer et al., 2023), A-Mem (Xu et al.,
3.3.3 Memory Knowledge Management                      2025), and MemoryOS (Kang et al., 2025). The
Unlike task-oriented scenarios such as web naviga-     multimodal memory includes Full Memory (Mul-
tion, conversations are inherently more open-ended.    timodal), MuRAG (Chen et al., 2022), Universal-
Users may update previous information or provide       RAG (Yeo et al., 2025), NGM (Fisher, 2025), and
incorrect details inadvertently during dialogues (Hu   AUGUSTUS (Jain et al., 2025). Details of these
et al., 2025a). To evaluate these challenges, we de-   models are provided in Appendix A.4. For methods
sign the following subtasks. (1) Knowledge Resolu-     that require a top-K retriever, we adopt a default
tion: Examine the ability to correctly update stored   retrieval size of K=10. To enable a fair comparison
knowledge when new, contradictory information          with multimodal memory systems, we provide tex-
appears in the dialogue, maintaining consistency       tual memory with high-quality image captions gen-
over time. (2) Conflict Detection: Test whether the    erated by GPT-5.1. The detailed benchmark setup
model can detect conflicts between newly observed      for fair comparison is shown in Appendix A.5.
information and existing memory. (3) Answer Re-
fusal: Assess the model’s capability to abstain or     4.1.2    Evaluation Metrics
refuse to answer when the requested information is     Following existing works, we evaluate memory
unsupported by prior memory.                           performance using F1, BLEU-1, EM, and LLM-as-
                                                       a-Judge metrics. For LLM-as-a-Judge, we adopt
4     Benchmarking Analysis                            Qwen-2.5-72B-Instruct (Bai et al., 2025) as the
We conduct extensive experiments on our proposed       judging model. Note that the conflict detection task
Mem-Gallery for the following research questions.      explicitly requires models to output either “Yes” or
• RQ1: How effective are different multimodal          “No”. Since MLLMs possess instruction-following
  storage designs?                                     capability, the values of all metrics are identical for
                                                       this task. For retrieval analysis, we adopt the widely
• RQ2: How important are memory organization           used Recall, Precision, and Hit as the metrics. The
  strategies in multimodal memory systems?             details can be found in Appendix A.5.4.
• RQ3: What are the strengths and weaknesses of
  different memory models across task types?           4.2     Main Results (RQ1-RQ3)
• RQ4: What are the runtime efficiency character-      Table 3 illustrates the main benchmarking results.
  istics of different memory approaches?               We can have the following observations.
• RQ5: How does the number of retrieved memory            RQ1: Explicit multimodal memory preserva-
  entries affect the overall performance?              tion is critical, but increased architectural com-
                                                       plexity does not necessarily bring better perfor-
4.1    Evaluation Setup
                                                       mance. Although with high-quality image cap-
4.1.1 Model Implementation                             tions, textual memory baselines still generally ex-
MLLM Backbones. To ensure a broad coverage             hibit a performance gap compared to multimodal
of model capacities, we use representative open-       approaches, particularly in memory extraction and
source MLLMs at two model scales, Qwen2.5-             adaptation. MuRAG, a simple multimodal method,
VL-3B-Instruct and Qwen2.5-VL-7B-Instruct (Bai         can achieve the best overall performance among
Table 3: Main evaluation results on our Mem-Gallery based on Qwen-2.5-VL-7B. The best and second-performed
memory model(s) are highlighted with orange and blue backgrounds. Results on Qwen-2.5-VL-3B, GPT-4.1-Nano,
and Gemini-2.5-Flash-Lite can be found in Table 8, Table 9, and Table 10, respectively.
                       Qwen-2.5-VL-7B    Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT     A-Mem    MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                F1        0.2376       0.1446    0.5852      0.2424      0.2391     0.5928     0.6072    0.6244     0.2150     0.6724      0.6632      0.6364    0.6162
                              BLEU-1      0.1865       0.1000    0.5045      0.1862      0.1903     0.5098     0.5138    0.5354     0.1626     0.5755      0.5658      0.5606    0.5309
                       FR      EM         0.0913       0.0457    0.3059      0.0868      0.0913     0.3288     0.2922    0.3470     0.0685     0.3607      0.3470      0.3744    0.3288
                             LLM-Judge    0.2626       0.1324    0.7763      0.2945      0.2626     0.8539     0.7808    0.8265     0.2260     0.8790      0.8744      0.8082    0.8082
Extract. & Adapt.




                                F1        0.1992       0.0612    0.7592      0.2970      0.1954     0.6239     0.7681    0.7853     0.1658     0.8818      0.8708      0.8531    0.8499
                              BLEU-1      0.1873       0.0549    0.7063      0.2616      0.1840     0.5834     0.6880    0.7181     0.1473     0.8442      0.8343      0.8145    0.7942
                       VS      EM         0.1601       0.0392    0.5686      0.2124      0.1569     0.4118     0.5882    0.6046     0.1078     0.6699      0.6699      0.6863    0.6536
                             LLM-Judge    0.1961       0.0556    0.7402      0.2958      0.1895     0.5964     0.7369    0.7729     0.1683     0.8856      0.8611      0.8480    0.8399
                                F1        0.4500       0.3351    0.6526      0.4851      0.4486     0.2924     0.6336    0.5484     0.4147     0.8177      0.7824      0.7817    0.7913
                              BLEU-1      0.3799       0.2692    0.5835      0.4150      0.3798     0.2295     0.5595    0.4697     0.3477     0.7449      0.7103      0.7168    0.7206
                       TTL     EM         0.2374       0.1365    0.4718      0.2967      0.2374     0.1009     0.4362    0.3561     0.2107     0.6172      0.6142      0.6024    0.6113
                             LLM-Judge    0.7092       0.6677    0.8457      0.7582      0.7033     0.7092     0.7997    0.7715     0.7107     0.9006      0.8501      0.9110    0.8932
                                F1        0.2545       0.1549    0.4887      0.2742      0.2553     0.5661     0.5604    0.5497     0.2294     0.5833      0.5460      0.5425    0.5800
                              BLEU-1      0.2363       0.1316    0.4587      0.2473      0.2363     0.5326     0.5361    0.5240     0.2065     0.5537      0.5137      0.5143    0.5527
                       TR      EM         0.1545       0.0894    0.3496      0.1626      0.1545     0.3496     0.4065    0.3821     0.1382     0.4309      0.4065      0.4146    0.4390
                             LLM-Judge    0.2805       0.1463    0.6382      0.3252      0.2764     0.8008     0.6951    0.7195     0.2480     0.7724      0.7520      0.7195    0.7398
                                F1        0.2552       0.1207    0.3022      0.1955      0.2594     0.4593     0.4477    0.4280     0.2015     0.4818      0.4879      0.4615    0.3866
Reasoning




                              BLEU-1      0.2442       0.1005    0.2873      0.1815      0.2480     0.4459     0.4331    0.4126     0.1912     0.4625      0.4682      0.4428    0.3726
                       VR      EM         0.2011       0.0690    0.1954      0.1207      0.2011     0.3851     0.3563    0.3391     0.1609     0.3793      0.3793      0.3678    0.2931
                             LLM-Judge    0.3046       0.1408    0.3793      0.2471      0.3046     0.6149     0.5747    0.5805     0.2586     0.6092      0.5977      0.5460    0.4856
                                F1        0.2411       0.1745    0.4640      0.2450      0.2428     0.4367     0.5000    0.4490     0.2101     0.5007      0.5013      0.4746    0.4866
                              BLEU-1      0.1739       0.1184    0.3543      0.1784      0.1770     0.3347     0.3908    0.3446     0.1429     0.3903      0.3868      0.3635    0.3778
                       MR      EM         0.0340       0.0146    0.0874      0.0243      0.0340     0.0631     0.0728    0.0728     0.0194     0.0874      0.0971      0.0728    0.0825
                             LLM-Judge    0.2985       0.1602    0.7791      0.3350      0.3058     0.8204     0.8083    0.8204     0.2791     0.8447      0.8422      0.7840    0.8228
                                F1        0.2354       0.1697    0.3595      0.2674      0.2354     0.4292     0.4515    0.5031     0.2181     0.4601      0.4336      0.3944    0.3752
                              BLEU-1      0.2005       0.1424    0.2986      0.2341      0.2000     0.3741     0.3992    0.4479     0.1883     0.4073      0.3772      0.3444    0.3159
                       KR      EM         0.1235       0.0864    0.1605      0.1235      0.1235     0.2099     0.2469    0.2716     0.1235     0.2593      0.2099      0.2099    0.1728
Knowledge Management




                             LLM-Judge    0.3395       0.2469    0.6358      0.3457      0.3395     0.7407     0.6667    0.7593     0.2840     0.7840      0.7222      0.6728    0.6728
                                F1        0.3457       0.3580    0.3457      0.3210      0.3333      0.3580    0.3333    0.3333     0.3580     0.3704      0.3457      0.3457    0.3210
                              BLEU-1      0.3457       0.3580    0.3457      0.3210      0.3333      0.3580    0.3333    0.3333     0.3580     0.3704      0.3457      0.3457    0.3210
                       CD      EM         0.3457       0.3580    0.3457      0.3210      0.3333      0.3580    0.3333    0.3333     0.3580     0.3704      0.3457      0.3457    0.3210
                             LLM-Judge    0.3457       0.3580    0.3457      0.3210      0.3333      0.3580    0.3333    0.3333     0.3580     0.3704      0.3457      0.3457    0.3210
                                F1         0.9958      1.0000    0.9581      0.9841      0.9958     0.9849     0.9278    0.9845     0.9946     0.9418      0.9473      0.9580    0.9460
                              BLEU-1       0.9953      1.0000    0.9575      0.9839      0.9953     0.9844     0.9257    0.9841     0.9946     0.9409      0.9466      0.9574    0.9459
                       AR      EM          0.9946      1.0000    0.9565      0.9837      0.9946     0.9837     0.9239    0.9837     0.9946     0.9402      0.9457      0.9565    0.9457
                             LLM-Judge     0.9783      0.9837    0.9402      0.9674      0.9783     0.9674     0.9375    0.9674     0.9837     0.9375      0.9429      0.9457    0.9429
                                F1        0.3625       0.2724    0.5974      0.3825      0.3619     0.5282     0.6228    0.6109     0.3354     0.6966      0.6827      0.6691    0.6610
                              BLEU-1      0.3279       0.2408    0.5441      0.3422      0.3279     0.4792     0.5629    0.5534     0.2999     0.6432      0.6286      0.6200    0.6069
         Overall               EM         0.2519       0.1835    0.4161      0.2613      0.2507     0.3402     0.4296    0.4278     0.2279     0.4985      0.4927      0.4944    0.4757
                             LLM-Judge    0.4331       0.3369    0.7241      0.4643      0.4307     0.7306     0.7431    0.7613     0.4129     0.8229      0.8016      0.7861    0.7797



well-designed textual memory models. Specifically,                                                            and MuRAG. As visual information is far more
it achieves 11.85%, 7.69%, 12.29%, and 29.06% F1                                                              token-heavy than text (Zhang et al., 2025a), under
improvement over the best-performed textual mem-                                                              the token limit, naïvely accumulating multimodal
ory on overall, FR, VS, and TTL tasks, respectively.                                                          content can introduce irrelevant visual noise, crowd
Furthermore, MuRAG and UniversalRAG, which                                                                    out informative text, and thus hurt performance.
simply preserve multimodal information without                                                                Besides, compared to MuRAG and UniversalRAG,
structured memory organizations, achieve stronger                                                             although existing models like NGM and AUGUS-
performance than complex multimodal memory                                                                    TUS organize multimodal information in a struc-
systems. For example, MuRAG outperforms 4.11%                                                                 tured manner, they still lack effective strategies for
and 5.39% overall F1 improvements over NGM and                                                                long-term conversational scenarios. Moreover, ex-
AUGUSTUS, respectively. This contrast suggests                                                                isting multimodal methods still lack agentic main-
that, beyond focusing on the memory architecture                                                              tenance strategies like A-Mem and MemoryOS,
itself, how to effectively preserve multimodal infor-                                                         which also constrain their performance.
mation so as to maximize the utility of the memory                                                               RQ3-1: Existing multimodal memory meth-
architecture remains an open challenge.                                                                       ods struggle on memory reasoning tasks. Al-
    RQ2: Visual information faces harsher to-                                                                 though multimodal memory methods overall out-
ken consumption, making principled organi-                                                                    perform textual memory approaches, we observe
zation and maintenance critical. Without the                                                                  that their advantages are not pronounced in scenar-
proper organization, inputting all multimodal infor-                                                          ios requiring reasoning. This is particularly evident
mation into the context leads Full Memory (MM)                                                                in multi-entity reasoning and temporal reasoning
to perform even worse than Full Memory (Text)                                                                 tasks. Notably, even in the visual-centric reasoning
in both memory extraction and reasoning tasks.                                                                task that explicitly relies on visual clues, MemGPT,
Overall, Full Memory (MM) performs 8.08% and                                                                  a textual memory method, can achieve near-optimal
51.85% worse F1 score than Full Memory (Text)                                                                 performance. This suggests that existing multi-
                                                       Textual Memory
                                                       Multimodal Memory
                                                                                                                                                          Table 4: Retrieval metrics on different size K.
                                      103
Memory time (s)


                                                                                                                                                             Recall@K       K=5      K=10     K=15     K=20
                                      101
                                                                                                                                                             Gen. Agent     0.1707   0.2385   0.2854   0.3153
                            10 1                                                                                                                             NaiveRAG       0.5381   0.6723   0.7420   0.7877
                                                                                                                                                             MuRAG          0.7506   0.8601   0.8990   0.9228
                                                      t)   FIFO          G      nt      n     T            OS       M)      AG     AG           S
                                                 (Tex                 eRA n. Age eflexio emGP        em
                                                                                                  A-M emory Full (M      MuR ersalR     NGM USTU             UniversalRAG   0.7311   0.8411   0.8781   0.8998
                                            Full                  Naiv    Ge       R      M                                                AUG
                                                                                                        M                   Univ                             NGM            0.6192   0.7475   0.7892   0.8065
    Retrieval & generation time (s)




                                      104                                                                                                                    AUGUSTUS       0.6729   0.7529   0.7785   0.7860
                                                                                                                                                             Precision@K    K=5      K=10     K=15     K=20
                                                                                                                                                             Gen. Agent     0.0955   0.0684   0.0544   0.0453
                                      103                                                                                                                    NaiveRAG       0.2694   0.1757   0.1315   0.1047
                                                                                                                                                             MuRAG          0.3686   0.2220   0.1572   0.1220
                                                      t)   FIFO          G      nt      n     T            OS       M)      AG     AG           S
                                                 (Tex                 eRA n. Age eflexio emGP        em
                                                                                                  A-M emory Full (M      MuR ersalR     NGM USTU             UniversalRAG   0.3691   0.2206   0.1555   0.1204
                                            Full                  Naiv    Ge       R      M                                                AUG
                                                                                                        M                   Univ                             NGM            0.3564   0.3457   0.3450   0.3424
                                                                                                                                                             AUGUSTUS       0.3341   0.2488   0.2213   0.2124
 Figure 5: Efficiency comparison results on information
 memorization time (top) and memory retrieval & answer                                                                                              and MemoryOS. This suggests that introducing
 generation time (bottom) in seconds with the log scale.                                                                                            multimodal signals fundamentally increases system
                                                                                                                                                    complexity, regardless of whether advanced mem-
modal memory approaches still primarily focus on                                                                                                    ory organization strategies are employed. Thus,
the storage and retrieval of multimodal informa-                                                                                                    while multimodal memory can provide richer con-
tion, yet how to effectively reason over multimodal                                                                                                 textual information, it also introduces considerable
contents in memory remains largely unexplored.                                                                                                      inference-time overhead, which may limit its practi-
   RQ3-2: Existing methods show limitations in                                                                                                      cality in long-horizon or real-time agent scenarios.
scenarios involving information updates or con-
flicts. From the memory knowledge management                                                                                                        4.4    Retriever Analysis (RQ5)
results, we derive two key observations. First, there                                                                                               Table 4 shows multimodal retrievers like MuRAG
is a trade-off in refusal behavior. Methods with                                                                                                    and UniversalRAG achieve substantially higher
weaker memory, e.g., FIFO, tend to exhibit stronger                                                                                                 clue recall. Though recall improves as K increases,
refusal performance, as they default to refusing                                                                                                    the expanded retrieval coverage does not con-
when relevant information cannot be retrieved. In                                                                                                   sistently translate into better QA performance.
contrast, methods with stronger capabilities on                                                                                                     Figure 15 shows multimodal memory models typ-
memory extraction and reasoning show poorer re-                                                                                                     ically exhibit diminishing or saturated task gains
fusal performance, like A-Mem and MuRAG, with                                                                                                       beyond a moderate K, and may degrade at larger
7.22% and 5.82% worse F1 score than FIFO. This                                                                                                      K. Table 4 explains this discrepancy. While the
suggests that future methods need to strike a bal-                                                                                                  Recall of MuRAG and UniversalRAG increases
ance between retrieving relevant information and                                                                                                    as K grows, its Precision drops sharply, indicat-
distinguishing outdated and conflicting informa-                                                                                                    ing the introduction of substantial noise. NGM
tion to build safe and hallucination-free memory                                                                                                    and AUGUSTUS exhibit more conservative recall
systems. Second, for both knowledge resolution                                                                                                      growth with relatively stable Precision, suggesting
and conflict detection tasks, neither multimodal nor                                                                                                structured organization or relevance filtering is an
textual memory methods achieve satisfactory per-                                                                                                    effective way to balance coverage and noise. Full
formance. This indicates that designing memory                                                                                                      analysis and results are reported in Appendix A.6.2.
systems to support dynamic conflict detection and
memory correction remains an important direction.                                                                                                   5     Conclusion
   Full analysis under different MLLMs and case                                                                                                     In this paper, we present Mem-Gallery, a bench-
studies can be found in Appendix A.6.1 and A.7.                                                                                                     mark for evaluating multimodal long-term conver-
                                                                                                                                                    sational memory. By grounding multi-session con-
 4.3                                                 Efficiency Analysis (RQ4)                                                                      versations in tightly coupled visual and textual con-
 Figure 5 presents an efficiency comparison of exist-                                                                                               texts, Mem-Gallery enables systematic assessment
 ing memory methods. From the results, we can see                                                                                                   beyond prior text-centric memory or localized mul-
 that multimodal memory incurs higher overhead                                                                                                      timodal context understanding benchmarks. Exten-
 than text-only memory in general. Even a rela-                                                                                                     sive evaluation reveals several takeaways. These
 tively simple multimodal memory like MuRAG can                                                                                                     findings highlight the need for principled memory
 approach the computational cost of highly sophis-                                                                                                  organization, selective retrieval, and robust mainte-
 ticated textual memory systems, such as A-Mem                                                                                                      nance for future multimodal memory design.
Limitations                                                 next generation agentic capabilities. arXiv preprint
                                                            arXiv:2507.06261.
While Mem-Gallery provides a comprehensive
evaluation of multimodal long-term conversational         Jizhan Fang, Xinle Deng, Haoming Xu, Ziyan Jiang,
                                                             Yuqi Tang, Ziwen Xu, Shumin Deng, Yunzhi
memory, the following limitations remain. First,             Yao, Mengru Wang, Shuofei Qiao, and 1 oth-
the benchmark focuses on vision–language con-                ers. 2025. Lightmem: Lightweight and efficient
versational settings and does not explicitly cover           memory-augmented generation. arXiv preprint
other modalities such as audio or embodied signals,          arXiv:2510.18866.
which may be relevant in broader agentic scenar-          Matthew Fisher. 2025. Neural graph memory: A struc-
ios. Second, the evaluation primarily focuses on           tured approach to long-term memory in multimodal
memory-centric capabilities in long-horizon con-           agents.
versations and does not aim to exhaustively assess
                                                          Yuanzhe Hu, Yu Wang, and Julian McAuley. 2025a.
other agent behaviors, such as planning or tool use.        Evaluating memory in llm agents via incre-
We leave these extensions to future work.                   mental multi-turn interactions. arXiv preprint
                                                            arXiv:2507.05257.

References                                                Yuyang Hu, Shichun Liu, Yanwei Yue, Guibin Zhang,
                                                            Boyang Liu, Fangyi Zhu, Jiahang Lin, Honglin
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama           Guo, Shihan Dou, Zhiheng Xi, and 1 others. 2025b.
  Ahmad, Ilge Akkaya, Florencia Leoni Aleman,               Memory in the age of ai agents. arXiv preprint
  Diogo Almeida, Janko Altenschmidt, Sam Altman,            arXiv:2512.13564.
  Shyamal Anadkat, and 1 others. 2023. Gpt-4 techni-
  cal report. arXiv preprint arXiv:2303.08774.            Jitesh Jain, Shubham Maheshwari, Ning Yu, Wen-mei
                                                             Hwu, and Humphrey Shi. 2025. Augustus: An llm-
Dirk Jan Ardesch, Lianne H Scholtens, Longchuan              driven multimodal agent system with contextualized
  Li, Todd M Preuss, James K Rilling, and Martijn P          user memory. arXiv preprint arXiv:2510.15261.
  van den Heuvel. 2019. Evolutionary expansion of
  connectivity between multimodal association areas       Zixi Jia, Qinghua Liu, Hexiao Li, Yuyan Chen, and
  in the human brain compared with chimpanzees.             Jiqiang Liu. 2025. Evaluating the long-term memory
  Proceedings of the National Academy of Sciences,          of large language models. In Findings of the Associa-
  116(14):7101–7106.                                        tion for Computational Linguistics: ACL 2025, pages
                                                            19759–19777.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wen-
  bin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie          Jiazheng Kang, Mingming Ji, Zhe Zhao, and Ting Bai.
  Wang, Jun Tang, and 1 others. 2025. Qwen2.5-vl             2025. Memory os of ai agent. In Proceedings of the
  technical report. arXiv preprint arXiv:2502.13923.         2025 Conference on Empirical Methods in Natural
                                                             Language Processing, pages 25972–25981.
Weihao Bo, Shan Zhang, Yanpeng Sun, Jingjing Wu,
 Qunyi Xie, Xiao Tan, Kunbin Chen, Wei He, Xiaofan        Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen,
 Li, Na Zhao, and 1 others. 2025. Agentic learner           Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin,
 with grow-and-refine multimodal semantic memory.           Peifeng Wang, Silvio Savarese, and 1 others. 2025.
 arXiv preprint arXiv:2511.21678.                           A survey of frontiers in llm reasoning: Inference scal-
                                                            ing, learning to reason, and agentic systems. arXiv
Gongwei Chen, Xurui Zhou, Rui Shao, Yibo Lyu, Kai-          preprint arXiv:2504.09037.
  wen Zhou, Shuai Wang, Wentao Li, Yinchuan Li,
  Zhongang Qi, and Liqiang Nie. 2025. Less is more:       Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng
  Empowering gui agent with context-aware simplifica-       Yao, and Fei-Fei Li. 2011. Novel dataset for fine-
  tion. In Proceedings of the IEEE/CVF International        grained image categorization: Stanford dogs. In Proc.
  Conference on Computer Vision, pages 5901–5911.           CVPR workshop on fine-grained visual categoriza-
                                                            tion (FGVC), volume 2.
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and
 William Cohen. 2022. Murag: Multimodal retrieval-        Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad
 augmented generator for open question answering            Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhat-
 over images and text. In Proceedings of the 2022           tacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu,
 Conference on Empirical Methods in Natural Lan-            and 1 others. 2025. From generation to judgment:
 guage Processing, pages 5558–5570.                         Opportunities and challenges of llm-as-a-judge. In
                                                            Proceedings of the 2025 Conference on Empirical
Gheorghe Comanici, Eric Bieber, Mike Schaekermann,          Methods in Natural Language Processing, pages
  Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Mar-      2757–2791.
  cel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and
  1 others. 2025. Gemini 2.5: Pushing the frontier with   Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and
  advanced reasoning, multimodality, long context, and      Xiaoou Tang. 2016. Deepfashion: Powering robust
  clothes recognition and retrieval with rich annota-     Noah Shinn, Federico Cassano, Ashwin Gopinath,
  tions. In Proceedings of the IEEE conference on           Karthik Narasimhan, and Shunyu Yao. 2023. Re-
  computer vision and pattern recognition, pages 1096–      flexion: Language agents with verbal reinforcement
  1104.                                                     learning. In Advances in Neural Information Pro-
                                                            cessing Systems, volume 36, pages 8634–8652.
Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi
  Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong,           Catherine Wah, Steve Branson, Peter Welinder, Pietro
  Yu Qiao, Dahua Lin, and 1 others. 2024. Mmdu: A           Perona, and Serge Belongie. 2011. The caltech-ucsd
  multi-turn multi-image dialog understanding bench-        birds-200-2011 dataset.
  mark and instruction-tuning dataset for lvlms. In
  Advances in Neural Information Processing Systems,      Tianxin Wei, Noveen Sachdeva, Benjamin Coleman,
  volume 37, pages 8698–8733.                               Zhankui He, Yuanchen Bei, Xuying Ning, Mengting
                                                            Ai, Yunzhe Li, Jingrui He, Ed H Chi, and 1 oth-
Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan            ers. 2025. Evo-memory: Benchmarking llm agent
  Lin, Hang Li, Junbo Zhao, and Wei Li. 2025. See-          test-time learning with self-evolving memory. arXiv
  ing, listening, remembering, and reasoning: A multi-      preprint arXiv:2511.20857.
  modal agent with long-term memory. arXiv preprint
  arXiv:2508.09736.                                       Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang,
                                                            Kai-Wei Chang, and Dong Yu. 2025. Longmemeval:
Andrea I Luppi, Pedro AM Mediano, Fernando E                Benchmarking chat assistants on long-term interac-
  Rosas, Negin Holland, Tim D Fryer, John T O’Brien,        tive memory. In The Thirteenth International Con-
  James B Rowe, David K Menon, Daniel Bor, and              ference on Learning Representations.
  Emmanuel A Stamatakis. 2022. A synergistic core
                                                          Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng
  for human brain evolution and cognition. Nature
                                                             Wan, and Philip S Yu. 2023. Multimodal large lan-
  neuroscience, 25(6):771–782.
                                                             guage models: A survey. In 2023 IEEE Interna-
Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov,              tional Conference on Big Data (BigData), pages
  Mohit Bansal, Francesco Barbieri, and Yuwei Fang.          2247–2256. IEEE.
  2024. Evaluating very long-term conversational          Zidi Xiong, Yuping Lin, Wenya Xie, Pengfei He, Zirui
  memory of llm agents. In Proceedings of the 62nd          Liu, Jiliang Tang, Himabindu Lakkaraju, and Zhen
  Annual Meeting of the Association for Computational       Xiang. 2025. How memory management impacts llm
  Linguistics (Volume 1: Long Papers), pages 13851–         agents: An empirical study of experience-following
  13870.                                                    behavior. arXiv preprint arXiv:2505.16067.
Maria-Elena Nilsback and Andrew Zisserman. 2008.          Wujiang Xu, Kai Mei, Hang Gao, Juntao Tan, Zujie
 Automated flower classification over a large number       Liang, and Yongfeng Zhang. 2025. A-mem: Agentic
 of classes. In 2008 Sixth Indian conference on com-       memory for llm agents. In The Thirty-ninth Annual
 puter vision, graphics & image processing, pages          Conference on Neural Information Processing Sys-
 722–729. IEEE.                                            tems.
Jiao Ou, Junda Lu, Che Liu, Yihong Tang, Fuzheng          Xinchao Xu, Zhibin Gou, Wenquan Wu, Zheng-Yu Niu,
   Zhang, Di Zhang, and Kun Gai. 2024. Dialogbench:         Hua Wu, Haifeng Wang, and Shihang Wang. 2022.
   Evaluating llms as human-like dialogue systems. In       Long time no see! open-domain conversation with
   Proceedings of the 2024 Conference of the North          long-term persona memory. In Findings of the As-
   American Chapter of the Association for Computa-         sociation for Computational Linguistics: ACL 2022,
   tional Linguistics: Human Language Technologies          pages 2639–2650.
   (Volume 1: Long Papers), pages 6137–6170.
                                                          Haochen Xue, Feilong Tang, Ming Hu, Yexin Liu, Qi-
Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang,      dong Huang, Yulong Li, Chengzhi Liu, Zhongxing
  Shishir G Patil, Ion Stoica, and Joseph E Gonzalez.       Xu, Chong Zhang, Chun-Mei Feng, and 1 others.
  2023. Memgpt: Towards llms as operating systems.          2025. Mmrc: A large-scale benchmark for under-
  arXiv preprint arXiv:2310.08560.                          standing multimodal large language model in real-
                                                            world conversation. In Proceedings of the 63rd An-
Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Mered-      nual Meeting of the Association for Computational
  ith Ringel Morris, Percy Liang, and Michael S Bern-       Linguistics (Volume 1: Long Papers).
  stein. 2023. Generative agents: Interactive simulacra
  of human behavior. In Proceedings of the 36th an-       Sikuan Yan, Xiufeng Yang, Zuchao Huang, Ercong Nie,
  nual acm symposium on user interface software and          Zifeng Ding, Zonggen Li, Xiaowen Ma, Kristian Ker-
  technology, pages 1–22.                                    sting, Jeff Z Pan, Hinrich Schütze, and 1 others. 2025.
                                                             Memory-r1: Enhancing large language model agents
Patrick Schramowski, Cigdem Turan, Nico Andersen,            to manage and utilize memories via reinforcement
  Constantin A Rothkopf, and Kristian Kersting. 2022.        learning. arXiv preprint arXiv:2508.19828.
  Large pre-trained language models contain human-
  like biases of what is right and wrong to do. Nature    Woongyeong Yeo, Kangsan Kim, Soyeong Jeong, Jin-
  Machine Intelligence, 4(3):258–268.                      heon Baek, and Sung Ju Hwang. 2025. Universalrag:
  Retrieval-augmented generation over corpora of di-
  verse modalities and granularities. arXiv preprint
  arXiv:2504.20734.
Duzhen Zhang, Yahan Yu, Jiahua Dong, Chenxing Li,
  Dan Su, Chenhui Chu, and Dong Yu. 2024a. Mm-
  llms: Recent advances in multimodal large language
  models. In Findings of the Association for Computa-
  tional Linguistics ACL 2024, pages 12401–12430.
Shaolei Zhang, Qingkai Fang, Zhe Yang, and Yang
  Feng. 2025a. LLaVA-mini: Efficient image and
  video large multimodal models with one vision to-
  ken. In The Thirteenth International Conference on
  Learning Representations.
Xin Zhang, Yanzhao Zhang, Wen Xie, Mingxin Li, Ziqi
  Dai, Dingkun Long, Pengjun Xie, Meishan Zhang,
  Wenjie Li, and Min Zhang. 2024b. Gme: Improving
  universal multimodal retrieval by multimodal llms.
  arXiv preprint arXiv:2412.16855.
Zeyu Zhang, Quanyu Dai, Xiaohe Bo, Chen Ma, Rui Li,
  Xu Chen, Jieming Zhu, Zhenhua Dong, and Ji-Rong
  Wen. 2025b. A survey on the memory mechanism of
  large language model-based agents. ACM Transac-
  tions on Information Systems, 43(6):1–47.
Zeyu Zhang, Quanyu Dai, Xu Chen, Rui Li, Zhongyang
  Li, and Zhenhua Dong. 2025c. Memengine: A uni-
  fied and modular library for developing advanced
  memory of llm-based agents. In Companion Pro-
  ceedings of the ACM on Web Conference 2025, pages
  821–824.

Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and
 Yanlin Wang. 2024. Memorybank: Enhancing large
  language models with long-term memory. In Pro-
  ceedings of the AAAI Conference on Artificial Intelli-
  gence, volume 38, pages 19724–19731.
A   Appendix
Contents
    A.1 Problem Definition . . . . . . . . 13
    A.2 Data Construction Details and
         Statistics . . . . . . . . . . . . . . 13
         A.2.1 Data Statistics . . . . . . 13
         A.2.2 Data Source . . . . . . . . 13
         A.2.3 Conversation Data Synthesis 14
         A.2.4 Evaluation Data Synthesis 16
    A.3 Detailed Evaluation Task Descrip-
         tions . . . . . . . . . . . . . . . . 16
    A.4 Baseline Details . . . . . . . . . . 18
         A.4.1 Textual Memory with Vi-
                  sual Caption . . . . . . . 18
         A.4.2 Multimodal Memory . . . 18
    A.5 Benchmark Evaluation Details . . 19
         A.5.1 Experimental Environments 19
         A.5.2 Benchmark Settings . . . 19
         A.5.3 Evaluation Pipeline . . . . 19
         A.5.4 Evaluation Metrics . . . . 20
    A.6 Additional Results and Analysis . 21
         A.6.1 MLLM Backbone Analysis 21
         A.6.2 Memory Retrieval Analysis 24
    A.7 Case Study . . . . . . . . . . . . 26
    A.8 Ethics Statement . . . . . . . . . 28
         A.8.1 Data Privacy . . . . . . . 28
         A.8.2 Annotation . . . . . . . . 28
         A.8.3 Scientific Artifacts . . . . 29
    A.9 Potential Risks . . . . . . . . . . 29
    A.10 Use of AI Assistants . . . . . . . 30
A.1   Problem Definition                                   Memory-Augmented Generation. The final ac-
Notations. We consider an MLLM-based agent                 tion or response yτ is generated by the MLLM
interacting with an environment (user) over a se-          agent by conditioning on the fusion of the working
quence of time steps t ∈ {1, . . . , T }. Let O =          context Cτ , retrieved external knowledge Mret ,
{o1 , o2 , . . . , oT } be a stream of multimodal obser-   and the specific task query qτ at timestamp τ :
vations, where each observation entry ot = ⟨vt , st ⟩
may contain visual vt and textual st modalities.                     yτ = MLLM(Cτ ⊕ Mret ⊕ qτ ),            (3)
The agentic memory system is defined by the tuple
S = ⟨M, fθ , Φ, R⟩, where:                                 where ⊕ denotes the integration of heterogeneous
                                                           tokens. This architecture ensures that the agent can
   • Mt = {m1 , m2 , . . . , mn } is the external mul-     maintain long-term coherence and persona consis-
     timodal memory, an unbounded set of atomic            tency without being limited by the context window
     units. Each unit mi encapsulates raw assets,          of the underlying model (Zhang et al., 2025b).
     cross-modal descriptions, and a joint latent
     embedding ei .                                        A.2     Data Construction Details and Statistics
                                                           A.2.1    Data Statistics
   • fθ is a multimodal encoder that projects het-
     erogeneous data into a unified d-dimensional          Mem-Gallery is a new benchmark featuring: (1)
     latent space Rd .                                     scenario diversity, with long, multimodal, and
                                                           multi-session conversations spanning twenty sce-
   • Φ is the update operator that governs the tran-       narios; (2) non-trivial multimodality, where visual
     sition of information between new informa-            and textual information are genuinely complemen-
     tion in O and M, including memory consoli-            tary and both important; and (3) multi-faceted ca-
     dation and eviction.                                  pabilities, comprehensively assessing key mem-
                                                           ory abilities including extraction, test-time adapta-
   • R is the retrieval operator used to identify and      tion, reasoning, and knowledge management. The
     surface relevant historical information from          dataset in Mem-Gallery is constructed across 20
     M based on current needs.                             diverse conversation scenarios, comprising a total
                                                           of 240 multi-session dialogues with 3,962 conver-
Memory Construction and Maintenance. The                   sational rounds. It incorporates 1,003 input images,
memory evolution occurs continuously as the agent          which are naturally grounded within the dialogue
ingests the observation stream O from the conver-          context. The details of each scenario can be found
sation. At each time step t, the update operator           in Table 5. Based on these dialogues, we further cu-
Φ consolidates new observations into the existing          rate 1,711 human-annotated question–answer pairs
memory repository:                                         to support systematic evaluation. Among them,
                                                           487 questions are explicitly associated with visual
             Mt+1 = Φ(Mt , ot , πevo ),             (1)    inputs, requiring models to reason over both tex-
                                                           tual and visual information. The details can be
where πevo represents an autonomous evolution
                                                           found in Table 6. This design ensures broad cov-
policy, such as memory add, merge, or delete (Yan
                                                           erage of multimodal conversational settings while
et al., 2025; Xiong et al., 2025).
                                                           enabling fine-grained analysis of long-term multi-
Memory Retrieval. Generation is invoked at a               modal memory capabilities.
specific task timestamp τ . Given a multimodal
                                                           A.2.2    Data Source
user query qτ , the retrieval operator R surfaces a
relevant subset Mret from the current state of the         Image Data Source. To ensure sufficient eval-
memory repository:                                         uation difficulty and diversity, we collect fine-
                                                           grained image categories as well as images that
Mret = {mi ∈ Mτ | rank(sim(fθ (qτ ), fθ (mi ))) ≤ K},      are closely related to the manually designed di-
                                                (2)        alogue topics as multimodal conversation con-
where sim(·) is a similarity scoring function which        struction materials. All collected images are se-
computes the multimodal similarity and K is the            lected from sources with permissive knowledge-
retrieval size.                                            sharing licenses. These image resources include
Table 5: Detailed statistics of the new dataset in our Mem-Gallery. Each multi-session conversation scenario
contains multiple related dialogue topics.

 Scenarios                                                    Sessions   Rounds   Images in Dialogs   QAs     Images in Questions
 AI, Robotics, Automation, Future Tech                           12        185           31             57              8
 Academic, Animal, Pet, Research, Life                           14        177           45             77             28
 Architecture, Art, Culture, Exhibition, Technology              14        186           30             74              6
 Astronomy, Physics, Scientific Experiments, Cosmology           11        203           30             54              2
 Baking, Dessert, Daily Life, Skill                              15        262           57            111             32
 Dog, Behavior, Research, Academic, Life                         11        180           38             63             21
 Education, Career, Research, Lifestyle                          12        227           56            106             21
 Entrepreneurship, Blockchain, Economics, Logistics, Nature      10        144           36             90             25
 Fashion, Personal Care, Lifestyle, Shopping                      9        168           37             61             13
 Global Travel, Culture, Sightseeing                             10        176           66             87             29
 Global Travel, Sustainable Fashion, Design                       9        159           53             78             29
 Home, Health, Lifestyle, Product                                10        163           36             74              8
 Home, Repair, Maintenance, Cleaning                             18        270           43             68              0
 Landscape, Travel, Architecture, Nature                         11        173           64             92             38
 Music, Dance, Theater, Performance, Learning                    14        220           43             57              1
 Nature, Economics, Programming, Student Life                    10        201           78            136             67
 Parenting, Commuting, Hobbies, Travel, Gear                     16        221           48             82             14
 Python, Botany, AI, Student Life                                12        252          117            205            110
 Real Estate, Home Decor, DIY, Lifestyle                         11        196           53             75             11
 Technology, Ethics, Future Society                              11        199           42             64             24
 Total Statistics                                               240       3,962        1,003          1,711           487



MMDU2 (Liu et al., 2024), CUB-200-20113 (Wah                         details can be found in Appendix A.2.3.
et al., 2011), Stanford Dogs4 (Khosla et al., 2011),
                                                                     A.2.3 Conversation Data Synthesis
Oxford flower5 (Nilsback and Zisserman, 2008),
and DeepFashion6 (Liu et al., 2016). After collec-                   As shown in Figure 6, our conversation data syn-
tion, we manually select appropriate images and                      thesis follows two main approaches.
insert them at information-relevant positions within                    First, we generate high-quality conversational
the generated textual conversations. The details can                 stories with the assistance of human annotators.
be found in Appendix A.2.3.                                          Annotators initially create the protagonist’s profile
   Single-Session Dialogue Data Source. In                           and overall conversation themes, and then spec-
addition to newly created materials, we recon-                       ify the core content of each session through story
struct existing single-session dialogues with mul-                   outlines, as well as the transition logic between
timodal information, sourced from prior dataset                      sessions. Based on this setup, dialogue generation
MMRC7 (Xue et al., 2025). Specifically, we first                     proceeds in two stages. LLMs are first used to
employ advanced LLMs, namely GPT-5.1 and                             generate the textual dialogue content. Specifically,
Gemini-2.5-Pro, to extract representative keywords                   we employ GPT-5.1 and Gemini-2.5-Pro in parallel
from each single-session dialogue, after which                       and manually select higher-quality outputs. The
the higher-quality outputs are manually selected.                    dialogues are then further refined, and appropriate
Based on the extracted keywords, we further use                      images are inserted at suitable positions to establish
LLMs to perform session-level clustering. Once                       multimodal dependencies. Each multimodal multi-
candidate multi-session dialogues are formed, we                     session long conversation is constructed following
conduct additional human filtering and enhance-                      this process, after which quality assurance and QA
ment to ensure the overall quality and coherence of                  generation are conducted.
the reconstructed multi-session conversations. The                      Second, we reconstruct existing single-session
                                                                     dialogue datasets via topic-based clustering. In this
   2
     https://github.com/Liuziyu77/MMDU                               approach, LLMs are first used to extract session-
   3
     https://www.vision.caltech.edu/datasets/cub_                    level topic keywords and perform clustering. Hu-
200_2011
   4                                                                 man annotators then review and refine the clustered
     http://vision.stanford.edu/aditya86/
ImageNetDogs                                                         results, and summarize the corresponding user pro-
   5
     https://www.robots.ox.ac.uk/~vgg/data/                          files and multi-session conversation topics. Since
flowers/102                                                          the resulting clustered long conversations often suf-
   6
     https://mmlab.ie.cuhk.edu.hk/projects/
DeepFashion.html                                                     fer from limited fluency, weak inter-session transi-
   7
     https://github.com/haochen-MBZUAI/MMRC                          tion logic, and insufficient length, we further refine
                                                                                                                                               Story Outline & Inter-Session Transition Logic for
                       User Profile:                                                                                                                           Each Conversation
                  {                                                             Thinking of   Adopting and AI for animal Online animal Classmate’s dog & Road Travel
                        "name": "Lena",                                        adopting a dog naming a dog   behavior behavior seminar discuss breeds with the dog
                        “persona_summary”: "Lena is a 21-year-old
                        university student majoring in Life Sciences.                                                                                                                                                                …                                          …
                                                                                                                                                                                                                                                                                                   Sessions
                        She has always been fascinated by animals and
                        spends much of her free time volunteering at                S-1                                                                    S-2                         S-3                      S-4                                     S-k                                S-n
                        the campus animal care center. ...",
                        "traits": ["curious", "caring", "enthusiastic"],
                        “conversation_style”: "Warm and inquisitive,
                        Lena’s conversations often blend empathy with                                                                                       Conversation Generation with LLMs and
                        scientific curiosity. She expresses genuine                                                                                           Image Insert by Human Annotators
                        interest in understanding how things work ..."
                  }
 Background                                                                                                                                                 Session-1                                                                   Session-n
  Creation
                                                                                                                                                                …                                                                            …
                                                                                                                                                                                                     …
                                            Academic, Animal,                                                                                       User            …                                                        User                  …
                                            Pet, Research, Life                                                                                                         Assistant                                                                       Assistant
                                                                                                                                                                 …                                                                               …

                                Conversation Topics
                                                                                                                                                                                                                                                                          Quality Assurance &
                                                                                                                                                                                                                                                                            QA Generation
                                              (a) Conversation Generation with Newly Created Story


                                     User Profile:
              …                 {
                                       "name": "Julian",                                                                                                        Reorder, Refinement, and Include more Sessions
                                       “persona_summary”: "Julian is a 19-year-                                                                                              by Human Annotators
                                       old first-year undergraduate student in
  Single-Session Dialog                architecture, fascinated by how historical
        Clustering                     spaces and modern technology can shape                                                                                           Session-1                                                                         Session-n
                                       human emotion and memory.",
                                       “traits”: [“inquisitive”, “introspective”,
                                                                                                                                                                              …                                                                                      …
                                       "creative", "methodical"],                                                                                                                                                        …
                                       “conversation_style”: "Thoughtful and
                                                                                                                                                                User              …                                                          User                        …
                                       inquisitive, often exploring concepts ..."                                                                                                      Assistant                                                                                Assistant
                                                                                                                                                                              …                                                                                      …
                                }



                                                        Architecture, Art,
    Refinement and
                                                        Culture, Exhibition,
    Summarization                                           Technology                                                                                                                                                                                                    Quality Assurance &
                                                                                                                                                                                                                                                                            QA Generation

                                            Conversation Topics

                                              (b) Conversation Generation with Topic-based Clustering


                      Figure 6: Illustration of the generation for each conversation, including two ways.


the dialogue content to ensure coherence and rea-                                                                                                                                                                                                                                                   / R & R 0 R
                                                                                                                                                                                                                                                                                                    0 H P  * D O O H U \
                                                                                                                                               
sonable session transitions. To address insufficient
                                                                                                                                               
conversation length, we additionally extend these
                                                                                       3 H U F H Q W D J H  R I  6 H V V L R Q V    




clustered conversations by introducing more ses-                                                                                               


sions, which are generated by LLMs and then man-                                                                                               

ually verified and augmented with images, while                                                                                                

preserving logical continuity across sessions.
                                                                                                                                               

   The distribution of image numbers per ses-                                                                                                   

sion in Mem-Gallery is described in Figure 7.
                                                                                                                                                
                                                                                                                                                                                                                                                                                        
From the distribution, we observe that, beyond                                                                                                                                                      1 X P E H U  R I  , P D J H V  S H U  6 H V V L R Q
the importance and high quality of visual infor-
mation in Mem-Gallery as compared in Figure 2,                                            Figure 7: Image number distribution of Mem-Gallery
Mem-Gallery significantly reduces the proportion                                          per conversation session compared with LoCoMo.
of sessions with no images or only a single image
compared to LoCoMo (Maharana et al., 2024). The
overall distribution shifts rightward, with most ses-                                 17 images. This design increases both the density
sions containing at least two images, and the num-                                    of visual information in conversations and the over-
ber of sessions with multiple images (e.g., seven                                     all difficulty of the dialogue tasks. The conversa-
or more images) is substantially increased. In the                                    tion structure of Mem-Gallery and the information
most extreme cases, a single session contains up to                                   feed way for memory agents are shown in Figure 8.
Table 6: Evaluation task statistics on the QA number          quire explicit visual cues are entirely constructed
and average clues in each category. * The AR task is          by human annotators. This process emphasizes
designed to evaluate refusal capability. The information
                                                              challenging cases that are difficult for automatic
in the question does not exist in the dialogue, and there-
fore clues are unsupported.                                   generation to reliably capture.
                                                                 For each finalized QA pair, annotators explicitly
        Evaluation Tasks         Number of QAs   Avg. Clues   annotate the evidence clues required to derive the
                           FR        219            2.90      correct answer with the assistance of LLMs. These
    Extract. & Adapt.      VS        306            1.91      clues specify the dialogue turns or visual content
                           TTL       337            4.09
                                                              where the relevant information is introduced or
                           TR        123            1.74
                           VR        174            3.05
                                                              referenced.
        Reasoning
                           MR        206            2.17
                           KR        81             2.21
                                                              A.3   Detailed Evaluation Task Descriptions
 Knowledge Management      CD        81             1.70      Mem-Gallery evaluates multimodal long-term con-
                           AR*       184             —
                                                              versational memory through three complementary
            Overall                  1,711          2.69
                                                              functional dimensions, each consisting of three
                                                              fine-grained task types. Together, these nine tasks
                                                              provide a systematic and comprehensive assess-
A.2.4    Evaluation Data Synthesis
                                                              ment of how memory is extracted, adapted, rea-
Evaluation QA pairs are constructed through                   soned over, and managed under long-horizon mul-
two complementary and independent generation                  timodal conversational settings:
pipelines, which jointly contribute to the final eval-
uation set. These two pipelines are designed to               Memory Extraction & Adaptation. This dimen-
balance coverage, difficulty, and diversity while             sion focuses on whether an agent can correctly
ensuring that evaluation instances genuinely cover            extract, store, and adaptively utilize information
long-term and multimodal memory access.                       accumulated throughout long-term multimodal in-
   In the first pipeline, advanced LLMs are                   teractions.
prompted to automatically generate candidate QA
                                                                 • Factual Retrieval (FR): This task evaluates
pairs. The input to the LLM consists of selected
                                                                   the model’s ability to accurately recall ex-
multi-session conversation histories together with
                                                                   plicit factual information mentioned in pre-
detailed task descriptions specifying the targeted
                                                                   vious multimodal dialogue turns, including
evaluation capability (e.g., factual retrieval, tem-
                                                                   user attributes, preferences, entities, events,
poral reasoning, or knowledge resolution). The
                                                                   and decisions. Questions may require retriev-
prompts explicitly constrain questions to depend
                                                                   ing information introduced long before the
on information distributed across multiple dialogue
                                                                   query, testing whether the memory system can
turns or sessions, and, when applicable, to require
                                                                   preserve and access long-horizon facts rather
visual evidence in addition to textual context. This
                                                                   than relying on short-term context.
pipeline enables scalable generation of QA candi-
dates with broad task coverage.                                  • Visual-centric Search (VS): This task as-
   In parallel, human annotators manually construct                sesses whether the model can correctly and
QA pairs by reviewing each entire multi-session                    explicitly identify or retrieve relevant visual
conversation from start to end. Annotators are in-                 content from the conversation history. Queries
structed to identify information dependencies that                 typically require selecting the correct image
span long temporal gaps, involve updates or contra-                or images associated with a specific entity,
dictions, or require integrating visual and textual                event, or user reference, testing the alignment
cues. Based on these observations, annotators de-                  between visual memory entries and textual
sign targeted questions that cannot be answered                    cues in long-term memory.
from local context alone and instead require ac-
cessing accumulated multimodal memory. Notably,                  • Test-Time Learning (TTL): This task mea-
LLMs are unable to reliably generate QA pairs that                 sures the model’s ability to adapt its under-
explicitly depend on visual evidence. The result-                  standing at inference time based on newly
ing questions are often inaccurate or of low quality.              provided multimodal examples. The agent
Therefore, evaluation instances for tasks that re-                 must incorporate new multimodal instances
               Session-1                                                  Session-k                         Session-n
                2024-05-12                                                 2024-08-10                        2024-11-29


                …                                                          …                                 …

        User        …                                              User          …                   User        …
                        Assistant                                                  Assistant                         Assistant

                    …                 Multi-Round        …                     …                …                …

                …                                                          …                                 …

        User        …                                              User          …                   User        …
                        Assistant                                                  Assistant                         Assistant


                                                                 Multi-Session
                                     Streaming Interaction
                                                                               Memory Action:
                                             …                              {Add; Delete; Update}

                                      User       …
                                                     Assistant

                                                                 Memory Agent             Stored Memory


                                    Figure 8: Structure example of each conversation.


    into its existing memory and apply this up-                                  sesses the model’s capacity to jointly reason
    dated knowledge to answer subsequent ques-                                   over multiple entities stored in memory, where
    tions, without any parameter updates. This                                   entities may be introduced at different times
    setting emphasizes online memory adaptation                                  and may involve both textual and visual rep-
    rather than static memorization.                                             resentations. Correct answers require synthe-
                                                                                 sizing information across entities rather than
Memory Reasoning. Beyond direct retrieval,                                       recalling isolated facts.
real-world conversational agents must reason over
stored memories. This dimension evaluates                                 Memory Knowledge Management. Conversa-
whether models can perform structured reasoning                           tional memory is inherently dynamic and imperfect.
using long-term multimodal information accumu-                            This dimension evaluates whether a memory sys-
lated over time.                                                          tem can properly manage evolving, conflicting, or
                                                                          missing information.
  • Temporal Reasoning (TR): This task evalu-
    ates whether the model can reason over tem-                                • Knowledge Resolution (KR): This task ex-
    porally ordered information in memory. Ques-                                 amines whether the model can correctly up-
    tions may involve comparing events across                                    date its memory when new information contra-
    different sessions, identifying the order of                                 dicts previously stored knowledge. The agent
    occurrences, or determining when a partic-                                   must discard or revise outdated beliefs and
    ular event happened based on dispersed multi-                                consistently rely on the most recent and cor-
    modal cues.                                                                  rect information when answering questions.

  • Visual-centric Reasoning (VR): This task                                   • Conflict Detection (CD): This task evaluates
    tests the model’s ability to use visual infor-                               the model’s ability to detect inconsistencies
    mation as a key component of the reasoning                                   between newly observed information and ex-
    process. Unlike visual-centric search, which                                 isting memory. Rather than resolving the con-
    focuses on retrieval, visual-centric reasoning                               flict, the agent is required to explicitly identify
    requires integrating visual cues with textual                                whether a contradiction exists, testing sensi-
    context to infer properties, relationships, or                               tivity to memory consistency.
    similarities that are not explicitly stated.
                                                                               • Answer Refusal (AR): This task assesses
  • Multi-entity Reasoning (MR): This task as-                                   whether the model can appropriately refuse to
      answer when the requested information has            model dynamically pages relevant memories
      never been mentioned or is unsupported by            into context and evicts less useful ones, en-
      the conversation history. It tests whether the       abling effective long-term conversation be-
      memory system avoids hallucination and rec-          yond fixed context limits.
      ognizes the absence of valid evidence in long-
      term memory.                                       • A-Mem (Xu et al., 2025): A-MEM pro-
                                                           poses an agentic memory system that orga-
A.4    Baseline Details                                    nizes experiences as structured atomic notes
                                                           with rich contextual descriptions, inspired
We include thirteen memory methods, with eight
                                                           by the Zettelkasten method. New memories
representative textual memory methods and five
                                                           autonomously establish links to related past
multimodal memory methods.
                                                           memories and can trigger updates to existing
A.4.1 Textual Memory with Visual Caption                   ones, enabling a dynamically evolving mem-
For the textual memory, we transfer the raw image          ory network that supports long-term reasoning
into an image caption with GPT-5.1. The details of         without predefined memory operations.
these models are illustrated as follows.
                                                         • MemoryOS (Kang et al., 2025): MemoryOS
  • Full Memory (Text): It includes all memory             introduces an operating system–inspired mem-
    information in textual form as part of the con-        ory framework that manages agent memory
    text, and truncates it according to the context        through hierarchical storage, dynamic updat-
    token limit.                                           ing, and semantic retrieval. It organizes mem-
                                                           ory into short-term, mid-term, and long-term
  • FIFO (First-in-first-out): It selects the most         persona layers, and applies OS-style mecha-
    recent memory information as context accord-           nisms such as FIFO, segmented paging, and
    ing to the temporal order of the memory.               heat-based eviction to maintain coherent and
                                                           personalized long-term conversations.
  • NaiveRAG: It encodes the memory informa-
    tion into semantic vectors, then retrieves and     A.4.2 Multimodal Memory
    returns the most similar memory entries based      For the multimodal memory, we include five repre-
    on vector similarity.                              sentative models. The details of these models are
                                                       illustrated as follows.
  • Generative Agent (Park et al., 2023): It intro-
    duces a generative agent memory that stores          • Full Memory (Multimodal): It includes all
    experiences as language memories and re-               multimodal memory information as context,
    trieves relevant entries to condition agent be-        estimates the token consumption of images
    havior. It further synthesizes higher-level re-        using predefined token costs, and truncates
    flections from past experiences to support co-         the input according to the context token limit.
    herent long-term planning and interaction.
                                                         • MuRAG (Chen et al., 2022): MuRAG uses a
  • Reflexion (Shinn et al., 2023): Reflexion              dense multimodal retriever that encodes both
    proposes a verbal memory mechanism in                  queries and memory entries into a shared em-
    which agents store self-generated reflections          bedding space using a joint vision–language
    as episodic memories. After each trial, feed-          encoder. At inference time, it performs max-
    back is converted into language reflections            imum inner product search over an external
    that are appended to memory and reused as              memory to retrieve the most relevant items,
    context to guide decisions, enabling trial-and-        which are then used to augment generation.
    error learning via memory accumulation.                We use the retrieval paradigm in this paper for
                                                           our benchmark implementation.
  • MemGPT (Packer et al., 2023): MemGPT
    proposes an OS-inspired memory architec-             • UniversalRAG (Yeo et al., 2025): It intro-
    ture that treats the LLM context window as             duces a modality- and granularity-aware RAG
    limited working memory and manages long-               framework that dynamically routes each query
    term information via hierarchical external stor-       to the most appropriate knowledge source be-
    age. Through self-directed function calls, the         fore retrieval. Instead of retrieving from a
      unified corpus, it first predicts the required                 Face repositories8 . Then, we deploy them on
      modality and granularity and then performs                     A100 GPUs under the Linux system with the
      targeted retrieval within the selected corpus,                 vLLM engine9 . For closed-sourced MLLM mod-
      reducing modality bias and retrieval noise.                    els, we access them via official APIs. The mem-
                                                                     ory model implementation and reproduction are
   • NGM (Fisher, 2025): It proposes Neural                          built based on the open-sourced memory frame-
     Graph Memory, a graph-structured multi-                         work MemEngine10 (Zhang et al., 2025c). The to-
     modal memory that stores episodic expe-                         tal API compute budget for open-sourced MLLM
     riences as nodes enriched with modality-                        models was under one thousand US dollars. The
     specific embeddings and temporal metadata.                      detailed running time for each memory model is
     Memories are retrieved via graph traversal, en-                 illustrated in Figure 5.
     abling associative, temporal, and cross-modal
     recall beyond flat vector similarity, and sup-                  A.5.2   Benchmark Settings
     porting long-horizon episodic reasoning.                        To eliminate the influence of backbone differences
                                                                     and minimize randomness, all experimental set-
   • AUGUSTUS (Jain et al., 2025): AUGUSTUS                          tings use the same MLLM backbone, with a fixed
     introduces a cognition-inspired multimodal                      random seed and the temperature set to 0. To fur-
     memory that organizes long-term memory                          ther ensure a fair comparison, for memory systems
     into recall memory for raw conversation his-                    that involve embedding-based storage and retrieval,
     tory and a hierarchical contextual memory for                   we uniformly adopt the widely used GME-Qwen2-
     semantic concepts linked to multimodal con-                     VL-2B-Instruct (Zhang et al., 2024b) as the embed-
     text. Abstracting user interactions into seman-                 ding model. For memory methods that require a
     tic tags and retrieving information through a                   retriever, we adopt a default retrieval size of K=10.
     concept-driven search enables efficient, per-                   For other hyperparameters of the models, we retain
     sonalized, and long-horizon multimodal mem-                     their optimal settings from the original codebase.
     ory access beyond flat vector databases.
                                                                     A.5.3   Evaluation Pipeline
   The corresponding papers and implementation
                                                                     The overall evaluation pipeline is based on the
details of these methods are summarized in Table 7.
                                                                     MemEngine (Zhang et al., 2025c) framework. The
For methods without publicly available implemen-
                                                                     setup is illustrated in Appendix A.5.2. For baseline
tation code (marked as N/A), we re-implement
                                                                     methods that involve calling an MLLM module,
them based on the methodological descriptions pro-
                                                                     we consistently use the corresponding backbone
vided in the original papers if available.
                                                                     MLLM of each method to ensure a fair comparison.
Table 7: The detailed resource list of models imple-                 Both the conversational data and the evaluation data
mented in our benchmark.                                             are stored in JSON format. For conversational data,
                                                                     following existing works, dialogue information is
 Model                       Venue       Paper   Implement. Source   streamed to each memory method at the granular-
 Full Memory-Text             N/A        N/A           Link          ity of dialogue rounds. During question-answering
 FIFO                         N/A        N/A           Link
 NaiveRAG                     N/A        N/A           Link          evaluation, the accumulated memory contents are
 Gen. Agent                UIST 2023     Link          Link          concatenated into the context, after which the cor-
 Reflexion                NeurIPS 2023   Link          Link
 MemGPT                    Arxiv 2023    Link          Link          responding MLLM is invoked to generate the fi-
 A-Mem                    NeurIPS 2025   Link          Link
 MemoryOS                 EMNLP 2025     Link          Link
                                                                     nal answer and save the output. For tasks such as
 Full Memory-Multimodal       N/A        N/A           N/A           visual-centric search, conflict detection, and answer
 MuRAG                    EMNLP 2022     Link          N/A
 UniversalRAG              Arxiv 2025    Link          Link
                                                                     refusal, which require specific output formats, ad-
 NGM                       Arxiv 2025    Link          Link          ditional format-constrained prompts are appended
 AUGUSTUS                  Arxiv 2025    Link          N/A
                                                                     after the question, as shown in Figure 9. Finally,
                                                                     all generated answers are normalized in a unified
                                                                     manner before metric computation. The prompt
A.5      Benchmark Evaluation Details
                                                                     template used for LLM-as-a-Judge evaluation is
A.5.1     Experimental Environments                                     8
                                                                          https://huggingface.co
For open-sourced MLLM models, we obtain the of-                         9
                                                                          https://github.com/vllm-project/vllm
                                                                       10
ficial models from their corresponding Hugging                            https://github.com/nuster1128/MemEngine
illustrated in Figure 10.                                  • 0: Incorrect or missing answer, including con-
                                                             tradictions or hallucinations.
A.5.4   Evaluation Metrics
We evaluate Mem-Gallery from three key perspec-            • 0.25: Poor or tangential answer that touches
tives: question-answering performance, retrieval             on the topic but misses the core entity or value.
effectiveness, and computational efficiency. All
metrics are computed at the instance level and then        • 0.5: Partially correct answer that captures the
averaged over the QA set.                                    main concept but lacks completeness or key
                                                             details.
Question-Answering Metrics. For answer qual-
ity, we adopt four widely used metrics in conversa-        • 0.75: Largely correct answer with only minor
tional question answering.                                   omissions or imprecision.
   F1 measures token-level overlap between the
predicted answer Ap and the reference answer Ar .          • 1: Correct and exact answer that fully matches
Let Tp and Tr denote the multisets of tokens in Ap           the ground truth.
and Ar , respectively. Precision P and recall R are
defined as:                                             The final LLM-as-a-Judge score is obtained by av-
                                                        eraging the assigned scores across evaluation in-
              |Tp ∩ Tr |           |Tp ∩ Tr |
        P =              ,    R=              .   (4)   stances. This metric complements lexical overlap
                |Tp |                 |Tr |             metrics by capturing semantic equivalence, para-
The F1 score is then computed as:                       phrasing, and partial correctness.
                                                           For the Conflict Detection (CD) task, the model
                              2P R                      is required to output a single binary token (Yes or
                   F1 =            .              (5)
                             P +R                       No). Since this is a binary classification task and
                                                        all evaluated models reliably follow the instruction
   Exact Match (EM) evaluates whether the pre-          format, the values of F1, EM, BLEU-1, and LLM-
dicted answer exactly matches the reference answer      as-a-Judge coincide for this task.
after normalization (e.g., lowercasing and punctua-
tion removal):                                          Retrieval Effectiveness Metrics. To assess mem-
                    (                                   ory access quality independently of answer gen-
                      1, if Ap = Ar ,                   eration, we adopt standard information retrieval
            EM =                                (6)
                      0, otherwise.                     metrics. For each evaluation query, let RK denote
                                                        the set of top-K retrieved memory entries, and G
  BLEU-1 measures unigram-level precision be-           denote the set of ground-truth relevant memory
tween the predicted answer Ap and the reference         entries annotated as evidence clues.
answer Ar . Let countAp (w) denote the number of          Recall@K is defined as:
occurrences of unigram w in Ap , and countAr (w)
denote its occurrences in Ar . The clipped count                                    |RK ∩ G|
                                                                     Recall@K =              .            (9)
c(w) is defined as:                                                                   |G|

                                                          Precision@K is defined as:
                                      
   c(w) = min countAp (w), countAr (w) .          (7)

BLEU-1 is then computed as:                                                           |RK ∩ G|
                                                                   Precision@K =               .         (10)
                     P                                                                  |RK |
                               w∈Ap c(w)
        BLEU-1 = P                            .   (8)    Hit@K measures whether at least one relevant
                         w∈Ap countAp (w)
                                                        memory entry is retrieved:
   LLM-as-a-Judge evaluates semantic correct-                           (
ness using Qwen-2.5-72B-Instruct as an automatic                         1, if RK ∩ G ̸= ∅,
                                                                Hit@K =                                  (11)
evaluator. Given the question, the ground-truth an-                      0, otherwise.
swer, and the model prediction, the judge assigns
a discrete score from the set {0, 0.25, 0.5, 0.75, 1}     All retrieval metrics are first computed per query
according to a predefined rubric. Specifically:         and then averaged across the evaluation set.
   Format Restriction Prompts

   Visual-centric Search:
   Return the image_id of the image(s). If there are multiple images, sort them in ascending order and
   separate them by commas. Format example: “D2:IMG_003, D2:IMG_010, D10:IMG_002” (for
   format reference only).
   Conflict Detection:
   Please check whether this information conflicts with the conversation, and reply strictly with either
   “Yes.” or “No.”
   Answer Refusal:
   Provide your answer based on the information in the conversation. Only if the information about
   the question is not present in the conversation, reply with: “Not mentioned.”


                                Figure 9: Prompts for format restriction tasks.


Efficiency Metrics. We evaluate efficiency from          memory performance across most tasks and meth-
a system-level perspective. For each model, we           ods. This trend is visible when scaling from
measure: (1) Information Memorization Time,              Qwen-2.5-VL-3B to Qwen-2.5-VL-7B, and further
defined as the cumulative wall-clock time spent          from GPT-4.1-Nano to Gemini-2.5-Flash-Lite. The
processing and storing conversational inputs during      gains are systematic rather than isolated, indicat-
interaction. (2) Memory Retrieval and Answer             ing that increased backbone capacity enhances the
Generation Time, defined as the wall-clock time          model’s ability to consume retrieved memory and
required to retrieve memory entries and generate         generate coherent responses. However, these im-
the final response at evaluation time. All efficiency    provements do not fundamentally alter which task
metrics are reported in seconds.                         categories remain difficult. Reasoning-intensive
                                                         tasks and knowledge management tasks continue
A.6   Additional Results and Analysis                    to exhibit lower performance relative to extraction-
A.6.1 MLLM Backbone Analysis                             oriented tasks, even under the strongest backbones.
To analyze how the choice of MLLM backbone               This suggests that backbone scaling primarily im-
influences memory performance, we evaluate rep-          proves memory utilization, rather than resolving
resentative open-source and closed-source MLLMs          intrinsic limitations in long-horizon reasoning and
under a unified memory framework. Specifically,          memory consistency of existing memory methods.
Table 8 and Table 3 report results on the open-             Backbone scaling amplifies memory design
source Qwen-2.5-VL-3B and Qwen-2.5-VL-7B                 quality rather than compensating for it. Despite
backbones, while Table 9 and Table 10 report             large differences in backbone capacity and model
results on the closed-source GPT-4.1-Nano and            family, the relative ranking of memory methods
Gemini-2.5-Flash-Lite backbones. Moreover, Fig-          remains largely stable across all evaluated settings.
ure 11, Figure 12, Figure 13, and Figure 14 illus-       Well-organized memory systems consistently out-
trate the radar chart of three representative textual    perform naïve or unstructured baselines under both
memory models (MemGPT, A-Mem, and Memo-                  weak and strong backbones, regardless of whether
ryOS) and three representative multimodal memory         the memory is textual or multimodal. This stabil-
models (MuRAG, UniversalRAG, and AUGUS-                  ity indicates that backbone scaling does not com-
TUS) to clearly present the advantages and disad-        pensate for poor memory organization. Instead,
vantages of different subtasks. Across all settings,     stronger backbones tend to amplify existing dif-
the same set of textual and multimodal memory            ferences between memory designs. Multimodal
methods is applied, enabling a controlled compari-       memory benefits from stronger backbones only
son of backbone effects.                                 when cross-modal alignment and retrieval quality
   Overall effect of backbone capability. Across         are sufficiently well controlled. Otherwise, strong
both open-source and closed-source backbones,            textual memory can achieve comparable effective-
stronger MLLMs consistently improve absolute             ness with lower complexity, such as the equipment
   LLM-as-a-Judge Prompt

   You are an impartial judge evaluating the memory capabilities of an AI assistant with the question-
   answering task. Your task is to compare the Assistant’s Answer against the Ground Truth and
   assign a score of 0, 0.25, 0.5, 0.75, or 1.
   Scoring Rubric
   Score 0 (Incorrect / Miss):
  - The answer contradicts the Ground Truth.
  - For Yes/No questions: The answer has the wrong polarity (e.g., says "Yes" when Ground Truth is
   "No").
  - For Open-ended questions: The answer provides factually wrong information or hallucinations.
  - The assistant fails to provide the required information.
   Score 0.25 (Poor / Tangential):
  - The answer touches on the topic but misses the core entity or key value required.
  - The answer contains a mix of minor correct details and significant hallucinations or wrong
   associations.
  - The answer is excessively vague to the point of being useless (e.g., answering "a dog" instead of
   "a golden retriever").
   Score 0.5 (Partial / Vague):
  - The answer is technically correct, but lacks confidence or is incomplete.
  - The answer captures the main entity or concept correctly but misses a part of the required
   supporting details.
  - For Yes/No questions: The polarity is correct, but the reasoning is flawed (if have), or the assistant
   is uncertain (e.g., "I think it might be Yes").
  - For Open-ended questions: The answer is too general or misses key adjectives/details present in
   the Ground Truth.
   Score 0.75 (Good / Minor Imperfection):
  - The answer is largely accurate and captures the core information confidently.
  - It misses only minor details (e.g., specific adjectives or secondary details) that do not alter the
   main truth.
  - The answer contains all the correct information but includes unnecessary "fluff" or slight conver-
   sational filler that reduces precision.
   Score 1 (Correct / Exact):
  - The answer is accurate, precise, and confident.
  - For Yes/No questions: The polarity matches the Ground Truth perfectly.
  - For Open-ended questions: The answer contains all the core information and necessary details
   required by the Ground Truth without hallucinations.
   Input Data
   Question: {question}
   Ground Truth: {ground_truth}
   Assistant Answer: {model_output}
   Output Format
   Output strictly in the following JSON format: "score": <0, 0.25, 0.5, 0.75, or 1>, "reasoning":
   "<short explanation>"


              Figure 10: The prompt of the LLM-as-a-Judge metric for Qwen-2.5-72B-Instruct.


with Gemini-2.5-Flash-Lite.                             stronger backbones. Although stronger back-
                                                        bones lead to consistent gains on reasoning-
  Reasoning remains a bottleneck despite
                       MemGPT               MemoryOS         UniversalRAG                                                       MemGPT               MemoryOS         UniversalRAG
                       A-Mem                MuRAG            AUGUSTUS                                                           A-Mem                MuRAG            AUGUSTUS
                 F1                                                     BLEU-1                                            F1                                                     BLEU-1
                 FR                                                         FR                                            FR                                                         FR
       AR                   1.0   VS                         AR                        1.0   VS                 AR                   1.0   VS                         AR                        1.0   VS
                          0.8                                                        0.8                                           0.8                                                        0.8
                        0.6                                                        0.6                                           0.6                                                        0.6
                      0.4                                                        0.4                                           0.4                                                        0.4
CD                  0.2                 TTL            CD                      0.2                 TTL   CD                  0.2                 TTL            CD                      0.2                 TTL
                  0.0                                                        0.0                                           0.0                                                        0.0



  KR                                   TR               KR                                        TR       KR                                   TR               KR                                        TR

            MR           VR                                        MR               VR                               MR           VR                                        MR               VR

                 EM                                               LLM-as-a-Judge                                          EM                                               LLM-as-a-Judge
                 FR                                                         FR                                            FR                                                         FR
       AR                   1.0   VS                         AR                        1.0   VS                 AR                   1.0   VS                         AR                        1.0   VS
                          0.8                                                        0.8                                           0.8                                                        0.8
                        0.6                                                        0.6                                           0.6                                                        0.6
                      0.4                                                        0.4                                           0.4                                                        0.4
CD                  0.2                 TTL            CD                      0.2                 TTL   CD                  0.2                 TTL            CD                      0.2                 TTL
                  0.0                                                        0.0                                           0.0                                                        0.0



  KR                                   TR               KR                                        TR       KR                                   TR               KR                                        TR

            MR           VR                                        MR               VR                               MR           VR                                        MR               VR


Figure 11: Radar chart of subtask performance for rep-                                                   Figure 13: Radar chart of subtask performance for rep-
resentative memory models on Qwen-2.5-VL-3B.                                                             resentative memory models on GPT-4.1-Nano.

                       MemGPT               MemoryOS         UniversalRAG                                                       MemGPT               MemoryOS         UniversalRAG
                       A-Mem                MuRAG            AUGUSTUS                                                           A-Mem                MuRAG            AUGUSTUS
                 F1                                                     BLEU-1                                            F1                                                     BLEU-1
                 FR                                                         FR                                            FR                                                         FR
       AR                   1.0   VS                         AR                        1.0   VS                 AR                   1.0   VS                         AR                        1.0   VS
                          0.8                                                        0.8                                           0.8                                                        0.8
                        0.6                                                        0.6                                           0.6                                                        0.6
                      0.4                                                        0.4                                           0.4                                                        0.4
CD                  0.2                 TTL            CD                      0.2                 TTL   CD                  0.2                 TTL            CD                      0.2                 TTL
                  0.0                                                        0.0                                           0.0                                                        0.0



  KR                                   TR               KR                                        TR       KR                                   TR               KR                                        TR

            MR           VR                                        MR               VR                               MR           VR                                        MR               VR

                 EM                                               LLM-as-a-Judge                                          EM                                               LLM-as-a-Judge
                 FR                                                         FR                                            FR                                                         FR
       AR                   1.0   VS                         AR                        1.0   VS                 AR                   1.0   VS                         AR                        1.0   VS
                          0.8                                                        0.8                                           0.8                                                        0.8
                        0.6                                                        0.6                                           0.6                                                        0.6
                      0.4                                                        0.4                                           0.4                                                        0.4
CD                  0.2                 TTL            CD                      0.2                 TTL   CD                  0.2                 TTL            CD                      0.2                 TTL
                  0.0                                                        0.0                                           0.0                                                        0.0



  KR                                   TR               KR                                        TR       KR                                   TR               KR                                        TR

            MR           VR                                        MR               VR                               MR           VR                                        MR               VR


Figure 12: Radar chart of subtask performance for rep-                                                   Figure 14: Radar chart of subtask performance for rep-
resentative memory models on Qwen-2.5-VL-7B.                                                             resentative memory models on Gemini-2.5-Flash-Lite.


oriented tasks, including temporal reasoning,                                                            tual memory methods remain competitive with mul-
visual-centric reasoning, and multi-entity reason-                                                       timodal approaches. These observations indicate
ing, the magnitude of these gains is noticeably                                                          that long-horizon reasoning over multimodal mem-
smaller than those observed in extraction-focused                                                        ory is not primarily constrained by backbone capac-
tasks. This pattern is consistent across both open-                                                      ity, but rather by how multimodal information is
source and closed-source backbones. In several                                                           structured, linked, and jointly reasoned over within
reasoning subtasks, the performance gap between                                                          the memory system. These results indicate that the
textual and multimodal memory narrows as back-                                                           organization and maintenance of multimodal mem-
bone capability increases, and well-designed tex-                                                        ory mechanisms remain an open and promising
direction for future research.                                0.70                               0.50

                                                              0.65
   Limited sensitivity in knowledge management                0.60
                                                                                                 0.45
                                                                                                                               Gen. Agent
                                                                                                                               NaiveRAG
tasks. For knowledge management tasks, includ-                0.55
                                                                                                 0.40                          A-Mem
                                                                                                                               MemoryOS




                                                                                            EM
                                                         F1
                                                                                                 0.35                          MuRAG
ing knowledge resolution, conflict detection, and             0.50                                                             UniversalRAG
                                                                                                                               NGM
                                                              0.45                               0.30                          AUGUSTUS
answer refusal, improvements from stronger back-              0.40
                                                                                                 0.25
bones are relatively limited across all evaluated             0.35
                                                                     5   10       15   20               5   10       15   20
                                                                              K                                  K
MLLMs. Both open-source and closed-source
backbones exhibit similar qualitative behavior, with      Figure 15: Parameter study on the retrieval size K.
only marginal performance differences attributable
to backbone choice. This suggests diminishing
returns from backbone scaling for safety- and           modal memory methods. While larger retrieval
consistency-oriented memory behaviors, which            sizes improve coverage of relevant multimodal en-
rely more heavily on explicit memory state track-       tries, they also introduce additional redundant or
ing, conflict awareness, and decision policies than     weakly aligned visual–textual information. As a re-
on raw generative or perceptual capacity.               sult, the benefits of higher recall are partially offset
                                                        by increased noise in the retrieved memory, limit-
A.6.2   Memory Retrieval Analysis                       ing the effectiveness of downstream reasoning and
This section analyzes how the number of re-             answer generation. In contrast, moderate retrieval
trieved memory entries affects both retrieval effec-    sizes such as K=10, as reflected in Table 3, appear
tiveness and downstream task performance. Ta-           to offer a better balance between relevance and
ble 11, Table 3, Table 12, and Table 13 report          redundancy for multimodal memory utilization.
task-level QA results under different retrieval sizes      For reasoning-oriented tasks, including tempo-
K ∈ {5, 10, 15, 20}. Figure 15 shows the over-          ral reasoning, visual-centric reasoning, and multi-
all task performance trend under K value changes.       entity reasoning, the impact of retrieval size is even
Based on the fine-grained annotated clues for each      more constrained. Performance differences across
evaluation data, Table 14 summarizes the corre-         K=5, K=10, K=15, and K=20 are relatively small,
sponding retrieval metrics, including Recall@K,         suggesting that reasoning performance is less sen-
Precision@K, and Hit@K for retrieval behavior           sitive to retrieval quantity and more dependent on
analysis. All results are evaluated on the Qwen-2.5-    how retrieved memory is structured and integrated.
VL-7B backbone.                                         This observation holds consistently across both tex-
   Impact of Retrieval Size on Downstream Task          tual and multimodal memory methods.
Performance. The results in Table 11, Table 3,             Knowledge management tasks, such as knowl-
Table 12, and Table 13 provide a continuous view        edge resolution, conflict detection, and answer re-
of how retrieval quantity affects memory utiliza-       fusal, exhibit the weakest sensitivity to retrieval
tion and task performance. Across most memory           size. Across all values of K, task performance
models and task categories, we generally observe a      remains relatively stable, indicating that simply
non-monotonic relationship between retrieval size       retrieving more memory entries is insufficient to
and downstream performance. Increasing the re-          improve consistency-oriented behaviors and knowl-
trieval size from a small value (K=5) to a moderate     edge management capability.
range (K=10 or K=15) generally leads to consis-            Overall, these results suggest that moderate re-
tent improvements, especially for memory extrac-        trieval sizes, exemplified by K=10, often provide
tion and adaptation tasks such as factual retrieval,    the most effective trade-off between retrieval cov-
visual-centric search, and test-time learning.          erage and memory usability. Increasing retrieval
   However, further increasing the retrieval size to    size beyond this range yields diminishing returns,
K=20 does not consistently yield additional gains.      particularly for multimodal memory systems, high-
As shown in Table 13, performance often satu-           lighting the importance of selective and relevance-
rates or fluctuates across tasks, and in some cases     aware retrieval rather than indiscriminate scaling
slightly degrades compared to the K=10 or K=15          of retrieved memory entries.
settings. This pattern indicates that retrieving more      Retrieval Effectiveness versus Memory Uti-
memory entries beyond a moderate threshold does         lization. Table 14 provides further insight into why
not necessarily translate into better task outcomes.    increasing retrieval size does not consistently im-
This effect is particularly pronounced for multi-       prove downstream task performance observed in
Table 8: Evaluation results on Qwen-2.5-VL-3B. The best and second-performed memory model(s) are highlighted
with orange and blue backgrounds.

                       Qwen-2.5-VL-3B    Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT     A-Mem    MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                F1        0.2501       0.1578    0.5664      0.2301      0.2467     0.4171     0.5985    0.5889     0.2282     0.6504      0.6601      0.6045    0.5660
                              BLEU-1      0.2000       0.1138    0.4862      0.1847      0.1976     0.3494     0.4937    0.4897     0.1795     0.5673      0.5811      0.5290    0.4868
                       FR      EM         0.0959       0.0502    0.2740      0.0868      0.0959     0.1872     0.2843    0.3014     0.0868     0.3699      0.3699      0.3470    0.3105
                             LLM-Judge    0.3105       0.1918    0.7443      0.2922      0.3082     0.5845     0.7690    0.7694     0.2717     0.8584      0.8516      0.7694    0.7626
Extract. & Adapt.




                                F1        0.2687       0.1175    0.7424      0.3244      0.2691     0.2631     0.7653    0.7771     0.2154     0.8661      0.8395      0.8493    0.8537
                              BLEU-1      0.2466       0.1012    0.6934      0.2826      0.2471     0.2381     0.7096    0.7123     0.1959     0.8337      0.8056      0.8066    0.8047
                       VS      EM         0.1569       0.0359    0.5033      0.1634      0.1601     0.1176     0.5802    0.5458     0.0980     0.6307      0.6275      0.6569    0.6242
                             LLM-Judge    0.2337       0.0801    0.7092      0.2696      0.2337     0.2843     0.6794    0.7565     0.1797     0.8676      0.8350      0.8284    0.8464
                                F1        0.4657       0.3640    0.6769      0.5454      0.4633     0.4833     0.6448    0.6595     0.4857     0.7966      0.7227      0.7732    0.7328
                              BLEU-1      0.4279       0.3161    0.6253      0.4845      0.4251     0.4427     0.5884    0.5997     0.4397     0.7402      0.6655      0.7145    0.6803
                       TTL     EM         0.3294       0.2344    0.5104      0.3769      0.3234     0.3620     0.4927    0.4777     0.3353     0.6261      0.5668      0.6083    0.5786
                             LLM-Judge    0.6365       0.5415    0.7433      0.6706      0.6380     0.6276     0.7080    0.7567     0.6320     0.8338      0.7982      0.8294    0.8027
                                F1        0.2866       0.2398    0.4785      0.3288      0.2866     0.3996     0.6036    0.4942     0.2972     0.5700      0.5484      0.5332    0.5422
                              BLEU-1      0.2688       0.2226    0.4454      0.3036      0.2683     0.3728     0.5817    0.4654     0.2807     0.5369      0.5119      0.4986    0.5072
                       TR      EM         0.1301       0.0976    0.3089      0.1626      0.1301     0.2114     0.4569    0.3089     0.1382     0.3984      0.3821      0.3659    0.3740
                             LLM-Judge    0.3252       0.2480    0.6667      0.3821      0.3211     0.5813     0.6940    0.6707     0.3130     0.7602      0.7886      0.6951    0.7561
                                F1        0.2545       0.1532    0.2966      0.1953      0.2552     0.3226     0.3441    0.3860     0.2285     0.4440      0.4554      0.4260     0.4681
Reasoning




                              BLEU-1      0.2398       0.1384    0.2778      0.1823      0.2395     0.3065     0.3319    0.3685     0.2176     0.4230      0.4352      0.4100     0.4504
                       VR      EM         0.2011       0.1092    0.1897      0.1379      0.2011     0.2414     0.2806    0.2931     0.1839     0.3333      0.3333      0.3333     0.3563
                             LLM-Judge    0.3161       0.1810    0.3822      0.2615      0.3190     0.4310     0.4101    0.5172     0.2615     0.5374      0.5489      0.5201     0.5690
                                F1        0.2600       0.1934    0.4380      0.2683      0.2615     0.3400     0.4401    0.4483     0.2400     0.4851      0.4908      0.4697    0.4773
                              BLEU-1      0.1943       0.1364    0.3277      0.1997      0.1969     0.2599     0.3162    0.3474     0.1787     0.3736      0.3788      0.3602    0.3639
                       MR      EM         0.0291       0.0243    0.0825      0.0388      0.0291     0.0340     0.0722    0.0971     0.0194     0.0874      0.0922      0.0874    0.1019
                             LLM-Judge    0.4442       0.2985    0.7670      0.4345      0.4442     0.6092     0.7474    0.7597     0.4223     0.8083      0.8301      0.7985    0.7840
                                F1        0.2034       0.1563    0.3382      0.2613      0.2061     0.2372     0.4280    0.3706     0.2007     0.3975      0.3883      0.3441    0.3604
                              BLEU-1      0.1627       0.1118    0.2754      0.2085      0.1639     0.1869     0.3693    0.3094     0.1625     0.3358      0.3323      0.2842    0.2987
                       KR      EM         0.0741       0.0370    0.1111      0.0864      0.0741     0.0741     0.2267    0.1235     0.0864     0.1481      0.1728      0.1358    0.1235
Knowledge Management




                             LLM-Judge    0.4012       0.2963    0.5926      0.4012      0.3951     0.5185     0.6600    0.6852     0.3704     0.6790      0.6852      0.6420    0.6914
                                F1        0.3333       0.3580    0.3210      0.3210      0.3333      0.3580    0.3200    0.2963     0.3457     0.3086      0.3086      0.3086    0.3333
                              BLEU-1      0.3333       0.3580    0.3210      0.3210      0.3333      0.3580    0.3200    0.2963     0.3457     0.3086      0.3086      0.3086    0.3333
                       CD      EM         0.3333       0.3580    0.3210      0.3210      0.3333      0.3580    0.3200    0.2963     0.3457     0.3086      0.3086      0.3086    0.3333
                             LLM-Judge    0.3333       0.3580    0.3210      0.3210      0.3333      0.3580    0.3200    0.2963     0.3457     0.3086      0.3086      0.3086    0.3333
                                F1         0.9742      0.9783    0.8942      0.9578      0.9742     0.8387     0.9080    0.8992     0.9647     0.8823      0.8661      0.8707    0.8553
                              BLEU-1       0.9730      0.9783    0.8924      0.9571      0.9730     0.8355     0.9080    0.8982     0.9628     0.8811      0.8648      0.8702    0.8540
                       AR      EM          0.9728      0.9783    0.8913      0.9565      0.9728     0.8315     0.9080    0.8967     0.9620     0.8804      0.8641      0.8696    0.8533
                             LLM-Judge     0.9620      0.9620    0.8804      0.9429      0.9620     0.8234     0.9052    0.8804     0.9484     0.8641      0.8478      0.8587    0.8424
                                F1        0.3798       0.2986    0.5832      0.4013      0.3793     0.4165     0.6059    0.6013     0.3665     0.6679      0.6479      0.6443    0.6383
                              BLEU-1      0.3492       0.2689    0.5333      0.3620      0.3487     0.3794     0.5505    0.5462     0.3360     0.6194      0.5995      0.5953    0.5881
         Overall               EM         0.2624       0.2040    0.3933      0.2671      0.2618     0.2700     0.4303    0.4103     0.2496     0.4728      0.4594      0.4670    0.4541
                             LLM-Judge    0.4541       0.3521    0.6856      0.4597      0.4538     0.5383     0.6886    0.7162     0.4272     0.7756      0.7662      0.7463    0.7528




Table 3 and Table 11-13. As the retrieval size K                                                              in Table 13 often saturates or fluctuates relative to
increases, retrieval effectiveness metrics exhibit a                                                          Table 3 and Table 12. In other words, higher re-
clear and expected trend: Recall@K and Hit@K                                                                  call does not guarantee better memory utilization.
consistently improve across all retrievers, indicat-                                                          The additional retrieved entries do not consistently
ing that more relevant memory entries are success-                                                            contribute useful evidence for reasoning or answer
fully retrieved. This trend holds for both textual                                                            generation, and may instead distract the model dur-
and multimodal retrievers, and is especially pro-                                                             ing inference.
nounced for multimodal methods such as MuRAG
                                                                                                                 The effect is more evident for multimodal
and UniversalRAG, which achieve high recall even
                                                                                                              memory systems. Compared to textual memory,
at moderate K value.
                                                                                                              multimodal retrieval introduces heterogeneous vi-
                                                                                                              sual–textual information, making it more sensitive
   However, this improvement in recall is accom-
                                                                                                              to redundancy and misalignment. Consequently,
panied by a systematic decline in Precision@K,
                                                                                                              multimodal methods benefit most from moderate
particularly as K increases beyond moderate val-
                                                                                                              retrieval sizes that balance recall and precision, as
ues. While larger K values ensure broader cover-
                                                                                                              exemplified by the strong performance at K=10.
age of relevant memory entries, they also introduce
                                                                                                              Beyond this range, further recall gains are offset by
a growing number of weakly relevant or redundant
                                                                                                              increased retrieval noise, limiting improvements in
items. As a result, the retrieved memory becomes
                                                                                                              downstream tasks.
increasingly noisy, especially in multimodal set-
tings where cross-modal alignment is imperfect.                                                                  Overall, these results indicate that retrieval qual-
This divergence explains the observed decoupling                                                              ity, rather than retrieval quantity, is the domi-
between retrieval effectiveness and downstream                                                                nant factor governing downstream memory per-
task performance. Although Recall@K continues                                                                 formance. Simply scaling the retrieval size im-
to improve from K=10 to K=20, task performance                                                                proves recall metrics but does not necessarily im-
Table 9: Evaluation results on GPT-4.1-Nano. The best and second-performed memory model(s) are highlighted
with orange and blue backgrounds.

                        GPT-4.1-Nano     Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT     A-Mem     MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                F1        0.2279       0.1379    0.5498      0.1672      0.2277     0.5983     0.6048     0.5848     0.1920     0.6256      0.6060      0.5922    0.5666
                              BLEU-1      0.1724       0.0907    0.4682      0.1164      0.1719     0.5193     0.5194     0.4936     0.1413     0.5392      0.5189      0.5036    0.4734
                       FR      EM         0.0685       0.0274    0.2374      0.0228      0.0594     0.2740     0.2648     0.2466     0.0548     0.2785      0.2511      0.2785    0.2466
                             LLM-Judge    0.2260       0.0936    0.7740      0.1918      0.2283     0.8082     0.8151     0.8425     0.1872     0.8653      0.8356      0.7968    0.7922
Extract. & Adapt.




                                F1        0.2330       0.0839    0.7457      0.1787      0.2356     0.6853     0.7965     0.7639     0.1318     0.8478      0.8410      0.8329    0.8354
                              BLEU-1      0.2175       0.0689    0.7021      0.1507      0.2210     0.6573     0.7553     0.6962     0.1155     0.8092      0.8109      0.7906    0.7831
                       VS      EM         0.1732       0.0294    0.5294      0.0980      0.1732     0.5392     0.5882     0.5621     0.0588     0.6275      0.6340      0.6340    0.6373
                             LLM-Judge    0.2124       0.0507    0.6944      0.1585      0.2124     0.6536     0.7761     0.7255     0.0997     0.8431      0.8252      0.8056    0.8121
                                F1        0.2278       0.1473    0.3618      0.2021      0.2226     0.2866     0.3279     0.3263     0.1991     0.4395      0.4407      0.5273    0.4488
                              BLEU-1      0.1609       0.0974    0.2523      0.1414      0.1567     0.1969     0.2232     0.2346     0.1406     0.3468      0.3581      0.4372    0.3491
                       TTL     EM         0.0504       0.0148    0.0504      0.0297      0.0445     0.0386     0.0326     0.0742     0.0356     0.1662      0.1958      0.2641    0.1424
                             LLM-Judge    0.5742       0.4585    0.7834      0.5801      0.5579     0.6588     0.8027     0.7389     0.5015     0.7804      0.8012      0.8709    0.8145
                                F1        0.2035       0.1356    0.4381      0.2028      0.1997     0.4306      0.5116    0.5017     0.1916     0.5007      0.4366      0.4924    0.4808
                              BLEU-1      0.1744       0.1143    0.4032      0.1746      0.1694     0.3932      0.4730    0.4649     0.1682     0.4676      0.3997      0.4606    0.4484
                       TR      EM         0.0569       0.0488    0.2764      0.0894      0.0650     0.2520      0.3415    0.3171     0.0650     0.3252      0.2683      0.3496    0.3252
                             LLM-Judge    0.2764       0.1382    0.7154      0.2886      0.2846     0.7276      0.7846    0.8130     0.2398     0.7520      0.7480      0.7236    0.7154
                                F1        0.2011       0.0714    0.2041      0.0987      0.2000      0.4140    0.3269     0.3697     0.0909     0.3426      0.4417      0.3043    0.3817
Reasoning




                              BLEU-1      0.1796       0.0491    0.1791      0.0747      0.1772      0.3855    0.2979     0.3431     0.0690     0.3077      0.4146      0.2738    0.3518
                       VR      EM         0.1379       0.0172    0.0862      0.0172      0.1322      0.2931    0.1954     0.2586     0.0345     0.2069      0.3218      0.1724    0.2529
                             LLM-Judge    0.3276       0.1379    0.3736      0.1810      0.3305      0.6207    0.5661     0.5862     0.2011     0.5575      0.6638      0.4713    0.5747
                                F1        0.2415       0.1814    0.4490      0.2354      0.2389     0.4428     0.4669     0.4523     0.2017     0.5024      0.4670      0.4570    0.4829
                              BLEU-1      0.1720       0.1174    0.3402      0.1704      0.1710     0.3388     0.3602     0.3479     0.1385     0.3946      0.3670      0.3506    0.3798
                       MR      EM         0.0194       0.0097    0.0388      0.0097      0.0146     0.0388     0.0437     0.0583     0.0097     0.0534      0.0534      0.0534    0.0680
                             LLM-Judge    0.2451       0.0922    0.7985      0.2597      0.2500     0.7985     0.8617     0.8325     0.1942     0.8617      0.7816      0.8180    0.8180
                                F1        0.1454       0.1001    0.2852      0.1408      0.1411      0.3733    0.3283     0.2873     0.1293     0.3626      0.2972      0.2878    0.2607
                              BLEU-1      0.0965       0.0633    0.2174      0.0960      0.0934      0.3017    0.2556     0.2190     0.0902     0.2897      0.2316      0.2252    0.1958
                       KR      EM         0.0123       0.0000    0.0494      0.0000      0.0123      0.1111    0.0741     0.0370     0.0123     0.0988      0.0494      0.0864    0.0617
Knowledge Management




                             LLM-Judge    0.2840       0.1975    0.6049      0.2407      0.2963      0.7593    0.6852     0.6790     0.2531     0.7469      0.6543      0.6173    0.5741
                                F1        0.3704       0.3704    0.4321      0.4074      0.3704     0.3951      0.4321    0.4444     0.3704     0.4074      0.3827      0.4321    0.3951
                              BLEU-1      0.3704       0.3704    0.4321      0.4074      0.3704     0.3951      0.4321    0.4444     0.3704     0.4074      0.3827      0.4321    0.3951
                       CD      EM         0.3704       0.3704    0.4321      0.4074      0.3704     0.3951      0.4321    0.4444     0.3704     0.4074      0.3827      0.4321    0.3951
                             LLM-Judge    0.3704       0.3704    0.4321      0.4074      0.3704     0.3951      0.4321    0.4444     0.3704     0.4074      0.3827      0.4321    0.3951
                                F1        0.9682       0.9891    0.8406      0.9751      0.9737     0.7911     0.8496     0.8874     0.9791     0.9039      0.9102      0.9382    0.9058
                              BLEU-1      0.9679       0.9891    0.8323      0.9738      0.9733     0.7811     0.8430     0.8831     0.9788     0.9003      0.9058      0.9335    0.9009
                       AR      EM         0.9674       0.9891    0.8261      0.9728      0.9728     0.7717     0.8370     0.8804     0.9783     0.8967      0.9022      0.9293    0.8967
                             LLM-Judge    0.9565       0.9728    0.8587      0.9647      0.9592     0.8370     0.8668     0.8940     0.9620     0.9022      0.9103      0.9321    0.9022
                                F1        0.3084       0.2291    0.5057      0.2769      0.3075     0.5034     0.5380     0.5339     0.2636     0.5832      0.5774      0.5849    0.5703
                              BLEU-1      0.2703       0.1974    0.4436      0.2389      0.2698     0.4480     0.4764     0.4705     0.2293     0.5243      0.5236      0.5266    0.5080
         Overall               EM         0.1923       0.1420    0.2800      0.1596      0.1899     0.2987     0.3092     0.3203     0.1572     0.3518      0.3600      0.3746    0.3489
                             LLM-Judge    0.3966       0.2779    0.7046      0.3720      0.3960     0.7063     0.7651     0.7507     0.3346     0.7814      0.7747      0.7659    0.7583




prove task outcomes. Effective multimodal mem-                                                                   Figure 16 presents a test-time learning scenario
ory systems therefore require selective, relevance-                                                           in which the agent must adapt to newly introduced
aware retrieval strategies that prioritize precision                                                          visual exemplars during the conversation and gen-
and alignment over exhaustive retrieval. Therefore,                                                           eralize the learned concept at inference time. The
this highlights the need for future work to consider                                                          example requires identifying the clothing category
the principle that “less but high-quality is more” in                                                         based on a sequence of visual demonstrations and
multimodal memory design (Chen et al., 2025).                                                                 prior conversational cues. Although textual mem-
                                                                                                              ory baselines are augmented with high-quality vi-
A.7                          Case Study
                                                                                                              sual captions, textual memory methods still exhibit
We further present a qualitative case study to illus-                                                         clear limitations. Although NaiveRAG and A-Mem
trate how representative memory systems behave                                                                retrieve relevant textual clues, they lack the ability
under different multimodal long-term memory chal-                                                             to integrate newly observed visual patterns, result-
lenges in Mem-Gallery. We select three textual                                                                ing in generic or incorrect predictions. MemoryOS
memory models (NaiveRAG, A-Mem, MemoryOS)                                                                     exhibits a similar limitation, indicating that strong
and three multimodal memory models (MuRAG,                                                                    textual organization alone is insufficient when the
UniversalRAG, AUGUSTUS) and analyze their re-                                                                 task requires visual abstraction at inference time.
sponses on three representative subtasks. These                                                               In contrast, multimodal memory methods that ex-
cases correspond to test-time learning of the mem-                                                            plicitly preserve visual representations demonstrate
ory extraction & adaptation task, visual-centric rea-                                                         stronger adaptability. MuRAG and AUGUSTUS
soning of the memory reasoning task, and knowl-                                                               successfully align the new image with previously
edge resolution of the memory knowledge manage-                                                               observed visual examples and correctly infer the
ment task, as shown in Figure 16, Figure 17, and                                                              category. This case highlights the limitation of
Figure 18, respectively. From the case studies, we                                                            caption-based textual memory. While high-quality
can have the following observations.
Table 10: Evaluation results on Gemini-2.5-Flash-Lite. The best and second-performed memory model(s) are
highlighted with orange and blue backgrounds.

                       Gemini-2.5-Flash-Lite   Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT     A-Mem     MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                  F1            0.2014       0.1014    0.5905      0.2329      0.2014     0.6667      0.5266    0.6622     0.1534     0.6888      0.7014      0.6341    0.6011
                                BLEU-1          0.1652       0.0732    0.4999      0.1802      0.1652     0.5633      0.4270    0.5683     0.1214     0.6009      0.6136      0.5510    0.5141
                        FR       EM             0.0822       0.0320    0.2648      0.0868      0.0822     0.2557      0.1690    0.3425     0.0594     0.3653      0.3562      0.3333    0.3151
                               LLM-Judge        0.2237       0.0868    0.7922      0.2945      0.2215     0.9475      0.8402    0.8562     0.1507     0.8927      0.9064      0.8082    0.7922
Extract. & Adapt.




                                  F1            0.2252       0.0598    0.7909      0.3450      0.2252      0.9204     0.8451    0.8427     0.1051     0.9077      0.9175      0.8753    0.8811
                                BLEU-1          0.2102       0.0553    0.7539      0.3156      0.2102      0.8957     0.8019    0.7918     0.0929     0.8837      0.8919      0.8456    0.8443
                        VS       EM             0.1863       0.0327    0.6078      0.2451      0.1863      0.8105     0.6797    0.6536     0.0654     0.7909      0.7745      0.7484    0.7157
                               LLM-Judge        0.2141       0.0458    0.7663      0.3170      0.2141      0.9265     0.8415    0.8350     0.0931     0.9052      0.9199      0.8611    0.8775
                                  F1            0.4697       0.2002    0.7839      0.6149      0.4697     0.8517      0.7173    0.8360     0.3031     0.8750      0.8464      0.8477    0.8549
                                BLEU-1          0.4322       0.1260    0.7259      0.5552      0.4322     0.8012      0.6229    0.7709     0.2654     0.8130      0.7689      0.7745    0.7908
                        TTL      EM             0.3501       0.0712    0.6113      0.4481      0.3501     0.6795      0.4243    0.6380     0.1958     0.6677      0.6410      0.6350    0.6499
                               LLM-Judge        0.5341       0.2507    0.8680      0.6869      0.5341     0.9021      0.9006    0.9021     0.3442     0.9258      0.8976      0.9184    0.9139
                                  F1            0.2105       0.1447    0.4801      0.2201      0.2105     0.5475      0.4714    0.5325     0.1796     0.5806      0.5675      0.5280    0.5555
                                BLEU-1          0.1953       0.1345    0.4513      0.2046      0.1953     0.5071      0.4301    0.5047     0.1637     0.5507      0.5392      0.5011    0.5267
                        TR       EM             0.0976       0.0407    0.3333      0.0732      0.0976     0.3496      0.2927    0.3821     0.0650     0.4309      0.4146      0.3984    0.4065
                               LLM-Judge        0.2317       0.1423    0.6870      0.2846      0.2317     0.8902      0.7967    0.7358     0.1911     0.7886      0.8089      0.6992    0.7886
                                  F1            0.1909       0.0423    0.3135      0.1466      0.1909      0.6294     0.4262    0.3997     0.0798     0.4470      0.4954      0.4422    0.4533
Reasoning




                                BLEU-1          0.1819       0.0356    0.2985      0.1398      0.1819      0.6083     0.4075    0.3843     0.0707     0.4267      0.4785      0.4200    0.4325
                        VR       EM             0.1494       0.0230    0.2241      0.1092      0.1494      0.5230     0.3276    0.3103     0.0517     0.3563      0.4023      0.3563    0.3563
                               LLM-Judge        0.2443       0.0402    0.4138      0.2443      0.2443      0.7845     0.5603    0.5805     0.1063     0.6264      0.6782      0.5920    0.5718
                                  F1            0.1648       0.0834    0.4521      0.1765      0.1648     0.5210      0.4529    0.5161     0.1268     0.5137      0.5137      0.4783    0.4691
                                BLEU-1          0.1250       0.0594    0.3508      0.1265      0.1250     0.4130      0.3553    0.4111     0.0987     0.4032      0.4011      0.3703    0.3575
                        MR       EM             0.0097       0.0097    0.0631      0.0097      0.0097     0.0680      0.0388    0.0825     0.0097     0.0680      0.0728      0.0437    0.0631
                               LLM-Judge        0.2184       0.0922    0.7767      0.2743      0.2184     0.9272      0.8519    0.8568     0.1481     0.8714      0.8738      0.8058    0.7864
                                  F1            0.1157       0.0604    0.2813      0.1050      0.1157      0.3915     0.3341    0.3609     0.0918     0.4037      0.3710      0.3111    0.3168
                                BLEU-1          0.0848       0.0370    0.2177      0.0810      0.0848      0.3225     0.2644    0.2956     0.0673     0.3534      0.3156      0.2607    0.2631
                        KR       EM             0.0247       0.0000    0.0864      0.0247      0.0247      0.1605     0.0988    0.1605     0.0247     0.1975      0.1481      0.1358    0.1235
Knowledge Management




                               LLM-Judge        0.2531       0.1296    0.6235      0.2037      0.2531      0.8025     0.7593    0.7840     0.1728     0.7716      0.7222      0.5988    0.6667
                                  F1            0.3951       0.3827    0.5185      0.4198      0.3951     0.5556      0.4938    0.6049     0.3827     0.5309      0.5432      0.6296    0.5679
                                BLEU-1          0.3951       0.3827    0.5185      0.4198      0.3951     0.5556      0.4938    0.6049     0.3827     0.5309      0.5432      0.6296    0.5679
                        CD       EM             0.3951       0.3827    0.5185      0.4198      0.3951     0.5556      0.4938    0.6049     0.3827     0.5309      0.5432      0.6296    0.5679
                               LLM-Judge        0.3951       0.3827    0.5185      0.4198      0.3951     0.5556      0.5123    0.6049     0.3827     0.5309      0.5432      0.6296    0.5679
                                  F1            0.9644       0.9946    0.9122      0.9688      0.9644     0.8512      0.9175    0.9220     0.9844     0.9242      0.9186      0.9332    0.9339
                                BLEU-1          0.9634       0.9946    0.9102      0.9682      0.9634     0.8454      0.9155    0.9205     0.9841     0.9218      0.9162      0.9318    0.9321
                        AR       EM             0.9620       0.9946    0.9076      0.9674      0.9620     0.8370      0.9130    0.9185     0.9837     0.9185      0.9130      0.9293    0.9293
                               LLM-Judge        0.9511       0.9783    0.8995      0.9592      0.9511     0.8478      0.9212    0.9212     0.9674     0.9130      0.9076      0.9212    0.9239
                                  F1            0.3408       0.2158    0.6282      0.3936      0.3408      0.7202     0.6294    0.6861     0.2627     0.7155      0.7157      0.6901    0.6877
                                BLEU-1          0.3178       0.1913    0.5796      0.3608      0.3178      0.6706     0.5702    0.6327     0.2424     0.6676      0.6644      0.6400    0.6370
         Overall                 EM             0.2595       0.1555    0.4436      0.2858      0.2595      0.5219     0.4120    0.4904     0.1940     0.5283      0.5207      0.5079    0.5020
                               LLM-Judge        0.3729       0.2236    0.7452      0.4407      0.3726      0.8755     0.8115    0.8165     0.2764     0.8437      0.8472      0.8030    0.8057




captions can convey semantic details, they remain                                                                   the retrieved textual clue and the visual evidence
a lossy and static representation of visual infor-                                                                  to reach the correct conclusion. This case exposes
mation, making it difficult for models to perform                                                                   a critical gap in existing memory systems, namely
visual pattern induction and concept generalization                                                                 that multimodal reasoning requires not only retriev-
at test time. Effective test-time learning therefore                                                                ing relevant information but also jointly reasoning
requires memory systems that can retain and com-                                                                    over visual and textual cues in a coherent manner.
pare visual evidence directly, rather than relying                                                                     Figure 18 focuses on knowledge resolution under
solely on textualized visual descriptions.                                                                          evolving conversational states. In this example, the
   Figure 17 illustrates a visual-centric reasoning                                                                 user explicitly corrects earlier information about
task, where the model must determine whether the                                                                    a preferred dog breed. The model must update its
dog in a given image matches the breed of a pet                                                                     memory and discard outdated knowledge. Most
mentioned earlier in the conversation. This case                                                                    evaluated methods struggle with this scenario. This
reveals a more subtle failure mode. Several models                                                                  example reveals two distinct failure modes. First,
retrieve the correct conversational evidence, such                                                                  some memory systems fail to update their knowl-
as the mention that Amy owns a Cairn Terrier, yet                                                                   edge, continuing to rely on outdated beliefs despite
still produce incorrect final answers. In particular,                                                               the explicit correction in the dialogue. This be-
UniversalRAG demonstrates correct retrieval but                                                                     havior is particularly evident in some multimodal
flawed reasoning, incorrectly concluding that the                                                                   memory methods, indicating that preserving mul-
dog in the image is not a Cairn Terrier despite vi-                                                                 timodal information alone does not guarantee ef-
sual similarity. This indicates that retrieval success                                                              fective knowledge revision. Second, even when
does not guarantee correct multimodal reasoning.                                                                    some textual memory models successfully incor-
Textual memory models, on the other hand, of-                                                                       porate the updated preference at the textual level,
ten fail earlier due to missing or incomplete visual                                                                they still fail to produce the correct answer due
grounding. Only MuRAG correctly integrates both                                                                     to insufficient visual grounding. Lacking the abil-
Table 11: Benchmark performance results on Qwen-2.5-VL-7B with the retrieval number K=5. The best and
second-performed memory model(s) are highlighted with orange and blue backgrounds.

                       Qwen-2.5-VL-7B (K=5)   Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT    A-Mem    MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                  F1           0.2376       0.1075    0.5293      0.2216      0.2391     0.5928    0.5562    0.6045     0.2150     0.6291      0.6321      0.5536    0.5745
                                BLEU-1         0.1865       0.0683    0.4531      0.1684      0.1903     0.5098    0.4668    0.5229     0.1626     0.5440      0.5464      0.4695    0.4870
                        FR       EM            0.0913       0.0228    0.2785      0.0685      0.0913     0.3288    0.2603    0.3288     0.0685     0.3516      0.3379      0.3105    0.2968
                               LLM-Judge       0.2626       0.0662    0.7146      0.2580      0.2626     0.8539    0.7100    0.7877     0.2260     0.8105      0.8059      0.7123    0.7489
Extract. & Adapt.




                                  F1           0.1992       0.0398    0.6864      0.2416      0.1954     0.6239    0.7366    0.7614     0.1658     0.8664      0.8740      0.7847    0.8301
                                BLEU-1         0.1873       0.0322    0.6307      0.2082      0.1840     0.5834    0.6596    0.7067     0.1473     0.8301      0.8387      0.7417    0.7731
                        VS       EM            0.1601       0.0131    0.4967      0.1634      0.1569     0.4118    0.5719    0.5817     0.1078     0.6928      0.6993      0.6340    0.6569
                               LLM-Judge       0.1961       0.0343    0.6650      0.2288      0.1895     0.5964    0.7059    0.7418     0.1683     0.8725      0.8627      0.7778    0.8235
                                  F1           0.4500       0.3240    0.5938      0.4731      0.4486     0.2924    0.6882    0.5512     0.4147     0.8161      0.7853      0.7706    0.7631
                                BLEU-1         0.3799       0.2557    0.5170      0.3980      0.3798     0.2295    0.6261    0.4759     0.3477     0.7464      0.7068      0.7027    0.6810
                        TTL      EM            0.2374       0.0742    0.3858      0.2493      0.2374     0.1009    0.5134    0.3650     0.2107     0.6024      0.5846      0.6053    0.5519
                               LLM-Judge       0.7092       0.6424    0.8353      0.7626      0.7033     0.7092    0.8220    0.7774     0.7107     0.9021      0.8620      0.8887    0.8754
                                  F1           0.2545       0.1484    0.4088      0.2017      0.2553     0.5661    0.5235    0.5598     0.2294     0.5439      0.5233      0.5199     0.5779
                                BLEU-1         0.2363       0.1305    0.3872      0.1836      0.2363     0.5326    0.4959    0.5306     0.2065     0.5211      0.4955      0.4952     0.5475
                        TR       EM            0.1545       0.0732    0.3089      0.1138      0.1545     0.3496    0.4065    0.3902     0.1382     0.4146      0.3740      0.3984     0.4309
                               LLM-Judge       0.2805       0.1341    0.5325      0.2276      0.2764     0.8008    0.6626    0.7276     0.2480     0.6707      0.6870      0.6463     0.7317
                                  F1           0.2552       0.0900    0.2711      0.1669      0.2594      0.4593   0.3963    0.4243     0.2015     0.4390      0.4205      0.3951    0.3950
Reasoning




                                BLEU-1         0.2442       0.0758    0.2548      0.1536      0.2480      0.4459   0.3797    0.4108     0.1912     0.4175      0.3969      0.3767    0.3795
                        VR       EM            0.2011       0.0460    0.1552      0.0977      0.2011      0.3851   0.2989    0.3391     0.1609     0.3276      0.3046      0.3103    0.3103
                               LLM-Judge       0.3046       0.1264    0.2931      0.1954      0.3046      0.6149   0.5000    0.5489     0.2586     0.5632      0.5632      0.4598    0.4885
                                  F1           0.2411       0.1516    0.4056      0.2116      0.2428     0.4367    0.4503    0.4385     0.2101     0.4980      0.4954      0.4343    0.4756
                                BLEU-1         0.1739       0.0982    0.3012      0.1497      0.1770     0.3347    0.3476    0.3355     0.1429     0.3915      0.3871      0.3295    0.3667
                        MR       EM            0.0340       0.0049    0.0631      0.0194      0.0340     0.0631    0.0777    0.0680     0.0194     0.0971      0.0971      0.0631    0.0874
                               LLM-Judge       0.2985       0.1238    0.6917      0.2767      0.3058     0.8204    0.7379    0.7961     0.2791     0.8228      0.8204      0.7063    0.7888
                                  F1           0.2354       0.1805    0.3331      0.2429      0.2354     0.4292    0.3863    0.4686     0.2181     0.4176      0.4062      0.3835    0.3707
                                BLEU-1         0.2005       0.1521    0.2798      0.2044      0.2000     0.3741    0.3305    0.4087     0.1883     0.3660      0.3606      0.3346    0.3152
                        KR       EM            0.1235       0.0988    0.1358      0.0988      0.1235     0.2099    0.1975    0.2222     0.1235     0.2099      0.1975      0.2099    0.1481
Knowledge Management




                               LLM-Judge       0.3395       0.2160    0.5988      0.3210      0.3395     0.7407    0.5988    0.7593     0.2840     0.7469      0.6852      0.6358    0.6173
                                  F1            0.3457      0.3580    0.3333      0.3457      0.3333      0.3580   0.3457    0.3457     0.3580     0.3580      0.3457      0.3457    0.3210
                                BLEU-1          0.3457      0.3580    0.3333      0.3457      0.3333      0.3580   0.3457    0.3457     0.3580     0.3580      0.3457      0.3457    0.3210
                        CD       EM             0.3457      0.3580    0.3333      0.3457      0.3333      0.3580   0.3457    0.3457     0.3580     0.3580      0.3457      0.3457    0.3210
                               LLM-Judge        0.3457      0.3580    0.3333      0.3457      0.3333      0.3580   0.3457    0.3457     0.3580     0.3580      0.3457      0.3457    0.3210
                                  F1            0.9958      1.0000    0.9581      0.9891      0.9958     0.9849    0.9159    0.9785     0.9946     0.9532      0.9421      0.9749    0.9461
                                BLEU-1          0.9953      1.0000    0.9575      0.9891      0.9953     0.9844    0.9141    0.9784     0.9946     0.9524      0.9413      0.9741    0.9459
                        AR       EM             0.9946      1.0000    0.9565      0.9891      0.9946     0.9837    0.9130    0.9783     0.9946     0.9511      0.9402      0.9728    0.9457
                               LLM-Judge        0.9783      0.9837    0.9402      0.9728      0.9783     0.9674    0.9375    0.9620     0.9837     0.9429      0.9375      0.9565    0.9429
                                  F1           0.3625       0.2558    0.5478      0.3560      0.3619     0.5282    0.6038    0.6021     0.3354     0.6791      0.6688      0.6322    0.6457
                                BLEU-1         0.3279       0.2255    0.4946      0.3164      0.3279     0.4792    0.5475    0.5483     0.2999     0.6288      0.6164      0.5817    0.5887
         Overall                 EM            0.2519       0.1596    0.3711      0.2350      0.2507     0.3402    0.4296    0.4208     0.2279     0.4915      0.4798      0.4711    0.4611
                               LLM-Judge       0.4331       0.3115    0.6715      0.4299      0.4307     0.7306    0.7119    0.7463     0.4129     0.7957      0.7823      0.7329    0.7586




ity to accurately recognize or verify the dog breed                                                            Commons licenses were used. All conversations
shown in the image, these models are unable to                                                                 and question–answer pairs were reviewed by an-
align the updated textual knowledge with the vi-                                                               notators to remove inaccurate, misleading, or in-
sual evidence, leading to incorrect final predictions.                                                         appropriate content and to avoid the inclusion of
This case demonstrates that effective knowledge                                                                sensitive material. The benchmark construction
resolution requires both explicit mechanisms for                                                               process does not involve real user interactions or
memory revision and robust visual understanding                                                                private user data. Thus, the dataset does not involve
to ground updated knowledge in perceptual evi-                                                                 data privacy violations and is intended solely for
dence, which remains largely underexplored in cur-                                                             research and benchmarking purposes.
rent multimodal designs.
                                                                                                               A.8.2        Annotation
A.8                           Ethics Statement
                                                                                                               All annotation and verification work in this bench-
A.8.1                          Data Privacy                                                                    mark was conducted exclusively for data quality
To ensure data privacy, the benchmark does not in-                                                             control and validation purposes. The annotation
clude any personally identifiable information from                                                             tasks were limited to checking the correctness, con-
real user conversations. All conversations were                                                                sistency, and completeness of conversational data
either generated by LLMs or reconstructed from                                                                 and model-generated outputs, including verifying
open-sourced paper repositories, and were subse-                                                               dialogue coherence across sessions, validating mul-
quently verified and edited under strong human                                                                 timodal grounding, and reviewing question–answer
supervision by the author team. The majority of                                                                pairs for clarity and correctness. Annotators were
images and portions of single-session dialogue data                                                            treated solely as an interface for data verification
were collected from open-source repositories. For                                                              rather than as research subjects, and no analysis
newly collected images sourced from the web, only                                                              of annotator behavior, agreement patterns, or in-
materials released under permissive or Creative                                                                dividual decision strategies was performed. No
Table 12: Benchmark performance results on Qwen-2.5-VL-7B with the retrieval number K=15. The best and
second-performed memory model(s) are highlighted with orange and blue backgrounds.

                       Qwen-2.5-VL-7B (K=15)   Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT    A-Mem    MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                   F1           0.2376       0.1511    0.6123      0.2765      0.2391     0.5928    0.6258    0.6382     0.2150     0.6674      0.6626      0.6482    0.6182
                                 BLEU-1         0.1865       0.1071    0.5273      0.2124      0.1903     0.5098    0.5321    0.5462     0.1626     0.5808      0.5678      0.5698    0.5319
                        FR        EM            0.0913       0.0502    0.3151      0.0913      0.0913     0.3288    0.3105    0.3516     0.0685     0.3744      0.3379      0.3699    0.3288
                                LLM-Judge       0.2626       0.1416    0.8082      0.3379      0.2626     0.8539    0.8037    0.8447     0.2260     0.8767      0.8721      0.8151    0.8174
Extract. & Adapt.




                                   F1           0.1992       0.0794    0.7812      0.3425      0.1954     0.6239    0.7905    0.8022     0.1658     0.8725      0.8452      0.8673    0.8607
                                 BLEU-1         0.1873       0.0680    0.7359      0.3032      0.1840     0.5834    0.7160    0.7337     0.1473     0.8362      0.8162      0.8295    0.8084
                        VS        EM            0.1601       0.0425    0.5752      0.2320      0.1569     0.4118    0.5915    0.6111     0.1078     0.6536      0.6471      0.6961    0.6569
                                LLM-Judge       0.1961       0.0719    0.7696      0.3366      0.1895     0.5964    0.7484    0.7958     0.1683     0.8742      0.8448      0.8676    0.8497
                                   F1           0.4500       0.3410    0.6410      0.4981      0.4486     0.2924    0.5900    0.5302     0.4147     0.8071      0.7517      0.7782     0.7975
                                 BLEU-1         0.3799       0.2838    0.5694      0.4283      0.3798     0.2295    0.5096    0.4502     0.3477     0.7307      0.6785      0.7128     0.7291
                        TTL       EM            0.2374       0.1662    0.4540      0.2908      0.2374     0.1009    0.3769    0.3264     0.2107     0.5964      0.5697      0.5994     0.6172
                                LLM-Judge       0.7092       0.6810    0.8279      0.7611      0.7033     0.7092    0.7953    0.7582     0.7107     0.9050      0.8591      0.9050     0.9065
                                   F1           0.2545       0.1930    0.4864      0.3335      0.2553     0.5661    0.5760    0.5614     0.2294     0.5798      0.5759      0.5752    0.5882
                                 BLEU-1         0.2363       0.1684    0.4539      0.3122      0.2363     0.5326    0.5485    0.5323     0.2065     0.5482      0.5413      0.5468    0.5602
                        TR        EM            0.1545       0.0976    0.3252      0.2276      0.1545     0.3496    0.4309    0.3902     0.1382     0.4228      0.4065      0.4472    0.4472
                                LLM-Judge       0.2805       0.1870    0.6707      0.3943      0.2764     0.8008    0.7236    0.7358     0.2480     0.7724      0.8008      0.7358    0.7561
                                   F1           0.2552       0.1155    0.3303      0.2119      0.2594     0.4593    0.4458    0.4676     0.2015     0.4678      0.4913      0.4697    0.3860
Reasoning




                                 BLEU-1         0.2442       0.0998    0.3144      0.2005      0.2480     0.4459    0.4335    0.4533     0.1912     0.4474      0.4712      0.4494    0.3713
                        VR        EM            0.2011       0.0690    0.2299      0.1379      0.2011     0.3851    0.3563    0.3678     0.1609     0.3563      0.3736      0.3621    0.2874
                                LLM-Judge       0.3046       0.1494    0.4368      0.2644      0.3046     0.6149    0.5632    0.5862     0.2586     0.5690      0.5833      0.5632    0.4799
                                   F1           0.2411       0.1747    0.4549      0.2471      0.2428     0.4367    0.4723    0.4547     0.2101     0.5042      0.5004      0.4841    0.4839
                                 BLEU-1         0.1739       0.1225    0.3372      0.1813      0.1770     0.3347    0.3698    0.3519     0.1429     0.3910      0.3840      0.3760    0.3748
                        MR        EM            0.0340       0.0146    0.0777      0.0243      0.0340     0.0631    0.0825    0.0825     0.0194     0.0922      0.0971      0.0874    0.0874
                                LLM-Judge       0.2985       0.1893    0.8058      0.3811      0.3058     0.8204    0.7913    0.8301     0.2791     0.8568      0.8665      0.8010    0.8252
                                   F1           0.2354       0.1685    0.3740      0.2752      0.2354     0.4292    0.4470    0.4888     0.2181     0.4701      0.4629      0.4242    0.4175
                                 BLEU-1         0.2005       0.1290    0.3203      0.2338      0.2000     0.3741    0.3935    0.4345     0.1883     0.4166      0.4052      0.3738    0.3508
                        KR        EM            0.1235       0.0617    0.1605      0.1235      0.1235     0.2099    0.2222    0.2593     0.1235     0.2469      0.2222      0.2222    0.1975
Knowledge Management




                                LLM-Judge       0.3395       0.2469    0.6296      0.3704      0.3395     0.7407    0.6605    0.7654     0.2840     0.7963      0.7716      0.7284    0.7099
                                   F1           0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3210    0.3580     0.3580     0.3704      0.3704      0.3457    0.3210
                                 BLEU-1         0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3210    0.3580     0.3580     0.3704      0.3704      0.3457    0.3210
                        CD        EM            0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3210    0.3580     0.3580     0.3704      0.3704      0.3457    0.3210
                                LLM-Judge       0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3210    0.3580     0.3580     0.3704      0.3704      0.3457    0.3210
                                   F1            0.9958      0.9958    0.9513      0.9843      0.9958     0.9849    0.9224    0.9842     0.9946     0.9513      0.9523      0.9635    0.9464
                                 BLEU-1          0.9953      0.9953    0.9512      0.9840      0.9953     0.9844    0.9201    0.9840     0.9946     0.9512      0.9518      0.9629    0.9461
                        AR        EM             0.9946      0.9946    0.9511      0.9837      0.9946     0.9837    0.9185    0.9837     0.9946     0.9511      0.9511      0.9620    0.9457
                                LLM-Judge        0.9783      0.9783    0.9375      0.9674      0.9783     0.9674    0.9429    0.9674     0.9837     0.9511      0.9348      0.9457    0.9375
                                   F1           0.3625       0.2794    0.6040      0.4035      0.3619     0.5282    0.6168    0.6182     0.3354     0.6925      0.6775      0.6788    0.6667
                                 BLEU-1         0.3279       0.2489    0.5503      0.3619      0.3279     0.4792    0.5574    0.5599     0.2999     0.6393      0.6244      0.6296    0.6129
         Overall                  EM            0.2519       0.1894    0.4150      0.2700      0.2507     0.3402    0.4214    0.4290     0.2279     0.4915      0.4804      0.4997    0.4793
                                LLM-Judge       0.4331       0.3504    0.7408      0.4906      0.4307     0.7306    0.7458    0.7694     0.4129     0.8209      0.8077      0.7969    0.7873




personally identifiable information or demographic                                                             ily targets English-language multimodal conversa-
attributes were collected or recorded during the an-                                                           tions across diverse daily-life and domain-specific
notation process. The annotation workflow did not                                                              scenarios, as illustrated in Appendix A.2. No de-
involve tracking individual annotators, comparing                                                              mographic attributes of real individuals are repre-
annotator performance, or modeling human judg-                                                                 sented or analyzed. The benchmark and dataset
ment patterns. All annotation results were used                                                                will be released publicly with clear documentation,
only to improve data quality and benchmark reli-                                                               permitting use for research purposes.
ability, with the research focus remaining only on
the properties of the data, models, and evaluation
framework rather than on any human behavior. All                                                               A.9       Potential Risks
annotators were fully informed that the annotated
data would be released publicly for research pur-
                                                                                                               This benchmark is designed for research and eval-
poses. Since all annotation and verification work
                                                                                                               uation purposes only. While it does not involve
was conducted by the author team as part of the re-
                                                                                                               real user data or personal information, potential
search process, no external recruitment or monetary
                                                                                                               risks may arise from unintended misuse, such as
compensation was involved. The benchmark study
                                                                                                               over-interpreting benchmark results as indicators
does not involve any privacy data of real users or
                                                                                                               of real-world deployment readiness. In addition,
human-subject research.
                                                                                                               models evaluated on this benchmark may inherit
                                                                                                               biases present in underlying pretrained language
A.8.3                          Scientific Artifacts
                                                                                                               model backbones (Schramowski et al., 2022). We
All data sources and models underlying the bench-                                                              mitigate these risks by clearly scoping the bench-
mark are used with explicit references and official                                                            mark to controlled memory research settings and
links. Detailed information can be found in Ap-                                                                by providing transparent documentation of data
pendix A.2.2, A.4, and A.5. The benchmark primar-                                                              sources, evaluation protocols, and limitations.
Table 13: Benchmark performance results on Qwen-2.5-VL-7B with the retrieval number K=20. The best and
second-performed memory model(s) are highlighted with orange and blue backgrounds.

                       Qwen-2.5-VL-7B (K=20)   Full (Text)   FIFO     NaiveRAG   Gen. Agent   Reflexion   MemGPT    A-Mem    MemoryOS   Full (MM)   MuRAG    UniversalRAG   NGM      AUGUSTUS
                                   F1           0.2376       0.1743    0.6303      0.3020      0.2391     0.5928    0.6145    0.6265     0.2150     0.6596      0.6609      0.6589    0.6216
                                 BLEU-1         0.1865       0.1295    0.5438      0.2366      0.1903     0.5098    0.5210    0.5396     0.1626     0.5656      0.5639      0.5813    0.5335
                        FR        EM            0.0913       0.0548    0.3242      0.1142      0.0913     0.3288    0.2922    0.3379     0.0685     0.3516      0.3425      0.3790    0.3379
                                LLM-Judge       0.2626       0.1598    0.8174      0.3744      0.2626     0.8539    0.8174    0.8288     0.2260     0.8950      0.8790      0.8402    0.8128
Extract. & Adapt.




                                   F1           0.1992       0.0953    0.8144      0.3567      0.1954     0.6239    0.7782    0.7963     0.1658     0.8676      0.8444      0.8583    0.8647
                                 BLEU-1         0.1873       0.0849    0.7704      0.3187      0.1840     0.5834    0.7026    0.7300     0.1473     0.8341      0.8160      0.8198    0.8230
                        VS        EM            0.1601       0.0588    0.5980      0.2386      0.1569     0.4118    0.5882    0.6078     0.1078     0.6503      0.6503      0.6765    0.6569
                                LLM-Judge       0.1961       0.0850    0.8023      0.3546      0.1895     0.5964    0.7337    0.7925     0.1683     0.8856      0.8611      0.8660    0.8611
                                   F1           0.4500       0.3789    0.6104      0.4900      0.4486     0.2924    0.5542    0.5353     0.4147     0.7841      0.7211      0.7911     0.7859
                                 BLEU-1         0.3799       0.3180    0.5355      0.4137      0.3798     0.2295    0.4735    0.4566     0.3477     0.7122      0.6478      0.7250     0.7184
                        TTL       EM            0.2374       0.1899    0.4243      0.2849      0.2374     0.1009    0.3561    0.3442     0.2107     0.5905      0.5312      0.6142     0.6172
                                LLM-Judge       0.7092       0.6929    0.8131      0.7448      0.7033     0.7092    0.7611    0.7760     0.7107     0.8961      0.8694      0.8976     0.9021
                                   F1           0.2545       0.1858    0.5154      0.3484      0.2553     0.5661    0.5502    0.5941     0.2294     0.5642      0.5613      0.5445    0.5890
                                 BLEU-1         0.2363       0.1652    0.4848      0.3255      0.2363     0.5326    0.5220    0.5679     0.2065     0.5345      0.5291      0.5180    0.5615
                        TR        EM            0.1545       0.0976    0.3577      0.2358      0.1545     0.3496    0.3821    0.4472     0.1382     0.3984      0.3984      0.4146    0.4390
                                LLM-Judge       0.2805       0.1870    0.7073      0.4268      0.2764     0.8008    0.7073    0.7398     0.2480     0.7967      0.7846      0.7073    0.7642
                                   F1           0.2552       0.1383    0.3654      0.2171      0.2594     0.4593    0.4216    0.4410     0.2015     0.5320      0.4344      0.4772    0.3695
Reasoning




                                 BLEU-1         0.2442       0.1208    0.3507      0.2016      0.2480     0.4459    0.4096    0.4304     0.1912     0.5130      0.4192      0.4563    0.3550
                        VR        EM            0.2011       0.0862    0.2644      0.1264      0.2011     0.3851    0.3391    0.3621     0.1609     0.4138      0.3333      0.3678    0.2644
                                LLM-Judge       0.3046       0.1954    0.4770      0.2672      0.3046     0.6149    0.5517    0.5891     0.2586     0.6322      0.5460      0.5718    0.4483
                                   F1           0.2411       0.1837    0.4678      0.2616      0.2428     0.4367    0.4732    0.4554     0.2101     0.4988      0.4955      0.4802    0.4742
                                 BLEU-1         0.1739       0.1271    0.3523      0.1932      0.1770     0.3347    0.3656    0.3550     0.1429     0.3863      0.3757      0.3664    0.3683
                        MR        EM            0.0340       0.0194    0.0922      0.0291      0.0340     0.0631    0.0874    0.0777     0.0194     0.0874      0.0874      0.0825    0.0971
                                LLM-Judge       0.2985       0.2039    0.8107      0.4029      0.3058     0.8204    0.7913    0.8374     0.2791     0.8786      0.8689      0.8107    0.8155
                                   F1           0.2354       0.1924    0.4200      0.2909      0.2354     0.4292    0.4290    0.4597     0.2181     0.4648      0.4604      0.4354    0.3967
                                 BLEU-1         0.2005       0.1546    0.3666      0.2530      0.2000     0.3741    0.3657    0.4083     0.1883     0.4060      0.3969      0.3858    0.3343
                        KR        EM            0.1235       0.0741    0.2099      0.1481      0.1235     0.2099    0.1852    0.2222     0.1235     0.2346      0.1975      0.2593    0.1728
Knowledge Management




                                LLM-Judge       0.3395       0.2593    0.7037      0.3951      0.3395     0.7407    0.7099    0.7531     0.2840     0.8025      0.7778      0.7593    0.6975
                                   F1           0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3333    0.3580     0.3580     0.3704      0.3457      0.3580    0.3333
                                 BLEU-1         0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3333    0.3580     0.3580     0.3704      0.3457      0.3580    0.3333
                        CD        EM            0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3333    0.3580     0.3580     0.3704      0.3457      0.3580    0.3333
                                LLM-Judge       0.3457       0.3580    0.3457      0.3086      0.3333      0.3580   0.3333    0.3580     0.3580     0.3704      0.3457      0.3580    0.3333
                                   F1            0.9958      0.9956    0.9514      0.9840      0.9958     0.9849    0.9233    0.9677     0.9946     0.9307      0.9322      0.9471    0.9519
                                 BLEU-1          0.9953      0.9952    0.9512      0.9838      0.9953     0.9844    0.9210    0.9676     0.9946     0.9297      0.9306      0.9466    0.9515
                        AR        EM             0.9946      0.9946    0.9511      0.9837      0.9946     0.9837    0.9185    0.9674     0.9946     0.9293      0.9293      0.9457    0.9511
                                LLM-Judge        0.9783      0.9783    0.9348      0.9674      0.9783     0.9674    0.9457    0.9565     0.9837     0.9266      0.9239      0.9348    0.9429
                                   F1           0.3625       0.2966    0.6157      0.4118      0.3619     0.5282    0.6018    0.6132     0.3354     0.6884      0.6602      0.6785    0.6629
                                 BLEU-1         0.3279       0.2651    0.5618      0.3683      0.3279     0.4792    0.5409    0.5572     0.2999     0.6357      0.6069      0.6286    0.6117
         Overall                  EM            0.2519       0.2005    0.4243      0.2741      0.2507     0.3402    0.4085    0.4296     0.2279     0.4874      0.4635      0.4985    0.4787
                                LLM-Judge       0.4331       0.3644    0.7554      0.5018      0.4307     0.7306    0.7390    0.7700     0.4129     0.8320      0.8068      0.7992    0.7846




A.10                          Use of AI Assistants
LLMs are used in this work strictly as auxiliary
tools for data construction, an evaluation metric,
and limited language polishing. During dataset
construction, LLMs assist in drafting candidate
multi-session conversational texts under predefined
structural constraints and scenario specifications.
These drafts are subsequently reviewed, edited, and
refined by annotators, who also insert appropriate
images and ensure that multimodal dependencies
are genuine, necessary, and non-trivial for down-
stream tasks. LLMs are further used to generate
part of the candidate question–answer pairs and
to perform preliminary checks for coherence, clar-
ity, and factual consistency. All QA pairs are then
manually verified and revised by annotators. In
addition, LLMs are used in a limited manner for
language polishing of the manuscript, including
improving fluency and presentation. In our eval-
uation, we also adopt LLM-as-a-Judge as one of
the evaluation metrics, following common practice
in prior work (Li et al., 2025). All technical con-
tent, experimental analysis, and scientific claims
are authored and finalized by the authors.
Table 14: Retrieval evaluation results on Qwen-2.5-VL-7B with different retrieval number K. Since answer refusal
(AR) questions intentionally introduce incorrect information and contain no supporting evidence in the conversation,
evaluation metrics for this category are not supported.

                                      Recall@K                            Precision@K                           Hit@K
          Qwen-2.5-VL-7B      5       10     15         20        5        10       15      20        5       10     15         20
             FIFO           0.0116   0.0381   0.0685   0.0950   0.0043   0.0087   0.0123   0.0129   0.0196   0.0491   0.0792   0.1081
             Gen. Agent     0.1707   0.2385   0.2854   0.3153   0.0955   0.0684   0.0544   0.0453   0.2718   0.3491   0.3962   0.4257
             NaiveRAG       0.5381   0.6723   0.7420   0.7877   0.2694   0.1757   0.1315   0.1047   0.7236   0.8179   0.8605   0.8900
   Overall   MuRAG          0.7506   0.8601   0.8990   0.9228   0.3686   0.2220   0.1572   0.1220   0.8861   0.9384   0.9548   0.9666
             UniversalRAG   0.7311   0.8411   0.8781   0.8998   0.3691   0.2206   0.1555   0.1204   0.8697   0.9227   0.9404   0.9496
             NGM            0.6192   0.7475   0.7892   0.8065   0.3564   0.3457   0.3450   0.3424   0.7708   0.8612   0.8874   0.9037
             AUGUSTUS       0.6729   0.7529   0.7785   0.7860   0.3341   0.2488   0.2213   0.2124   0.8553   0.8893   0.8959   0.8978
             FIFO           0.0006   0.0354   0.0514   0.0793   0.0009   0.0059   0.0064   0.0075   0.0046   0.0365   0.0548   0.0822
             Gen. Agent     0.1027   0.1363   0.1708   0.2062   0.0320   0.0237   0.0225   0.0215   0.1370   0.1872   0.2420   0.2922
             NaiveRAG       0.4802   0.6297   0.7379   0.7811   0.2082   0.1511   0.1239   0.1014   0.6849   0.8265   0.8813   0.9041
     FR      MuRAG          0.6248   0.7712   0.8228   0.8647   0.2630   0.1840   0.1397   0.1146   0.7991   0.9041   0.9178   0.9406
             UniversalRAG   0.6387   0.7940   0.8328   0.8768   0.2676   0.1913   0.1431   0.1180   0.8265   0.9178   0.9269   0.9498
             NGM            0.5025   0.6557   0.6999   0.7102   0.2348   0.2311   0.2329   0.2289   0.6712   0.8037   0.8265   0.8447
             AUGUSTUS       0.5766   0.6626   0.6771   0.6862   0.2428   0.2025   0.1779   0.1748   0.7717   0.8174   0.8219   0.8402
             FIFO           0.0172   0.0485   0.0678   0.0842   0.0059   0.0082   0.0085   0.0074   0.0261   0.0556   0.0719   0.0882
             Gen. Agent     0.2173   0.2801   0.3309   0.3505   0.0706   0.0467   0.0379   0.0312   0.2843   0.3660   0.4183   0.4444
             NaiveRAG       0.6723   0.7979   0.8300   0.8724   0.2366   0.1474   0.1037   0.0815   0.7712   0.8693   0.8856   0.9216
     VS      MuRAG          0.9080   0.9534   0.9660   0.9774   0.3379   0.1827   0.1237   0.0935   0.9444   0.9673   0.9739   0.9869
             UniversalRAG   0.8906   0.9438   0.9570   0.9651   0.3301   0.1801   0.1218   0.0918   0.9444   0.9739   0.9837   0.9902
             NGM            0.7889   0.8799   0.8982   0.9217   0.3445   0.3204   0.3134   0.3058   0.8562   0.9183   0.9346   0.9542
             AUGUSTUS       0.8363   0.8872   0.9056   0.9081   0.3124   0.2284   0.2062   0.1960   0.9248   0.9510   0.9542   0.9510
             FIFO           0.0026   0.0263   0.0938   0.1358   0.0042   0.0128   0.0265   0.0288   0.0208   0.0504   0.1246   0.1721
             Gen. Agent     0.3152   0.4425   0.5036   0.5488   0.2570   0.1831   0.1375   0.1119   0.5460   0.6528   0.7003   0.7240
             NaiveRAG       0.6837   0.8412   0.9017   0.9234   0.5501   0.3436   0.2457   0.1884   0.9496   0.9703   0.9881   0.9911
    TTL      MuRAG          0.8523   0.9566   0.9760   0.9785   0.6754   0.3932   0.2671   0.2007   0.9822   0.9852   0.9941   0.9970
             UniversalRAG   0.8617   0.9583   0.9682   0.9787   0.6967   0.3970   0.2669   0.2018   0.9763   0.9822   0.9881   0.9941
             NGM            0.7617   0.8919   0.9194   0.9203   0.7029   0.6919   0.6934   0.6930   0.9407   0.9763   0.9822   0.9822
             AUGUSTUS       0.7231   0.8508   0.8903   0.9088   0.5569   0.3690   0.3094   0.2914   0.9733   0.9822   0.9881   0.9911
             FIFO           0.0081   0.0122   0.0325   0.0569   0.0033   0.0024   0.0033   0.0037   0.0081   0.0163   0.0325   0.0569
             Gen. Agent     0.0447   0.1043   0.1301   0.1558   0.0130   0.0154   0.0130   0.0122   0.0650   0.1463   0.1707   0.2033
             NaiveRAG       0.3828   0.5000   0.5413   0.6009   0.1301   0.0870   0.0640   0.0524   0.5610   0.6585   0.6829   0.7317
     TR      MuRAG          0.6640   0.7724   0.8266   0.8557   0.2163   0.1285   0.0927   0.0732   0.8455   0.8943   0.9187   0.9187
             UniversalRAG   0.6938   0.8089   0.8652   0.8733   0.2293   0.1341   0.0970   0.0736   0.8618   0.9106   0.9350   0.9431
             NGM            0.4898   0.5928   0.6179   0.6321   0.1846   0.1688   0.1626   0.1609   0.6504   0.7561   0.7805   0.8049
             AUGUSTUS       0.6497   0.7243   0.7595   0.7683   0.2280   0.1743   0.1601   0.1509   0.8455   0.8780   0.8780   0.8780
             FIFO           0.0000   0.0235   0.0551   0.0893   0.0000   0.0075   0.0126   0.0155   0.0000   0.0287   0.0632   0.1092
             Gen. Agent     0.1439   0.1888   0.2521   0.2817   0.0885   0.0621   0.0544   0.0448   0.2874   0.3391   0.4138   0.4310
             NaiveRAG       0.3093   0.4774   0.6010   0.6632   0.1540   0.1264   0.1115   0.0948   0.5057   0.6782   0.7759   0.8333
     VR      MuRAG          0.6158   0.7774   0.8551   0.8916   0.3299   0.2236   0.1701   0.1353   0.8276   0.9080   0.9483   0.9598
             UniversalRAG   0.5497   0.6978   0.7560   0.7979   0.3356   0.2149   0.1586   0.1259   0.7759   0.8333   0.8621   0.8851
             NGM            0.4803   0.6351   0.7183   0.7541   0.2603   0.2707   0.2761   0.2762   0.6839   0.7989   0.8506   0.8793
             AUGUSTUS       0.5110   0.5722   0.5796   0.5924   0.3430   0.3139   0.3032   0.3045   0.7644   0.8103   0.8103   0.8218
             FIFO           0.0178   0.0591   0.0736   0.0914   0.0058   0.0117   0.0097   0.0085   0.0243   0.0728   0.0777   0.0971
             Gen. Agent     0.1219   0.1752   0.2122   0.2376   0.0505   0.0359   0.0298   0.0252   0.2039   0.2621   0.2913   0.3204
             NaiveRAG       0.5659   0.6600   0.7306   0.7793   0.2010   0.1257   0.0958   0.0784   0.7379   0.8252   0.8738   0.8981
     MR      MuRAG          0.7626   0.8527   0.9034   0.9237   0.2641   0.1597   0.1175   0.0925   0.9175   0.9612   0.9806   0.9806
             UniversalRAG   0.7506   0.8590   0.9108   0.9364   0.2612   0.1631   0.1204   0.0949   0.8981   0.9563   0.9806   0.9854
             NGM            0.6023   0.7464   0.7927   0.8133   0.2551   0.2415   0.2401   0.2394   0.7573   0.8835   0.9223   0.9369
             AUGUSTUS       0.7162   0.7748   0.8048   0.8014   0.2596   0.1964   0.1783   0.1684   0.8786   0.9175   0.9320   0.9223
             FIFO           0.0412   0.0607   0.0885   0.1091   0.0099   0.0086   0.0099   0.0099   0.0494   0.0864   0.1111   0.1235
             Gen. Agent     0.0782   0.1091   0.1214   0.1389   0.0247   0.0185   0.0148   0.0136   0.1111   0.1728   0.1852   0.2346
             NaiveRAG       0.3899   0.5300   0.5808   0.6529   0.1605   0.1111   0.0823   0.0685   0.5679   0.7160   0.7531   0.8025
     KR      MuRAG          0.5752   0.7495   0.8015   0.8424   0.2272   0.1506   0.1119   0.0895   0.7531   0.8889   0.9012   0.9259
             UniversalRAG   0.4750   0.6453   0.7433   0.7841   0.1852   0.1259   0.1029   0.0827   0.6790   0.8272   0.8889   0.8889
             NGM            0.4141   0.5599   0.6626   0.6863   0.1934   0.1841   0.1914   0.1907   0.6049   0.7037   0.7778   0.8025
             AUGUSTUS       0.4483   0.5523   0.5860   0.5970   0.1895   0.1480   0.1324   0.1287   0.6420   0.7160   0.7284   0.7284
             FIFO           0.0432   0.0494   0.0617   0.0741   0.0099   0.0062   0.0049   0.0043   0.0494   0.0494   0.0617   0.0741
             Gen. Agent     0.0432   0.1091   0.1728   0.1934   0.0148   0.0198   0.0230   0.0204   0.0617   0.1852   0.2469   0.2593
             NaiveRAG       0.3868   0.4650   0.5539   0.6284   0.1333   0.0827   0.0667   0.0574   0.5432   0.5926   0.7037   0.7407
     CD      MuRAG          0.6383   0.7868   0.8218   0.8881   0.2148   0.1296   0.0897   0.0759   0.7407   0.8519   0.8765   0.9259
             UniversalRAG   0.4877   0.6008   0.6605   0.6687   0.1704   0.1037   0.0765   0.0586   0.5926   0.7160   0.7407   0.7284
             NGM            0.4428   0.5621   0.6074   0.6383   0.1774   0.1673   0.1642   0.1596   0.5802   0.7160   0.7407   0.7654
             AUGUSTUS       0.6049   0.6584   0.6893   0.6770   0.2123   0.1578   0.1446   0.1385   0.6914   0.7531   0.7778   0.7654
                               Relevant clue snippets in the conversation history

         Great! Let me start with the Cap-Sleeved Midi Dresses. Here’s the first one.


Lin
         Oh wow, the moment I opened this one, the shine caught my eye immediately. Let me
         take in the details.
                                                                                                         Assistant

        What do you think of this dress?


Lin      This dress has such a striking metallic sheen—like it was made for evening lights. The
         bronze tones and scalloped stripes feel glamorous but still structured. It’s definitely a
         confidence-boosting piece.
                                                                                                         Assistant


         Here's the second dress.


Lin
         Aww, this one immediately gives off a softer energy. The floral pattern feels calming in
         contrast to the first one.
                                                                                                         Assistant
               More conversation on Cap-Sleeved Midi Dresses with more visual examples…




                According to the conversation, what category is the clothing in the picture?



        Insufficient visual                      Insufficient visual
                                                                                         Insufficient visual
       & adaptation ability.                    & adaptation ability.
                                                                                        & adaptation ability.
                                            The clothing in the picture is
             Midi Dress.                                                                        dress.
                                            a fitted, knee-length dress.


                      NaiveRAG                                 A-Mem                              MemoryOS


                                                       Insufficient
                                                     adaptation ability.


      Cap-Sleeved Midi Dresses.                            dress.                 Cap-Sleeved Midi Dress.


                        MuRAG                               UniversalRAG                          AUGUSTUS



                           Figure 16: Case study with the test-time learning example.
                              Relevant clue snippets in the conversation history

          Not yet! I want something that fits her personality — gentle but lively. I’ve been asking
          my friends for ideas too. My friend Amy has a Cairn Terrier, and she gave me some tips.

Lena
          Oh, Cairn Terriers! They’re quite different from Maltese dogs: more energetic and
          adventurous. Amy’s must keep her busy! What did she say about naming pets?
                                                                                                       Assistant

          Amy said her dog is super active and always digging around in the yard. Apparently, that’s
          part of their nature. They were bred to hunt small animals in rocky areas!


Lena      Exactly! Cairn Terriers are brave and curious little dogs. They love to explore and have
          tons of energy. They can be stubborn at times, but that’s just because they’re so
          independent. Did Amy mention if it’s hard to train her dog?
                                                        …                                              Assistant
                             Othe dialogs about Lucy’s and Lena’s classmate’s dog…



   Is the dog in the picture the same breed as the pet of any friend or classmate mentioned by Lena?
    Answer Yes or No. If yes, then please specify which person's pet it shares the same breed with.


                    Insufficient                             Insufficient                        Insufficient
                     retrieval.                               retrieval.                          retrieval.
              No.                                      No.                                 No.

                NaiveRAG                                  A-Mem                             MemoryOS


                                               Correct retrieved clues,
                                               but incorrect reasoning.                          Insufficient
                                                                                                  retrieval.
       Yes, the dog in the              No. The dog in the picture is a
                                          Cairn Terrier, while Lena's
                                                                                           No.
       picture is the same
         breed as Amy’s.               friend Amy has a Cairn Terrier.
                                         The dog in the picture is not                      AUGUSTUS
                                        the same breed as Amy's pet.
                 MuRAG

                                                       UniversalRAG


                       Figure 17: Case study with the visual-centric reasoning example.
                                 Relevant clue snippets in the conversation history

         My sister actually loves Standard Poodles. She says their elegance and athletic build
         make them her dream dog.

Lena
         That totally makes sense! People who appreciate both intelligence and style often adore
         Standard Poodles. They’re such graceful companions and do so well in active families.
                                                                                                           Assistant

         I think I’d love spending time with a Miniature Poodle. They seem like the perfect
         balance: smart, active, but still small enough for apartment life.

         You’re absolutely right, Lena. Miniature Poodles often fit well into both active and cozy
Lena
         lifestyles. They enjoy long walks and puzzles but also love curling up beside their owners.
         Given your interest in animal behavior, you’d probably find their problem-solving
         instincts fascinating.
                                                                                                           Assistant
                                                  Other dialogs…

         Oh wait, I just realized I got it wrong earlier. My sister doesn’t actually like Standard
         Poodles. Her favorite breed is the Standard Schnauzer, as shown in the image!

Lena     Ah, that explains it! Standard Schnauzers have a totally different charm — confident,
         spirited, and with those adorable beards. They’re smart and loyal too, just in their own
         unique way.
                                                         …                                                 Assistant



                            Is the dog in the picture the breed that Lena's sister likes?


               Insufficient                                                                   Insufficient
               visual ability.                                                                visual ability.
                                                   Insufficient
   No, the dog in the picture is                   visual ability.           No, the dog in the picture is not the
   not the breed that Lena's                                                 breed that Lena's sister likes.
   sister likes. Lena's sister                                               Lena's sister prefers Standard
   prefers Standard Schnauzers.                          No.                 Schnauzers over Standard Poodles.


                        NaiveRAG                             A-Mem                                    MemoryOS

             Lack knowledge                       Lack knowledge                             Lack knowledge
             update ability.                      update ability.                            update ability.
 No, the dog in the picture is not
 the breed that Lena's sister                            No.                                         No.
 likes. Lena's sister actually loves
 Standard Poodles.
                                                        UniversalRAG                                  AUGUSTUS
                         MuRAG



                          Figure 18: Case study with the knowledge resolution example.
