results point to a clear next step: architectures that combine long-context attention with structured
multimodal retrieval, rather than scaling either component in isolation.

2     Related Work
Memory-Augmented LLM Agents. Recent surveys systematize memory representations and oper-
ations across LLM-based agents [22]. Text-only memory agents span structured-fact stores [18, 23],
OS-inspired paging [8], tree-summarized retrieval [24], neurobiological graphs [25], relational em-
beddings [9], agentic self-organizing notes [26], intent-driven memory selection [27], RL-selected
time-aware memory [28], layered memory tiers [29], and sliding-window RL agents [30]. Mul-
timodal extensions add ColPali-style document retrieval [31, 32], multimodal embeddings [33],
dual-layer semantic memory [34], entity-centric audio-visual memory [35], sparse video mem-
ory [36], LoRA-tuned session retrieval [37], and intent-guided multimodal response retrieval over
multi-session conversations [38]. Recent work probes the intersection of long-context LLMs and
retrieval-augmented generation [10, 39, 40], suggesting retrieval and long attention are complemen-
tary rather than competing.

Long-Context and Conversational Memory Benchmarks. Most current long-context benchmarks
established core protocols for retrieval and length scaling [41–47], but neglect the visual modality.
Multimodal extensions, such as MMLongBench [11], scale context length over documents [11, 13],
needle-style retrieval [12, 20], long-form video [48, 49], super-long documents [50, 51], and multi-
image inputs [52–55]. Yet their inputs are documents or videos rather than multi-session conver-
sations. Memory under conversational interaction therefore remains unexercised. Conversational
memory benchmarks restore the multi-session structure but rarely require visual perception: Long-
MemEval [14] (500 questions over five memory abilities, up to 1.5M tokens), PerLTQA [56],
and MemoryAgentBench [15] (retrieval, test-time learning, long-range understanding, selective
forgetting) discard images entirely, while LoCoMo [16] and Mem-Gallery [17] embed images in
persona-grounded dialogue but allow most questions to be answered from text alone, so neither
stress-tests multimodal memory.

3     The M EM L ENS Benchmark
We propose M EM L ENS, a multimodal long-term conversational memory benchmark that comprises
789 questions instantiated at four standardized input lengths (32K/64K/128K/256K tokens). In
contrast to prior benchmarks restricted to text-only conversations and questions, M EM L ENS provides
multimodal conversation sessions in which text and images are interleaved, together with questions
that require cross-modal reasoning over evidence images and the surrounding textual context. §3.1
defines the five memory abilities, §3.2 describes the construction pipeline (Figure 1), §3.3 presents
the quality control, and §3.4 describes the cross-modality validation.

3.1   Memory Abilities

To comprehensively reflect the retrieval, reasoning, and update that a conversational assistant must
perform over a long multimodal history, M EM L ENS formulates five core memory abilities. We break
each ability into subtypes that target specific reasoning operations.

       • Information Extraction (IE) tests the recall of a specific fact from a single evidence session,
         with two subtypes. Entity questions form a two-hop chain: the model first identifies an
         abstracted entity in the evidence image, then retrieves the associated information from the
         surrounding text. PrevInfo (previous information) questions instead abstract the session
         reference, asking the model to recall a visual detail from an image shared in an earlier
         session. The visual identification hop draws on image-understanding skills such as fine-
         grained recognition, object counting, spatial reasoning, and numerical reasoning.
       • Multi-Session Reasoning (MSR) evaluates aggregation across three to eight sessions: count-
         ing tallies unique items identified only by their evidence images, arithmetic sums values
         stated in text or embedded in visual artifacts, and entity resolution determines co-reference
         by comparing images across sessions.


                                                   3
