Table 1: Left: Comparison of M EM L ENS with existing long-context and conversational memory
benchmarks. Type — Dial (Dialogue) or Doc (Document); Max L — maximum context length
reported (“—” if not applicable); L Control — standardized token length control; Comp. Eval. —
paper benchmarks both LVLMs and memory-augmented agents on the same data; Memory Tasks —
dot pattern showing coverage of the five evaluation tasks (• = supported, ◦ = not supported) in fixed
order: IE (Information Extraction), MSR (Multi-Session Reasoning), TR (Temporal Reasoning),
KU (Knowledge Update), AR (Answer Refusal). Right: Distribution of the 789 evaluation questions
in M EM L ENS across the five major task types (inner ring) and nine reporting subtypes (outer ring).

                              Max    Multi- Multi- L     Comp. Memory
Benchmark              Type                                                            Refuse               Entity
                               L     modal Sess. Control Eval.  Tasks                  11.4%                15.2%
MemoryBank [18]        Dial    —      ✗     ✓      ✗      ✗   •◦◦◦◦                     AR
LoCoMo [16]            Dial   10K     ✓     ✓      ✗      ✗   •••◦◦      Update        11.4%
                                                                         14.7%                                IE
MM-NIAH [12]           Doc    128K    ✓     ✗      ✓      ✗   •◦◦◦◦                KU                       31.2%
                                                                                  14.7%                              PrevInfo
                                                                                                                      16.0%
MMLongBench-Doc [13]   Doc    128K    ✓     ✗      ✗      ✗   •◦◦◦◦
MRAG-Bench [19]        Doc     —      ✓     ✗      ✗      ✗   •◦◦◦◦
                                                                         Count.
LongMemEval [14]       Dial   1.5M    ✗     ✓      ✓      ✗   •••••       5.8%      MSR
MMLongBench [11]       Doc    128K    ✓     ✗      ✓      ✗   •◦◦◦◦                18.1%              TR
                                                                           Ent. Res.                 24.6% Temp. Gnd.
Multimodal NIAH [20]   Doc    128K    ✓     ✗      ✓      ✗   •◦◦◦◦         6.0%
                                                                                   Arith.                        13.1%
MemAgentBench [15]     Dial   115K    ✗     ✓      ✗      ✓   •◦◦◦◦                 6.3%        Dur. Cmp.
                                                                                                 11.5%
M EM L ENS (Ours)      Dial   256K    ✓     ✓      ✓      ✓   •••••




Despite progress along both directions, no current benchmark conducts a length-controlled compari-
son of long-context LVLMs and memory-augmented agents on questions that genuinely require visual
evidence. Long-context multimodal benchmarks measure context-length scaling on long documents
and retrieval tasks [11–13], but cover mainly LVLMs and do not place them alongside memory-
augmented agents for direct comparison. Text-only conversational memory benchmarks such as
LongMemEval [14] and MemoryAgentBench [15] overlook the visual modality entirely, treating
memory as a single-modality problem. Multimodal conversational benchmarks such as LoCoMo [16]
and Mem-Gallery [17] retain both visual and text modalities, yet most of their questions admit a
text-only shortcut, rendering the visual modality redundant. As Table 1 summarizes, no existing
benchmark requires visual evidence to answer its questions while supporting a length-controlled
comparison of long-context LVLMs and memory-augmented agents.
To bridge the gap, we introduce M EM L ENS, a comprehensive benchmark for assessing memory in
multimodal multi-session conversations. M EM L ENS consists of 789 questions covering five core
memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge
update, and answer refusal. We design questions that demand cross-modal reasoning over both
visual and textual evidence, requiring the retrieval of multimodal information hidden within one
or more conversations between a user and an assistant. To verify that solving M EM L ENS requires
visual evidence, we conduct an image-ablation study on the 80.4% of questions whose evidence
includes images: when these images are removed, the accuracy of two frontier LVLMs collapses
below 2% (§3.4). Following the needle-in-a-haystack construction paradigm [21], we design a
four-stage pipeline that builds a coherent multi-session chat history for each question, distributing
the evidence across one or more user-assistant sessions alongside topically related distractor turns.
Because distractor turns can be added independently of the evidence, the chat length is freely
extensible. We release four standard context lengths (32K, 64K, 128K, and 256K tokens) under a
cross-modal token-counting scheme [11] that aligns text and vision tokens. M EM L ENS thus enables a
length-controlled comparison of long-context LVLMs and memory-augmented agents on multimodal
questions spanning five comprehensive memory abilities.
Using M EM L ENS, we evaluate 27 LVLMs and 7 memory-augmented agents across all four context
lengths (32K, 64K, 128K, and 256K). Our evaluation yields three key findings. First, the five
memory abilities are largely independent: strong information extraction does not predict multi-
session reasoning, and multi-session reasoning caps most evaluated systems below 30%. Second,
the two approaches exhibit complementary failure modes: long-context LVLMs deliver high short-
context accuracy through direct visual grounding, but this advantage shrinks as conversations grow;
memory agents, in contrast, are length-stable but lose visual fidelity under storage-time compression,
and memory-oriented post-training of agent backbones can additionally weaken their abstention
behavior. Third, neither approach comes close to solving the task of long-term memory. These


                                                    2
