5   Conclusion
We introduced M EM L ENS, the first benchmark for multimodal conversational memory to evaluate
LVLMs and memory-augmented agents under a unified, length-controlled protocol (32K–256K
tokens; five memory abilities). The task remains challenging: even the strongest LVLM reaches
only 58.68% at 32K. LVLMs degrade sharply once input contexts grow, while memory agents stay
length-stable but discard fine-grained visual evidence at storage time, and post-training on memory
agent backbones can erode the agents’ abstention behavior. The five abilities show low cross-type
correlation, confirming that per-type evaluation is necessary, and nearly 90% of information-extraction
errors stem from visual perception rather than comprehension, indicating that scaling first harms
evidence retrieval, not reasoning. Visual-evidence retention and retrieval, rather than raw scaling of
either context or memory, therefore emerges as the principal bottleneck to address in the future.

Acknowledgments and Disclosure of Funding
The authors of this paper were supported by the National Key Research and Development Program of
China (2025YFE0200500), the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong
Kong, SAR, China, and the AoE (AoE/E-601/24-N), the RIF (R6021-20) and the GRF (16205322)
from RGC of Hong Kong, SAR, China. We also thank the NVIDIA AI Technology Center (NVAITC)
and OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. for their support and additional
funding.

Ethics Statement
M EM L ENS is designed exclusively for evaluating multimodal long-term conversational memory
in large vision-language models (LVLMs) and memory-augmented agents under a unified, length-
controlled protocol. The benchmark is not intended as a training dataset for developing memory-
equipped systems but rather as a controlled diagnostic for fair comparison across architectures. We
release every artifact under frozen version tags so that any reported leaderboard cell remains traceable
to the exact item set on which it was scored.
The 4,695 source images underlying M EM L ENS are retrieved from public web image search via
iCrawler against a non-person-centric topic ontology; candidates carrying watermarks, stock-photo
logos, or copyright overlays are excluded at retrieval time, and the construction prompts never request
the model to identify, name, or describe any depicted person. Author-produced artifacts (questions,
evidence facts, prompts, and human-annotation records) are released under CC-BY-4.0 and the
evaluation harness under MIT, while third-party images retain their original source-site licenses;
per-image provenance metadata (source URL, retrieval timestamp, perceptual hash) accompanies the
release, and a takedown contact in the project repository allows seven-day removal of any flagged
image.
Human review is conducted by project members rather than crowd-workers, so no third-party data
collection or human-subjects protocol is involved. Three sequential rounds (Appendix E.1) audit
cross-modal necessity at the question level, naturalness and recoverability at the session level, and a
stratified sample of haystack sessions before any item reaches the released set.
We caution that M EM L ENS is not intended as a training dataset: exposure of the 789 evaluation items
as supervised data would compromise their diagnostic value, and the datasheet explicitly discourages
such use. While the benchmark is grounded in multi-session multimodal conversation, the five
memory abilities it isolates (information extraction, multi-session reasoning, temporal reasoning,
knowledge update, and answer refusal) describe general functional requirements of any long-horizon
conversational assistant. We therefore expect M EM L ENS to serve as a useful indicator of memory
robustness in adjacent settings, such as document-grounded or voice-grounded assistants, where
retrieval, integration, and calibrated abstention play comparable roles.

Reproducibility Statement
All closed-source LVLM evaluations and synthetic data construction were performed via the public
APIs of OpenAI, Anthropic, and Google, with a total experimental cost of approximately 4,500 USD


                                                  10
