covering both pipeline construction and benchmarking. Open-weight LVLMs are served locally
with vLLM v0.17–0.18 on 8×A100-80GB nodes with tensor parallelism for context windows at
and above 128K. Per-model decoding budgets, judge protocol, retrieval depth for memory agents,
and adapter details for the four text-only pipelines (Mem0, MemOS, MemAgent-7B, Memory-
T1) are documented in Appendix B. The four-stage construction pipeline (multimodal session
simulation, question construction, evidence session construction, conversation history assembly)
is described in Section 3, with per-subtype generation routes in Appendix C.4 and full prompt
templates in Appendix F. Quality control combines automated filtering with a three-round human
review (Section 3.3, Appendix E.1) and an LLM-as-Judge protocol cross-validated by an independent
judge family and a three-annotator human consensus; agreement statistics and per-question-type
breakdowns appear in Appendix E.2. The 789-question benchmark, 4,695 source images with
provenance metadata, evaluation harness, and prompt templates are publicly released under frozen
version tags at https://huggingface.co/datasets/xiyuRenBill/MEMLENS (dataset) and
https://github.com/xrenaf/MEMLENS (code) so that every reported leaderboard cell can be
reproduced end-to-end.

References
  [1] Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complex-
      ity. Technical report, Technical report, Bytedance, 2025. URL https://lf3-static. bytednsdoc.
      com . . . .
  [2] OpenAI. Openai gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267.
  [3] OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774.
  [4] Anthropic. Claude 3 model card. https://assets.anthropic.com/m/61e7d27f8c8f5
      919/original/Claude-3-Model-Card.pdf, 2024. Accessed: 2026-04-30.
  [5] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett
      Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal
      understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
  [6] Bytedance Seed. Seed1. 8 model card: Towards generalized real-world agency. arXiv preprint
      arXiv:2603.20633, 2026.
  [7] Anthropic. Claude Sonnet 4.5 System Card. https://assets.anthropic.com/m/12f21
      4efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf, 2025.
  [8] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and
      Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL https:
      //arxiv.org/abs/2310.08560.
  [9] Deshraj Yadav, Taranjeet Singh, and Prashant Srivastava. Mem0: Building production-ready
      AI agents with scalable long-term memory, 2025. URL https://arxiv.org/abs/2504.1
      9413.
 [10] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag:
      Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983, 2024.
 [11] Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng,
      Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, and Mark Steedman. MMLong-
      Bench: Benchmarking long-context vision-language models effectively and thoroughly, 2025.
      URL https://arxiv.org/abs/2505.10610.
 [12] Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang
      Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai,
      Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024. URL https:
      //arxiv.org/abs/2406.07230.
 [13] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu
      Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin
      Cao, and Aixin Sun. Mmlongbench-doc: Benchmarking long-context document understanding
      with visualizations, 2024. URL https://arxiv.org/abs/2407.01523.
 [14] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long-
      memeval: Benchmarking chat assistants on long-term interactive memory, 2025. URL
      https://arxiv.org/abs/2410.10813.


                                               11
