sphinx: archive MemLens (arXiv 2605.14906v1) multimodal long-term memory benchmark with 16 verified quotations

wizard-work/sphinx/README-archive-lane.md

@@ -1,6 +1,32 @@
 # sphinx — Archive lane work (multimodal, long-context research)
 
-## Latest: `archive-mm-prm-2026-10-22`
+## Latest: `archive-memlens-2026-10-22`
+
+Verifiable archive of arXiv 2605.14906v1 (*MemLens*), a multimodal long-term
+conversational-memory benchmark — 789 questions, five memory abilities, 32K–256K
+token lengths, evaluated on 27 LVLMs and 7 memory-augmented agents.
+
+Artifacts:
+- `archive-memlens-2026-10-22/memlens.pdf` — the paper, 14 pages (sha256 recorded)
+- `archive-memlens-2026-10-22/memlens.txt` + `page_01..14.txt` — full text and per-page text
+- `archive-memlens-2026-10-22/metadata.xml` — raw arXiv Atom API response
+- `archive-memlens-2026-10-22/quotes.json` — 16 anchor quotations + the page each was found on
+- `archive-memlens-2026-10-22/build_memlens_archive.py` — stdlib-only builder (re-fetch + re-extract)
+- `archive-memlens-2026-10-22/verify_memlens_archive.py` — offline verifier (exit 0 = correct)
+- `archive-memlens-2026-10-22/REPORT-memlens-archive-2026-10-22.md` — the research note
+
+Grounding test: every one of the 16 quotations in the note is matched verbatim
+(whitespace-normalised) against the extracted page text, and the page number cited must
+actually contain that quotation. 16/16 pass; 33 checks, 0 failures. Negative control:
+injecting one fabricated quotation makes the verifier exit 1 and print
+`QUOTE NOT IN PAPER TEXT`. Reproduction re-ran the builder from scratch and reproduced
+byte-identical digests (`b92a5b6d…` PDF, `628bd8d6…` text).
+
+Reproduce:
+    python3 build_memlens_archive.py <outdir>
+    python3 verify_memlens_archive.py   # exit 0 = correct
+
+## Previous: `archive-mm-prm-2026-10-22`
 
 Verifiable multimodal/long-context archive of arXiv 2505.13427 (*MM-PRM*),
 with a machine-checkable grounding test over an 18-chunk index.

wizard-work/sphinx/archive-memlens-2026-10-22/REPORT-memlens-archive-2026-10-22.md

@@ -0,0 +1,184 @@
+# REPORT — MemLens archive (arXiv 2605.14906v1)
+
+Lane: Archive (multimodal, long-context research)
+Date: 2026-10-22 (UTC) · Archivist: sphinx (Hermes Agent)
+
+---
+
+## 1. Identity (MEASURED — arXiv Atom API, saved as `metadata.xml`)
+
+| Field | Value |
+|---|---|
+| Title | MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models |
+| arXiv | 2605.14906v1 [cs.CV] |
+| Submitted | 2026-05-14 |
+| Comment field | "Work in progress" |
+| Authors | Xiyu Ren, Zhaowei Wang, Yiming Du, Zhongwei Xie, Chi Liu, Xinlin Yang, Haoyue Feng, Wenjun Pan, Tianshi Zheng, Baixuan Xu, Zhengnan Li, Yangqiu Song, Ginny Wong, Simon See (14) |
+| Code | https://github.com/xrenaf/MEMLENS |
+| Affiliations | HKUST CSE; CUHK; OmniMemory (Shenzhen); NVIDIA AI Technology Center |
+
+## 2. Provenance (MEASURED)
+
+| File | What | sha256 |
+|---|---|---|
+| `memlens.pdf` | Paper as published, 14 pages, PDF v1.7, 2,627,199 B | `b92a5b6d89999ba6ad77fbc32f8f6e9e86a39a49dfab0a8c9ee310edf380affd` |
+| `memlens.txt` | Full text, `pdftotext -layout`, 293,612 B | `628bd8d62beb83c57b0dfe7792d3c1b25ed480adf04e37a8e93aa235f44c2bdb` |
+| `page_01.txt` … `page_14.txt` | Per-page text (page-anchored citation) | — |
+| `metadata.xml` | Raw arXiv API response | — |
+| `quotes.json` | 16 quotations + the page each was found on | — |
+| `SHA256SUMS` | Digests for the two large files | — |
+
+**Reproduction (MEASURED):** `build_memlens_archive.py` was re-run from scratch against the
+live arXiv endpoints into a clean directory; it re-downloaded the PDF and re-extracted the text,
+and produced byte-identical digests (`b92a5b6d…` / `628bd8d6…`). The archive is not a
+hand-assembled copy — it rebuilds.
+
+**Every quotation in §7 was matched programmatically against the extracted page text
+(whitespace-normalised) before being written down: 16/16 found. The check re-runs offline — see §9.**
+
+---
+
+## 3. One-line verdict
+
+MemLens is a 789-question, length-controlled (32K–256K token) multimodal conversational-memory
+benchmark — the first to put long-context vision-language models and memory-augmented agents on
+the same questions — and its central measured result is that the task is nowhere near solved:
+the best LVLM reaches 58.68% at 32K, multi-session reasoning caps most systems below 30%, and
+nearly 90% of information-extraction errors come from visual perception failure rather than reasoning.
+
+## 4. What the paper contributes
+
+- **A benchmark, not a model.** It evaluates 27 LVLMs and 7 memory-augmented agents.
+- **Cross-modal necessity enforced by construction.** Evidence facts are abstracted in the text
+  (e.g. "Golden Gate Bridge" → "the bridge shown in <image>"), so only the image resolves the
+  reference; removing evidence images drops two frontier LVLMs below 2% accuracy.
+- **Length control.** Four standard lengths (32K/64K/128K/256K tokens) under the MMLongBench
+  cross-modal token-counting scheme, so LVLMs and agents can be compared at equal nominal length.
+- **Five memory abilities** — Information Extraction (IE), Multi-Session Reasoning (MSR),
+  Temporal Reasoning (TR), Knowledge Update (KU), Answer Refusal (AR) — across 9 subtypes.
+
+### Dataset statistics (Table 2, p.5 — MEASURED)
+
+| Statistic | Value |
+|---|---|
+| Questions | 789 |
+| Types / Subtypes | 5 / 9 |
+| Evidence sessions | 2,145 |
+| Avg. turns/session | ~10 |
+| Avg. images/session | ~1.5 |
+| Tokens/image | ~2,000 |
+| Sessions/instance | 14 → 93 |
+| Images/instance | 20 → 138 |
+
+The paper also states 4,695 unique source images are released (p.10), and that an initial pool of
+20k candidates was reduced to 789 by automated filters plus three rounds of human review (p.5).
+
+## 5. Cross-modality validation (p.6 — MEASURED)
+
+Image-necessity split: 65.7% image-essential, 14.7% image-supportive, 19.6% text-sufficient.
+
+Answerability test (n = 634, evidence supplied): GPT-5.4 reaches 93.13% overall, Gemini-3.1-Pro 89.42%.
+
+Ablation (evidence images removed): overall accuracy collapses below 2% for both models —
+GPT-5.4 to 1.74% (Δ −91.39), Gemini-3.1-Pro to 1.89% (Δ −87.53), per Table 3.
+
+## 6. Key findings
+
+### 6.1 Headline findings (p.2 — quoted in §7)
+
+1. The five memory abilities are largely independent; multi-session reasoning caps most systems below 30%.
+2. The two approaches fail in complementary ways: LVLMs win short context and degrade as it grows;
+   memory agents are length-stable but lose visual fidelity through storage-time compression.
+3. Neither approach comes close to solving long-term memory.
+
+### 6.2 Main results (p.7 — MEASURED numbers)
+
+- At 32K the top eight LVLMs fall within a **6.34% band** — short-context accuracy no longer separates frontier systems.
+- At 128K the picture inverts: several open-weight leaders lose more than 13%, while
+  **Gemini-3.1-Pro retains 51.99% and degrades least overall (a 2.11% drop)**.
+- Per-type ceilings range from **97.78% on AR** down to **44.06% on MSR (Kimi-K2.5)** at 32K.
+- Only Kimi-K2.5 (44.06%) and Gemini-3.1-Pro (32.17%) clear 30% on MSR — the shared capability ceiling.
+- Answer refusal is the ability most exposed to context growth: GLM-4.6V AR falls 93.33% (32K) → 30.00% (128K).
+- Memory agents trail LVLMs on nearly all types, with the largest gaps on IE, KU and AR.
+
+### 6.3 Analysis (p.8–10 — MEASURED numbers)
+
+- **Agents lose visual faithfulness.** Both text-only and multimodal pipelines compress image
+  evidence at storage time; the gap persists when BLIP captions replace raw images, so the
+  bottleneck is lossy cross-modality storage, not the answering model.
+- **Post-training on agent backbones weakens abstention.** Frozen-backbone Mem0 (77.27%) and
+  MemOS (68.18%) keep much of the base model's refusal behaviour; the five backbone-finetuned
+  agents (M2A, M3-Agent, M3C, MemAgent-7B, Memory-T1) collapse to 9–22% AR. M2A reaches only
+  22.73% on the same Qwen3-VL-8B backbone that scores 81.82% under direct inference.
+- **Complementary scaling.** Six of seven agents stay within ±7% from 32K to 256K
+  (retrieve-then-reason is length-invariant); LVLMs degrade substantially (IE ~−20%, KU ~−12%,
+  AR ~75% at 32K → ~45% at 128K). MSR's apparent flatness is a floor effect near 30%.
+- **Error decomposition at 128K.** Nearly 90% of IE and KU errors are in the "Visual" category;
+  MSR is the only type whose errors are dominated by "Reasoning" (73%). An oracle-retrieval
+  diagnostic restores MSR accuracy to 90–100% when only the required sessions are supplied,
+  indicating long-context reasoning failure is largely induced by upstream retrieval failure.
+- **Correlations.** IE–KU is the strongest type pair at 32K (ρ = 0.65); MSR correlates weakly with
+  IE and AR, supporting an evidence-retrieval axis (IE, KU) versus an aggregation axis (MSR).
+
+### 6.4 Conclusion (p.10 — quoted in §7)
+
+"even the strongest LVLM reaches only 58.68% at 32K". The paper closes on visual-evidence
+retention and retrieval — not raw scaling of context or memory — as the principal bottleneck.
+
+## 7. Verified quotations (16/16 matched against the extracted page text)
+
+| # | Page | Quotation |
+|---|---|---|
+| Q1 | 1 | "removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images" |
+| Q2 | 2 | "the five memory abilities are largely independent: strong information extraction does not predict multi-session reasoning, and multi-session reasoning caps most evaluated systems below 30%" |
+| Q3 | 2 | "long-context LVLMs deliver high short-context accuracy through direct visual grounding, but this advantage shrinks as conversations grow" |
+| Q4 | 2 | "neither approach comes close to solving the task of long-term memory" |
+| Q5 | 3 | "789 questions instantiated at four standardized input lengths" |
+| Q6 | 5 | "Evidence sessions" (Table 2 row label, value 2,145) |
+| Q7 | 6 | "We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS" |
+| Q8 | 6 | "65.7% of questions are image-essential" |
+| Q9 | 6 | "GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%" |
+| Q10 | 7 | "At 32K, the top eight LVLMs fall within a 6.34% band" |
+| Q11 | 7 | "Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% drop)" |
+| Q12 | 7 | "Per-type accuracy ceilings range from 97.78% on AR down to 44.06% on MSR (Kimi-K2.5) at 32K" |
+| Q13 | 9 | "M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score 81.82% under direct inference" |
+| Q14 | 10 | "nearly 90% of information-extraction errors stem from visual perception rather than comprehension" |
+| Q15 | 10 | "even the strongest LVLM reaches only 58.68% at 32K" |
+| Q16 | 10 | "The benchmark is not intended as a training dataset for developing memory-equipped systems" |
+
+*(Q7/Q16 reproduce the paper's own typography, including the spaced small-caps "M EM L ENS"
+and the original "that score" agreement; both are verbatim.)*
+
+## 8. Caveats and limits (CLAIMED — my reading, not the paper's words)
+
+- The arXiv comment field says "Work in progress"; no peer-review venue is stated.
+- LVLMs are evaluated at only three lengths (32K/64K/128K) because many do not natively support
+  256K; agents get all four. Overall numbers are therefore not a clean 4×4 matrix.
+- Agents are scored on a 195-question canonical subset (a quarter of the dataset) while LVLMs are
+  scored on all 789 — cross-family per-system comparison inherits that asymmetry.
+- Human review was conducted by project members, not crowd-workers.
+- Scoring is LLM-as-Judge (Qwen3-VL-235B-A22B-Instruct), cross-validated by GPT-5.4-mini
+  re-judging (κ = 0.93) and a three-annotator human consensus.
+- The benchmark is explicitly released as a diagnostic, not training data.
+
+## 9. Verification (offline, deterministic)
+
+    python3 verify_memlens_archive.py
+
+Exit 0 only if **all** hold (33 checks in the passing run):
+
+1. All 16 quotations in `quotes.json` are present, verbatim (whitespace-normalised), in the
+   archived page text — i.e. no quotation in this report is invented.
+2. Every quoted page number actually contains that quotation.
+3. `SHA256SUMS` matches the archived PDF and text.
+4. `metadata.xml` carries the expected arXiv ID, title and 14 authors.
+5. This report contains all required section headers and records the 16/16 count.
+6. `memlens.pdf` has a valid PDF header.
+
+**Negative control (MEASURED):** a copy of the archive with one fabricated quotation injected
+("MemLens solves multi-session reasoning with 99.9% accuracy") makes the verifier exit **1** and
+print `QUOTE NOT IN PAPER TEXT`. The check is a real test, not a tautology.
+
+Reproduce from source:
+
+    python3 build_memlens_archive.py <outdir>   # exit 0 = built, 16/16 quotes matched

wizard-work/sphinx/archive-memlens-2026-10-22/SHA256SUMS

@@ -0,0 +1,2 @@
+b92a5b6d89999ba6ad77fbc32f8f6e9e86a39a49dfab0a8c9ee310edf380affd  memlens.pdf
+628bd8d62beb83c57b0dfe7792d3c1b25ed480adf04e37a8e93aa235f44c2bdb  memlens.txt

wizard-work/sphinx/archive-memlens-2026-10-22/build_memlens_archive.py

@@ -0,0 +1,123 @@
+#!/usr/bin/env python3
+"""Build the MemLens archive (arXiv 2605.14906v1) from scratch.
+
+Usage:  python3 build_memlens_archive.py <outdir>
+
+Steps (all network + local, no external deps):
+  1. fetch the arXiv Atom metadata for the paper id
+  2. fetch the paper PDF
+  3. extract full text and per-page text with pdftotext
+  4. match the 16 anchor quotations against the extracted text -> quotes.json
+  5. write SHA256SUMS for the large artifacts
+
+Exit 0 = archive built and every anchor quotation found.
+"""
+import hashlib
+import json
+import os
+import re
+import subprocess
+import sys
+import urllib.request
+
+ARXIV_ID = "2605.14906v1"
+UA = "Mozilla/5.0"
+
+ANCHOR_QUOTES = [
+    "removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images",
+    "the five memory abilities are largely independent: strong information extraction does not predict multi-session reasoning, and multi-session reasoning caps most evaluated systems below 30%",
+    "long-context LVLMs deliver high short-context accuracy through direct visual grounding, but this advantage shrinks as conversations grow",
+    "neither approach comes close to solving the task of long-term memory",
+    "789 questions instantiated at four standardized input lengths",
+    "Evidence sessions",
+    "We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS",
+    "65.7% of questions are image-essential",
+    "GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%",
+    "At 32K, the top eight LVLMs fall within a 6.34% band",
+    "Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% drop)",
+    "Per-type accuracy ceilings range from 97.78% on AR down to 44.06% on MSR (Kimi-K2.5) at 32K",
+    "M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score 81.82% under direct inference",
+    "nearly 90% of information-extraction errors stem from visual perception rather than comprehension",
+    "even the strongest LVLM reaches only 58.68% at 32K",
+    "The benchmark is not intended as a training dataset for developing memory-equipped systems",
+]
+
+
+def norm(s):
+    s = s.replace("\u00ad", "")
+    s = re.sub(r"-\s*\n\s*", "-", s)
+    s = re.sub(r"\s+", " ", s)
+    return s.strip()
+
+
+def fetch(url, dest):
+    req = urllib.request.Request(url, headers={"User-Agent": UA})
+    with urllib.request.urlopen(req, timeout=120) as resp, open(dest, "wb") as fh:
+        fh.write(resp.read())
+    return os.path.getsize(dest)
+
+
+def pdftotext(args, pdf):
+    return subprocess.run(["pdftotext"] + args + [pdf, "-"], capture_output=True, text=True).stdout
+
+
+def main():
+    out = sys.argv[1] if len(sys.argv) > 1 else "memlens-archive"
+    os.makedirs(out, exist_ok=True)
+
+    meta = os.path.join(out, "metadata.xml")
+    pdf = os.path.join(out, "memlens.pdf")
+    txt = os.path.join(out, "memlens.txt")
+
+    print("[1/5] arXiv metadata -> %s" % meta)
+    n = fetch("https://export.arxiv.org/api/query?id_list=%s" % ARXIV_ID, meta)
+    print("      %d bytes" % n)
+
+    print("[2/5] PDF -> %s" % pdf)
+    n = fetch("https://arxiv.org/pdf/%s" % ARXIV_ID, pdf)
+    print("      %d bytes" % n)
+    head = open(pdf, "rb").read(5)
+    if head != b"%PDF-":
+        print("FATAL: downloaded file is not a PDF", file=sys.stderr)
+        return 1
+
+    print("[3/5] text extraction")
+    open(txt, "w").write(pdftotext(["-layout"], pdf))
+    raw = open(txt, encoding="utf-8").read()
+    pages = {}
+    for i in range(1, 15):
+        pages[i] = pdftotext(["-layout", "-f", str(i), "-l", str(i)], pdf)
+        open(os.path.join(out, "page_%02d.txt" % i), "w").write(pages[i])
+    print("      %d chars full text, 14 page files" % len(raw))
+
+    print("[4/5] anchor quotation match")
+    nraw, npages = norm(raw), {k: norm(v) for k, v in pages.items()}
+    results, misses = [], 0
+    for q in ANCHOR_QUOTES:
+        nq = norm(q)
+        if nq not in nraw:
+            misses += 1
+            results.append({"q": q, "ok": False})
+            print("      MISS: %s" % q[:70])
+            continue
+        hits = [k for k in sorted(npages) if nq in npages[k]]
+        results.append({"q": q, "ok": True, "pages": hits})
+        print("      OK p%s: %s" % (hits, q[:55]))
+    json.dump(results, open(os.path.join(out, "quotes.json"), "w"), indent=1)
+
+    print("[5/5] SHA256SUMS")
+    lines = []
+    for name in ("memlens.pdf", "memlens.txt"):
+        h = hashlib.sha256(open(os.path.join(out, name), "rb").read()).hexdigest()
+        lines.append("%s  %s" % (h, name))
+    open(os.path.join(out, "SHA256SUMS"), "w").write("\n".join(lines) + "\n")
+    for line in lines:
+        print("      " + line)
+
+    found = sum(1 for r in results if r["ok"])
+    print("\n%d/%d anchor quotations found" % (found, len(ANCHOR_QUOTES)))
+    return 1 if misses else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

wizard-work/sphinx/archive-memlens-2026-10-22/metadata.xml

@@ -0,0 +1,64 @@
+<?xml version='1.0' encoding='UTF-8'?>
+<feed xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/" xmlns:arxiv="http://arxiv.org/schemas/atom" xmlns="http://www.w3.org/2005/Atom">
+  <id>https://arxiv.org/api/9bXDYvkY10G8yc5ebyvdMPgP1Lw</id>
+  <title>arXiv Query: search_query=&amp;id_list=2605.14906v1&amp;start=0&amp;max_results=10</title>
+  <updated>2026-10-22T22:18:57Z</updated>
+  <link href="https://arxiv.org/api/query?search_query=&amp;start=0&amp;max_results=10&amp;id_list=2605.14906v1" type="application/atom+xml"/>
+  <opensearch:itemsPerPage>10</opensearch:itemsPerPage>
+  <opensearch:totalResults>1</opensearch:totalResults>
+  <opensearch:startIndex>0</opensearch:startIndex>
+  <entry>
+    <id>http://arxiv.org/abs/2605.14906v1</id>
+    <title>MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models</title>
+    <updated>2026-05-14T14:41:17Z</updated>
+    <link href="https://arxiv.org/abs/2605.14906v1" rel="alternate" type="text/html"/>
+    <link href="https://arxiv.org/pdf/2605.14906v1" rel="related" type="application/pdf" title="pdf"/>
+    <summary>Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.</summary>
+    <category term="cs.CV" scheme="http://arxiv.org/schemas/atom"/>
+    <published>2026-05-14T14:41:17Z</published>
+    <arxiv:comment>Work in progress</arxiv:comment>
+    <arxiv:primary_category term="cs.CV"/>
+    <author>
+      <name>Xiyu Ren</name>
+    </author>
+    <author>
+      <name>Zhaowei Wang</name>
+    </author>
+    <author>
+      <name>Yiming Du</name>
+    </author>
+    <author>
+      <name>Zhongwei Xie</name>
+    </author>
+    <author>
+      <name>Chi Liu</name>
+    </author>
+    <author>
+      <name>Xinlin Yang</name>
+    </author>
+    <author>
+      <name>Haoyue Feng</name>
+    </author>
+    <author>
+      <name>Wenjun Pan</name>
+    </author>
+    <author>
+      <name>Tianshi Zheng</name>
+    </author>
+    <author>
+      <name>Baixuan Xu</name>
+    </author>
+    <author>
+      <name>Zhengnan Li</name>
+    </author>
+    <author>
+      <name>Yangqiu Song</name>
+    </author>
+    <author>
+      <name>Ginny Wong</name>
+    </author>
+    <author>
+      <name>Simon See</name>
+    </author>
+  </entry>
+</feed>

wizard-work/sphinx/archive-memlens-2026-10-22/page_01.txt

@@ -0,0 +1,61 @@
+                                             M EM L ENS: Benchmarking Multimodal Long-Term
+                                                Memory in Large Vision-Language Models
+
+
+                                                       Xiyu Ren1 Zhaowei Wang1 Yiming Du2 Zhongwei Xie1
+                                                 Chi Liu1 Xinlin Yang1 Haoyue Feng1 Wenjun Pan1 Tianshi Zheng1
+                                                Baixuan Xu1 Zhengnan Li3 Yangqiu Song1 Ginny Wong4 Simon See4
+arXiv:2605.14906v1 [cs.CV] 14 May 2026
+
+
+
+
+                                                                        1
+                                                                     CSE Deparment, HKUST 2 CUHK
+                                                            3
+                                                           OmniMemory (Shenzhen) Intelligent Technology Co., Ltd.
+                                                     4
+                                                       NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA
+                                                                {xrenaf, zwanggy, yqsong}@cse.ust.hk
+                                                            ydu@se.cuhk.edu.hk lzhengnan389@gmail.com
+
+
+
+                                                                                      Abstract
+
+                                                  Memory is essential for large vision-language models (LVLMs) to handle long,
+                                                  multimodal interactions, with two method directions providing this capability: long-
+                                                  context LVLMs and memory-augmented agents. However, no existing benchmark
+                                                  conducts a systematic comparison of the two on questions that genuinely require
+                                                  multimodal evidence. To close this gap, we introduce M EM L ENS, a comprehensive
+                                                  benchmark for memory in multimodal multi-session conversations, comprising
+                                                  789 questions across five memory abilities (information extraction, multi-session
+                                                  reasoning, temporal reasoning, knowledge update, and answer refusal) at four
+                                                  standard context lengths (32K–256K tokens) under a cross-modal token-counting
+                                                  scheme. An image-ablation study confirms that solving M EM L ENS requires vi-
+                                                  sual evidence: removing evidence images drops two frontier LVLMs below 2%
+                                                  accuracy on the 80.4% of questions whose evidence includes images. Evaluating
+                                                  27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs
+                                                  achieve high short-context accuracy through direct visual grounding but degrade
+                                                  as conversations grow, whereas memory agents are length-stable but lose visual fi-
+                                                  delity under storage-time compression. Multi-session reasoning caps most systems
+                                                  below 30%, and neither approach alone solves the task. These results motivate hy-
+                                                  brid architectures that combine long-context attention with structured multimodal
+                                                  retrieval. Our code is available at https://github.com/xrenaf/MEMLENS.
+
+
+
+                                         1    Introduction
+
+                                         Memory is essential for enabling large vision-language models (LVLMs) [1, 2] to maintain con-
+                                         sistency across extended multimodal interactions and to continuously incorporate new information
+                                         over time [3–5]. When LVLMs are deployed as agents, the inputs they handle accumulate incre-
+                                         mentally rather than arriving all at once. As interaction histories expand, the agent must recall past
+                                         content and reason over the growing context to remain consistent with prior turns. At the same time,
+                                         it must integrate new facts as they arrive and revise outdated information to keep its knowledge
+                                         current. Two directions have emerged to provide this capability, namely long-context LVLMs and
+                                         memory-augmented agents. Long-context LVLMs enlarge the native context window so that the
+                                         complete dialogue history, including interleaved images, can be processed directly by the model [5–7].
+                                         Memory-augmented agents, building on retrieval-augmented generation, instead compress, index,
+                                         and selectively retrieve past content from an external store [8–10].
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_02.txt

@@ -0,0 +1,73 @@
+Table 1: Left: Comparison of M EM L ENS with existing long-context and conversational memory
+benchmarks. Type — Dial (Dialogue) or Doc (Document); Max L — maximum context length
+reported (“—” if not applicable); L Control — standardized token length control; Comp. Eval. —
+paper benchmarks both LVLMs and memory-augmented agents on the same data; Memory Tasks —
+dot pattern showing coverage of the five evaluation tasks (• = supported, ◦ = not supported) in fixed
+order: IE (Information Extraction), MSR (Multi-Session Reasoning), TR (Temporal Reasoning),
+KU (Knowledge Update), AR (Answer Refusal). Right: Distribution of the 789 evaluation questions
+in M EM L ENS across the five major task types (inner ring) and nine reporting subtypes (outer ring).
+
+                              Max    Multi- Multi- L     Comp. Memory
+Benchmark              Type                                                            Refuse               Entity
+                               L     modal Sess. Control Eval.  Tasks                  11.4%                15.2%
+MemoryBank [18]        Dial    —      ✗     ✓      ✗      ✗   •◦◦◦◦                     AR
+LoCoMo [16]            Dial   10K     ✓     ✓      ✗      ✗   •••◦◦      Update        11.4%
+                                                                         14.7%                                IE
+MM-NIAH [12]           Doc    128K    ✓     ✗      ✓      ✗   •◦◦◦◦                KU                       31.2%
+                                                                                  14.7%                              PrevInfo
+                                                                                                                      16.0%
+MMLongBench-Doc [13]   Doc    128K    ✓     ✗      ✗      ✗   •◦◦◦◦
+MRAG-Bench [19]        Doc     —      ✓     ✗      ✗      ✗   •◦◦◦◦
+                                                                         Count.
+LongMemEval [14]       Dial   1.5M    ✗     ✓      ✓      ✗   •••••       5.8%      MSR
+MMLongBench [11]       Doc    128K    ✓     ✗      ✓      ✗   •◦◦◦◦                18.1%              TR
+                                                                           Ent. Res.                 24.6% Temp. Gnd.
+Multimodal NIAH [20]   Doc    128K    ✓     ✗      ✓      ✗   •◦◦◦◦         6.0%
+                                                                                   Arith.                        13.1%
+MemAgentBench [15]     Dial   115K    ✗     ✓      ✗      ✓   •◦◦◦◦                 6.3%        Dur. Cmp.
+                                                                                                 11.5%
+M EM L ENS (Ours)      Dial   256K    ✓     ✓      ✓      ✓   •••••
+
+
+
+
+Despite progress along both directions, no current benchmark conducts a length-controlled compari-
+son of long-context LVLMs and memory-augmented agents on questions that genuinely require visual
+evidence. Long-context multimodal benchmarks measure context-length scaling on long documents
+and retrieval tasks [11–13], but cover mainly LVLMs and do not place them alongside memory-
+augmented agents for direct comparison. Text-only conversational memory benchmarks such as
+LongMemEval [14] and MemoryAgentBench [15] overlook the visual modality entirely, treating
+memory as a single-modality problem. Multimodal conversational benchmarks such as LoCoMo [16]
+and Mem-Gallery [17] retain both visual and text modalities, yet most of their questions admit a
+text-only shortcut, rendering the visual modality redundant. As Table 1 summarizes, no existing
+benchmark requires visual evidence to answer its questions while supporting a length-controlled
+comparison of long-context LVLMs and memory-augmented agents.
+To bridge the gap, we introduce M EM L ENS, a comprehensive benchmark for assessing memory in
+multimodal multi-session conversations. M EM L ENS consists of 789 questions covering five core
+memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge
+update, and answer refusal. We design questions that demand cross-modal reasoning over both
+visual and textual evidence, requiring the retrieval of multimodal information hidden within one
+or more conversations between a user and an assistant. To verify that solving M EM L ENS requires
+visual evidence, we conduct an image-ablation study on the 80.4% of questions whose evidence
+includes images: when these images are removed, the accuracy of two frontier LVLMs collapses
+below 2% (§3.4). Following the needle-in-a-haystack construction paradigm [21], we design a
+four-stage pipeline that builds a coherent multi-session chat history for each question, distributing
+the evidence across one or more user-assistant sessions alongside topically related distractor turns.
+Because distractor turns can be added independently of the evidence, the chat length is freely
+extensible. We release four standard context lengths (32K, 64K, 128K, and 256K tokens) under a
+cross-modal token-counting scheme [11] that aligns text and vision tokens. M EM L ENS thus enables a
+length-controlled comparison of long-context LVLMs and memory-augmented agents on multimodal
+questions spanning five comprehensive memory abilities.
+Using M EM L ENS, we evaluate 27 LVLMs and 7 memory-augmented agents across all four context
+lengths (32K, 64K, 128K, and 256K). Our evaluation yields three key findings. First, the five
+memory abilities are largely independent: strong information extraction does not predict multi-
+session reasoning, and multi-session reasoning caps most evaluated systems below 30%. Second,
+the two approaches exhibit complementary failure modes: long-context LVLMs deliver high short-
+context accuracy through direct visual grounding, but this advantage shrinks as conversations grow;
+memory agents, in contrast, are length-stable but lose visual fidelity under storage-time compression,
+and memory-oriented post-training of agent backbones can additionally weaken their abstention
+behavior. Third, neither approach comes close to solving the task of long-term memory. These
+
+
+                                                    2
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_03.txt

@@ -0,0 +1,59 @@
+results point to a clear next step: architectures that combine long-context attention with structured
+multimodal retrieval, rather than scaling either component in isolation.
+
+2     Related Work
+Memory-Augmented LLM Agents. Recent surveys systematize memory representations and oper-
+ations across LLM-based agents [22]. Text-only memory agents span structured-fact stores [18, 23],
+OS-inspired paging [8], tree-summarized retrieval [24], neurobiological graphs [25], relational em-
+beddings [9], agentic self-organizing notes [26], intent-driven memory selection [27], RL-selected
+time-aware memory [28], layered memory tiers [29], and sliding-window RL agents [30]. Mul-
+timodal extensions add ColPali-style document retrieval [31, 32], multimodal embeddings [33],
+dual-layer semantic memory [34], entity-centric audio-visual memory [35], sparse video mem-
+ory [36], LoRA-tuned session retrieval [37], and intent-guided multimodal response retrieval over
+multi-session conversations [38]. Recent work probes the intersection of long-context LLMs and
+retrieval-augmented generation [10, 39, 40], suggesting retrieval and long attention are complemen-
+tary rather than competing.
+
+Long-Context and Conversational Memory Benchmarks. Most current long-context benchmarks
+established core protocols for retrieval and length scaling [41–47], but neglect the visual modality.
+Multimodal extensions, such as MMLongBench [11], scale context length over documents [11, 13],
+needle-style retrieval [12, 20], long-form video [48, 49], super-long documents [50, 51], and multi-
+image inputs [52–55]. Yet their inputs are documents or videos rather than multi-session conver-
+sations. Memory under conversational interaction therefore remains unexercised. Conversational
+memory benchmarks restore the multi-session structure but rarely require visual perception: Long-
+MemEval [14] (500 questions over five memory abilities, up to 1.5M tokens), PerLTQA [56],
+and MemoryAgentBench [15] (retrieval, test-time learning, long-range understanding, selective
+forgetting) discard images entirely, while LoCoMo [16] and Mem-Gallery [17] embed images in
+persona-grounded dialogue but allow most questions to be answered from text alone, so neither
+stress-tests multimodal memory.
+
+3     The M EM L ENS Benchmark
+We propose M EM L ENS, a multimodal long-term conversational memory benchmark that comprises
+789 questions instantiated at four standardized input lengths (32K/64K/128K/256K tokens). In
+contrast to prior benchmarks restricted to text-only conversations and questions, M EM L ENS provides
+multimodal conversation sessions in which text and images are interleaved, together with questions
+that require cross-modal reasoning over evidence images and the surrounding textual context. §3.1
+defines the five memory abilities, §3.2 describes the construction pipeline (Figure 1), §3.3 presents
+the quality control, and §3.4 describes the cross-modality validation.
+
+3.1   Memory Abilities
+
+To comprehensively reflect the retrieval, reasoning, and update that a conversational assistant must
+perform over a long multimodal history, M EM L ENS formulates five core memory abilities. We break
+each ability into subtypes that target specific reasoning operations.
+
+       • Information Extraction (IE) tests the recall of a specific fact from a single evidence session,
+         with two subtypes. Entity questions form a two-hop chain: the model first identifies an
+         abstracted entity in the evidence image, then retrieves the associated information from the
+         surrounding text. PrevInfo (previous information) questions instead abstract the session
+         reference, asking the model to recall a visual detail from an image shared in an earlier
+         session. The visual identification hop draws on image-understanding skills such as fine-
+         grained recognition, object counting, spatial reasoning, and numerical reasoning.
+       • Multi-Session Reasoning (MSR) evaluates aggregation across three to eight sessions: count-
+         ing tallies unique items identified only by their evidence images, arithmetic sums values
+         stated in text or embedded in visual artifacts, and entity resolution determines co-reference
+         by comparing images across sessions.
+
+
+                                                   3
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_04.txt

@@ -0,0 +1,97 @@
+                                                             Multimodal Session Simulation                                                                                                   Conversation History Assembly
+   1 Topic Ontology                             2 Web Image                                    3      Two-agent                           4 Multimodal Session
+                                                                                                                                                                                             M M: Multimodal Session
+                                                     Retrieval + Filtering                            Dialogue
+                                                                                                                                                                                             E E: Evidence Session
+                                                                                                   VLM                   VLM                                                                 F F: Filler Session (Text-only)
+                                                                                                   User                Assistant               ···
+                                                                                                                                                                                     ···     Assembled Multimodal Conversation History
+                                                                                                                                               ···
+                                                                                                                                                                                               11/02       M       ···
+                                                                                                                                                                                     ···
+           Topic Pool
+                                                                                                                                                                                               12/02        F      ···
+
+                                     ✓ image selected before dialogue → structural image-text alignment                                                                                           。         E      ···
+                                                                                                                                                                                                  。         F      ···
+                                     Question-Answer Pair Construction                                                                  Evidence Session Construction                             。
+                                                                                                                                                                                                  。        M       ···
+  1 Background Generation                 2     Visual Anchor Retrieval             3     Entity Ambiguity                                                     VLM / Agent
+                                                                                                                                                                                                  。         E      ···
+      Sampled Topic Ontology                                              replace name with generic placeholder                                                                                   。
+                                     Anchor Entity     Golden             BEFORE (named entity)           AFTER (generic placeholder)
+                                                      Gate Bridge                                                                                : What time did I meet Emma                                F      ···
+                                                                               Emma and I                   Emma and I                       at the Golden Gate Bridge last
+                                                                               finally met                  finally met                      Saturday?                                         14/04        E      ···
+                                                                               crossed the                  crossed the
+                                                                               Golden Gate                  bridge <image>                           : 6:42 AM
+                                                                               Bridge together              together last                                                                        Fillers (text-only sessions)
+      Emma and I finally met                                                   last Saturday ···            Saturday ···                             Atomic Facts (evidence units)
+      crossed the Golden Gate
+      Bridge together last                                                                                                                           F1                        F2
+                                                                                                                                            My sister Emma              My sister and I
+      Saturday, three years after
+                                                                                                                                            and I crossed the           started crossing
+                                                                                                                                                                                                      From ShareGPT / UltraChat
+      we first planned to. We met                                                                                                           bridge in <image>
+                                                                               Golden Gate Bridge             the bridge <image>                                        the bridge at 6:42
+      on the the south side of the                                                                                                          last Saturday               AM that morning.
+      bridge at at 6:42 AM to                                                                                                               morning.
+      catch the sunrise, and I
+      wore Dad’s red windbreaker
+                                                                             Cross with    Saturday       Cross with      Saturday                                                                   Fixed text/image ratio
+      from his 1998 photo.
+                                                                             Emma          6:42 AM        Emma            6:42 AM           indirectly mention the fact in conversation                 Context Budgets
+      Halfway across, near the
+                                                                                                                                                     S1                        S2
+      second tower, Marcus
+      proposed to Emma. We                                                      text-only path (unsolvable)
+      celebrated afterward at Pier                                                                                                                                                             32K         64K           128K     256K
+                                     Anchor Image
+      39.
+                                                                          ✗ Requires image evidence
+
+
+                                                                    Figure 1: M EM L ENS construction pipeline.
+
+
+               • Temporal Reasoning (TR) assesses joint reasoning over temporal references, including both
+                 natural-language expressions and session timestamps, together with visual content: duration
+                 comparison compares two intervals derived from textual or visual cues, and temporal
+                 grounding either orders events chronologically or extracts the specific date of an event.
+                 Beyond entity abstraction, TR also replaces temporal expressions with visual artifacts such
+                 as clock faces and calendar pages.
+               • Knowledge Update (KU) tests the ability to track an evolving user attribute across a chain of
+                 four successive updates [57] (e.g., “I used to like apples” → “now I prefer kiwi”), requiring
+                 the model to reason from the final state of the chain rather than earlier superseded ones.
+               • Answer Refusal (AR) removes all supporting evidence from an otherwise answerable ques-
+                 tion, so it can no longer be answered from the remaining context; a correct model must
+                 decline rather than hallucinate [58]. AR serves as a calibration check for hallucination
+                 detection, not a core memory retrieval task.
+
+A single cross-modal principle unifies the four answerable types: the evidence image carries infor-
+mation that the text deliberately withholds, so correct answers require joint visual–textual reasoning
+(§3.2); §3.4 validates this dependency empirically. The formal problem specification appears in
+Appendix C.1. Appendix I grounds this taxonomy in established memory research, and Appendix C.3
+provides the complete subtype breakdown with representative examples.
+
+3.2        Data Curation
+
+The construction proceeds in four components (Figure 1): (i) first, multimodal session simulation
+generates topic-grounded multimodal dialogue sessions that form the conversation background;
+(ii) next, question construction produces evaluation questions whose answers require visual content
+through entity abstraction; (iii) evidence session construction wraps each evidence fact into a complete
+session that matches its topical and stylistic profile; and (iv) conversation history assembly interleaves
+evidence, haystack, and text-only filler sessions in timestamp order and compiles each question into
+four standardized input lengths.
+
+Multimodal session construction. Each session begins with topic sampling from a hierarchical
+ontology. For each topic, we generate an image query to retrieve a batch of candidate images from the
+web1 . We then filter these candidates for visual relevance. From this filtered set, we select the final
+   1 https://github.com/hellock/icrawler; sourcing, licensing, privacy, and release detailed in Appendix C.7.
+
+
+
+
+                                                                                                                            4
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_05.txt

@@ -0,0 +1,62 @@
+image. Given this image, we generate sessions through a dual-model simulation with GPT-5.1 [2]
+as the user and Gemini-3-Pro [59] as the assistant, producing multi-turn dialogues that incorporate
+the selected images into interleaved image-text sequences [60]. Because image selection precedes
+dialogue generation, image–text alignment is a structural property of the data rather than a post-hoc
+filter. The image retrieval pipeline (§C.5), full topic ontology (§C.2), and image diversity statistics
+(§C.6) are detailed in the Appendix.
+
+Question-Answer Pair. Question generation follows a four-step pipeline. (i) A topic is sampled
+from the hierarchical ontology, and a LVLM generates a background paragraph containing salient
+named-entities, for instance, a paragraph about San Francisco landmarks that mentions the Golden
+Gate Bridge. (ii) One entity is selected as the visual anchor for the question. A text query derived
+from the entity is issued to the same web-crawling and multi-model scoring pipeline used for haystack
+images (§C.5), yielding a high-relevance image [61]; the anchor may range from a famous landmark
+to a specific product model. (iii) Cross-modal dependency is enforced through entity abstraction [62–
+64]: the entity in the background paragraph is replaced with a higher level concept drawn from
+a dictionary spanning 55 entity categories (e.g., “Golden Gate Bridge” → “the bridge shown in
+<image>”). After replacement, the paragraph no longer names the entity, and only the evidence
+image can resolve the reference. (iv) The abstracted paragraph, evidence image, and original entity
+name are provided to Gemini-3-Pro [59], which generates a (question, answer) pair together with
+atomic evidence facts. The generation prompt constrains the question to require both the image and
+the surrounding text, closing the text-only shortcut documented in prior work [16]. The full pipeline
+and per-subtype generation routes appear in Appendix C.4.
+
+Evidence session. Directly inserting evidence statements into the conversation history creates
+abrupt semantic shifts that make the evidence trivially locatable by similarity-based retrieval. Prior
+work has shown that increasing contextual similarity between evidence and distractors raises retrieval
+difficulty, a design principle also adopted by LongMemEval [14]. Accordingly, each evidence fact
+is wrapped in a complete evidence session generated using the same pipeline as haystack sessions
+but grounded on the evidence fact rather than a sampled topic. Consequently, the evidence facts
+match the topical and stylistic profile of the surrounding haystack, so that evidence cannot be located
+by surface-level shortcuts. To further increase the difficulty, we also prompt the user model to
+mention the facts indirectly without stressing them: To embed the fact “I started a new job last
+month,” the user turn might open by asking about updating the tax withholding and mention the job
+change incidentally later. When an evidence fact carries an evidence image, the image is placed
+adjacent to the corresponding textual mention within the session, preserving unambiguous image-text
+co-reference after entity abstraction.
+
+Conversation history assembly. For each question, evidence              Table 2: Dataset statistics.
+sessions are inserted into a timestamp-ordered history of haystack
+sessions, with positions chosen uniformly at random (for KU, we Statistic                        Value
+preserve the evidence-session order; Appendix C.8). Haystack Questions                              789
+sessions are curated to be contextually related but uninformative Types / Subtypes                 5/9
+for the question, and never include answer-relevant details. We Evidence sessions                 2,145
+vary the number of haystack sessions to produce four standardized Avg. turns/session               ∼10
+context lengths (32K/64K/128K/256K tokens) using the cross- Avg. images/session                   ∼1.5
+modal counting scheme of MMLongBench [11] (Table 2). To Tokens/image                           ∼2,000
+avoid revealing evidence positions via image clustering, we keep                          32K → 256K
+a fixed text-per-image ratio across the history, padding with text- Sessions/instance         14 → 93
+only filler sessions from ShareGPT and UltraChat [65]. A post- Images/instance              20 → 138
+hoc classifier achieves only marginally above-chance accuracy
+when separating evidence from haystack text (Appendix G.8), and the generator scales beyond 256K.
+
+3.3   Quality Control
+
+Automated filtering. Each candidate question is screened by two automated checks. A rule-based
+pre-filter removes images from the question and evidence and drops cases where the remaining text
+already determines the answer. An LLM judge (GPT-5.1 [2]) then sees only the question text (without
+evidence or conversation history) and removes items solvable from parametric knowledge alone [66].
+
+
+                                                  5
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_06.txt

@@ -0,0 +1,64 @@
+Table 3: Cross-modality ablation on the image-essential and image-supportive questions. With
+evidence: question + evidence facts + evidence images, no haystack (n = 634). W/o evidence:
+question + evidence facts, no evidence images.
+        Model            Input                 Overall    IE     MSR     TR     KU       ∆
+                         With evidence image    93.13    94.31 100.00 96.91 75.86 —
+        GPT-5.4
+                         W/o evidence image      1.74     0.41  0.00  5.15 0.00 −91.39
+                         With evidence image    89.42    89.02   90.21   96.19 82.24 —
+        Gemini-3.1-Pro
+                         W/o evidence image      1.89     0.00   0.00    6.19 0.00 −87.53
+
+
+
+Remaining questions therefore require using the evidence image, which is a prerequisite for the
+multimodal analyses below.
+
+Human review. Three rounds of human review operate as a cumulative quality gate on the filtered
+questions. Round 1 verifies that the evidence image carries answer-critical information; for AR
+questions, which are constructed by removing the evidence facts, this round instead confirms that the
+removed facts was answer-critical. Round 2 checks that evidence facts are naturally embedded and
+recoverable from their sessions, and that each session reads as a plausible user–assistant exchange
+with natural dialogue flow; sessions that fail the naturalness criterion are subsequently refined by
+humans. Round 3 reviews the haystack sessions to assess both the quality of the images and the
+naturalness of the dialogues. Together, the automated filters and three-round review reduce the initial
+pool of 20k candidates to the final 789 questions. The filler sessions drawn from ShareGPT and
+UltraChat (§3.2) further ground the surrounding context in authentic user–AI conversational patterns.
+Annotation guidelines and inter-annotator agreement appear in Appendix E.1.
+
+3.4   Cross-modality Validation
+
+The construction pipeline targets image-necessary questions: across M EM L ENS, 65.7% of questions
+are image-essential (the answer is unrecoverable without the evidence image), 14.7% are image-
+supportive (the image confirms or disambiguates a textual fact), and 19.6% are text-sufficient (all AR
+questions plus a subset of MSR items that test cross-session reasoning over purely textual evidence).
+We verify this cross-modal dependency with two empirical checks (Table 3). An answerability test
+supplies each image-essential and image-supportive question (n = 634) with its full evidence (textual
+facts and evidence images) and confirms that the questions are answerable: GPT-5.4 reaches 93.13%
+overall and Gemini-3.1-Pro 89.42%. A multimodal ablation then removes all evidence images:
+overall accuracy collapses below 2% for both models. Two frontier proprietary LVLMs converge on
+near-identical collapses, showing our questions are highly multimodal.
+
+
+4     Evaluation and Analysis
+
+4.1   Experimental Setup
+
+We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS. The LVLMs span
+closed-source systems including GPT-5.4 [2], Claude Sonnet 4.5 [7], and Gemini-3.1-Pro [67],
+alongside major open-source families such as Kimi-K2.5 [68], Qwen3.5 [69], GLM-4.6V [70],
+and Gemma3 [71]; the complete list is provided in Appendix B. The memory-augmented agents
+comprise three multimodal pipelines (M3-Agent [35], M2A [34], and M3C [37]) and four text-only
+pipelines (Mem0 [9], MemOS [29], MemAgent-7B [30], and Memory-T1 [28]), with backbone
+and adapter details reported in Appendix G. Since the four text-only memory agents do not accept
+visual inputs, we follow standard practice and replace the image inputs with captions generated by
+BLIP-2 [72]; the remaining input adaptations are documented in Appendix B. For comparability,
+the LVLMs are evaluated at three context lengths (32K, 64K, and 128K), as many of them do not
+natively support 256K, whereas the agents are evaluated across all four context lengths (32K, 64K,
+128K, and 256K). We report LLM-as-Judge accuracy [73] using Qwen3-VL-235B-A22B-Instruct
+as the judge, cross-validated by GPT-5.4-mini re-judging (κ = 0.93) and a three-annotator human
+consensus (Appendix E.2).
+
+
+                                                  6
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_07.txt

@@ -0,0 +1,74 @@
+                            IE            MSR                   TR              KU              AR             Overall
+  Claude Sonnet 4.5 28.0 26.4 19.5     13.3 9.1    2.8   40.2 35.1 33.0   29.3 21.6 16.4    97.8 94.4 93.3    36.5 32.4 27.8
+
+       Gemini-3.1-Pro 57.3 58.9 55.8   32.2 31.5 29.4    40.9 42.3 41.2   49.1 48.6 46.0    97.8 95.6 96.7    54.1 53.0 52.0
+
+              GPT-5.4 69.5 63.0 60.2   28.2 27.3 21.7    39.2 42.3 39.7   47.4 43.1 43.1    97.8 96.7 94.4    52.7 52.3 49.6
+
+            Kimi-K2.5 54.8 52.4 51.6   44.1 35.7 28.7    53.1 52.1 48.2   50.9 49.1 48.6    97.8 97.8 93.3    54.9 54.0 52.0
+
+Qwen3.5-122B-A10B 74.4 67.1 43.1       30.1 27.3 22.4    51.5 51.5 41.8   49.1 48.6 46.0    88.9 84.4 83.3    58.7 55.9 45.5
+
+         Qwen3.5-27B 70.3 62.2 54.1    29.4 21.0 14.0    39.2 35.6 34.0   42.2 44.0 38.8    86.7 67.8 63.3    53.0 46.1 40.7
+
+Nemotron-Nano-12B 63.8 56.1 —          16.8 21.0   —     44.8 42.8   —    39.7 34.5   —     45.6 35.6   —     45.0 40.9   —
+
+Cosmos-Reason2-8B 55.7 46.3 37.0       23.8 20.3 20.3    48.5 45.9 43.8   31.9 24.1 12.9    68.9 54.4 35.6    46.1 39.2 31.9
+
+      Phi4-Multimodal 31.7 28.0 22.8   20.3 21.0 22.4    43.8 39.2 30.9   19.8 24.1 14.7    43.3 21.1 15.6    32.2 28.1 22.7
+
+            GLM-4.6V 59.3 61.8 52.8    20.3 23.8 23.1    53.1 53.1 43.8   44.0 34.5 28.4    93.3 64.4 30.0    52.3 49.0 39.0
+
+         Gemma3-27B 35.0 26.4 19.1     23.8 22.4 18.2    41.2 44.3 37.1   30.2 27.6 13.8    68.9 61.1 48.9    37.6 34.2 26.0
+
+      Qwen3-VL-235B 51.6 38.6 29.3     25.2 18.2 15.4    49.5 42.8 43.3   36.2 35.3 30.2    88.9 75.6 67.8    48.3 39.7 34.7
+
+        Qwen3-VL-30B 41.5 39.0 33.7    11.9 11.9 10.5    35.6 37.1 36.6   30.2 29.3 20.7    71.1 65.6 60.0    36.4 35.2 31.3
+
+                Mem0 13.1 11.5 8.2     25.7 22.9 20.0    50.0 45.8 50.0   17.2 24.1 24.1    77.3 77.3 77.3    31.8 31.3 30.3
+
+              MemOS 18.0 18.0 16.4     22.9 22.9 22.9    39.6 39.6 43.8   24.1 24.1 17.2    68.2 68.2 81.8    30.3 30.8 31.3
+
+        MemAgent-7B 18.0 13.1 14.8     25.7 28.6 28.6    62.5 45.8 60.4   41.4 24.1 20.7    13.6 4.5    9.1   32.8 25.6 28.2
+
+           Memory-T1 18.0 19.7 21.3    25.7 22.9 22.9    62.5 56.2 56.2   13.8 17.2 27.6    9.1 18.2 13.6     28.7 28.7 30.3
+
+            M3-Agent 18.0 18.0 22.9    22.9 17.1 20.0    29.2 37.5 22.9   6.9   3.5   6.9   13.6 4.5 18.2     19.5 19.0 19.5
+
+                 M2A 14.8 18.0 21.3    8.6 14.3 11.4     2.1 25.0 27.1    0.0   0.0   0.0   22.7 13.6 13.6    15.4 15.9 16.9
+
+                       32K 64K128K     32K 64K128K       32K 64K128K      32K 64K128K       32K 64K128K       32K 64K128K
+
+Figure 2: Per-type accuracy (%) by context length for representative 13 LVLMs and 6 memory-
+augmented agents. Each panel shows one question type; cells use a green colormap. Missing
+cells indicate models that exceed their usable context budget at 128K. LVLMs are evaluated on the
+full 789-question benchmark, agents on the 195-question canonical subset. Full rosters appear in
+Tables 13 and 14.
+
+
+4.2     Main Results
+
+Overall performance. Figure 2 reports per-type accuracy across context lengths for a represen-
+tative 13-LVLM cohort and six memory agents; the full 27-LVLM and seven-agent rosters are in
+Appendix G.1. At 32K, the top eight LVLMs fall within a 6.34% band, so short-context accuracy
+no longer separates frontier systems. The picture inverts at 128K: several open-weight leaders lose
+more than 13%, while Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11%
+drop). AR shows the steepest context-driven decline in the open-weight LVLM family, suggesting
+that hallucination control is the ability most exposed to evidence dilution at long contexts. Memory
+agents occupy a narrower range, with the top four text-only systems clustering within 5% of each
+other, while M3-Agent, M3C, and M2A fall substantially lower. Memory agents trail LVLMs across
+nearly all types, with the largest gaps on visually grounded retrieval (IE, KU) and on answer refusal
+(AR).
+
+Type-specific difficulty. Per-type accuracy ceilings range from 97.78% on AR down to 44.06%
+on MSR (Kimi-K2.5) at 32K. AR, which serves as a calibration check for hallucination detection,
+is the easiest type as expected, although its apparent ease erodes substantially at long contexts
+for open-weight LVLMs (e.g., GLM-4.6V AR drops from 93.33% at 32K to 30.00% at 128K).
+TR follows (60.82% ceiling), because timestamps and dates in session metadata provide explicit
+retrieval anchors. IE (74.39% ceiling) and KU (50.86%) form an intermediate band: IE requires
+visual grounding, with Entity questions demanding two-hop reasoning through the abstracted image
+reference, while KU requires tracking a four-fact update chain in which missing a single image anchor
+
+
+                                                            7
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_08.txt

@@ -0,0 +1,85 @@
+                                                                                              Visual         Textual        Output
+                                                                                              Mixed          Reasoning
+                                       IE    TR   AR                              100
+                                                                                                       8%
+               80                      MSR   KU                                                                                  14%
+
+
+
+
+                                                             % of wrong answers
+               70                                                                  80                                            11%
+                                                                                                                    44%
+               60
+Accuracy (%)
+
+
+                                                                                   60
+               50                                                                                      73%
+
+               40                                                                  40
+                                                                                        89%
+                                                                                                                                 75%
+               30                                                                                                   42%
+
+
+               20                                                                  20
+
+               10                                                                                      10%          11%
+                                                                                    0
+                0                                                                       IE         MSR               TR           KU
+                    32K         64K               128K                                                 Memory type
+                           Context length
+                                                             (b) Wrong-answer error decomposition at 128K con-
+(a) Per-type accuracy vs. input length. Solid: LVLM          text across the four answerable memory types, split
+average (27 models, n=789); dashed: agent average            by failure modality (visual, textual, mixed, reasoning,
+(7 systems, n=195). Bands: 95% CI.                           output). Category definitions appear in Table 19.
+
+
+flips the predicted state. MSR is the hardest, as cross-session aggregation over three to eight sessions
+defeats every evaluated system: only Kimi-K2.5 (44.06%) and Gemini-3.1-Pro (32.17%) clear 30%
+by margin, exposing this as the shared capability ceiling of M EM L ENS. Within types, IE-Entity is
+consistently harder than single-hop IE-PrevInfo, and MSR-Arithmetic is the most difficult subtype
+overall (Appendix C.3).
+
+No model dominates all memory abilities. No single model fam-                                                      IE MSR TR KU AR
+ily dominates across all types (Figure 3). GLM-4.6V [70] leads                                            GPT-5.4 69.5 28.2 39.2 47.4 97.8
+TR but collapses on KU, while Qwen3.5 [69] inverts the pattern.                                 Gemini-3.1-Pro 57.3 32.2 40.9 49.1 97.8
+Kimi-K2.5 [68] is relatively strongest on MSR at 32K, though this
+                                                                                                       Kimi-K2.5 54.8 44.1 53.1 50.9 97.8
+advantage fades at longer contexts. Gemini-3.1-Pro is the only
+                                                                                                   Qwen3.5 (2) 72.4 29.7 45.4 45.7 87.8
+model simultaneously competitive on IE, KU, and MSR at 128K.
+                                                                                                        GLM-4.6V 59.3 20.3 53.1 44.0 93.3
+Memory agents exhibit an inverted profile: Memory-T1 reaches
+                                                                                                            Mem0 13.1 25.0 50.0 17.2 77.3
+high TR accuracy through BM25 date matching but falls well below
+direct LVLMs on IE, substituting keyword retrieval for the visual                                MemAgent-7B 17.2 25.0 61.5 40.6 13.6
+
+grounding that IE demands.                                                                             Memory-T1 18.0 25.0 63.5 13.8 9.1
+
+                                                                                                        M3-Agent 19.0 22.2 30.6 6.2 13.6
+
+4.3             Analysis
+                                                                                                Figure 3: Memory-ability spe-
+Current memory pipelines lose faithfulness to original visual cialization across representative
+                                                                         LVLMs and memory agents.
+evidence. The gap between agents and LVLMs is largest on the
+visually grounded types (IE and KU). Despite their different input formats (§4.1), both text-only
+and multimodal pipelines compress evidence visual information into a fixed memory representation
+at storage time, leaving the original image pixels inaccessible at query time. Retrieval over these
+compressed encodings is also less reliable than retrieval over the raw conversation text. The loss
+is sharpest on IE and KU because captions and embeddings retain only the gist of an image and
+discard fine-grained visual cues, such as counts, attributes, and spatial relations, that these two
+types specifically probe. The same gap appears when blip-captions replace raw images for text-only
+agents, indicating that the bottleneck lies in lossy cross-modality storage rather than in the answering
+model. Closing this gap requires memory architectures that preserve image-level evidence rather than
+caption-based compression.
+
+Post-training on memory agent backbones weakens abstention. Memory agents fall far below
+their direct-inference counterparts on AR. We evaluate two agents that keep the backbone frozen,
+Mem0 (77.27%) and MemOS (68.18%), and five that further finetune the backbone via RL or LoRA
+
+
+                                                         8
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_09.txt

@@ -0,0 +1,79 @@
+for memory management (M2A, M3-Agent, M3C, MemAgent-7B, and Memory-T1). While the
+frozen-backbone agents preserve much of the abstention behavior of the underlying LLM, the others
+collapse to 9–22% AR. M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score
+81.82% under direct inference, and the backbone ablation in Appendix G.6 confirms that stronger
+backbones alone do not close this gap. This suggests that the reward design of current RL/SFT
+fine-tuning on memory agent backbones optimizes mainly answer correctness and retrieval success,
+providing no signal that refusing an unanswerable question is correct, so abstention degrades after
+training. Future agent designs should therefore optimize memory access, answer accuracy, and
+evidence-sensitive abstention jointly [74], rather than treating memory management as independent
+of hallucination control.
+
+LVLMs and memory agents show complementary scaling trade-offs. Figure 4a plots accuracy
+across five abilities from 32K to 128K for both LVLMs and memory agents. The two approaches
+respond to context scaling in structurally different ways. LVLMs degrade substantially on retrieval-
+heavy types, with IE and KU losing ∼20% and ∼12% respectively as evidence images become harder
+to locate among growing filler content. MSR shows apparent flatness that reflects a floor effect near
+30% rather than genuine robustness. AR shows the steepest LVLM decline (∼75% at 32K to ∼45%
+at 128K), suggesting that growing filler content erodes abstention and pushes LVLMs to hallucinate
+on unanswerable questions rather than refuse them. Six of seven memory agents, by contrast, stay
+within ±7% from 32K to 256K because their retrieve-then-reason pipeline is length-invariant by
+construction. The two failure modes are orthogonal: LVLMs lose to context length, while memory
+agents lose to lossy multimodal compression at storage time. Because each architecture covers only
+one axis, scaling along that axis leaves the other failure mode untreated, motivating hybrid designs
+that span both axes.
+
+Memory ability correlations reveal distinct sources of difficulty.
+We analyze pairwise Spearman correlations among the five question         IE 1.00 -0.19 0.12 0.65 0.36 0.23
+                                                                                                        0.31
+
+
+
+types at 32K (Figure 5). Correlations vary substantially across type MSR -0.19 1.00 0.33 0.21 -0.11 0.060.21
+
+pairs, indicating that the types do not measure a single capability. IE
+and KU form the most consistent retrieval-oriented pair: they have       TR 0.12 0.33 1.00 0.29 -0.00 0.18
+                                                                                                        0.13
+
+
+
+the strongest correlation at 32K and remain significantly correlated     KU 0.65 0.21 0.29 1.00 0.59 0.43
+                                                                                                        0.19
+
+
+across context lengths, reflecting their shared need to first locate
+the relevant evidence image. KU also correlates with AR at 32K,          AR 0.36 -0.11-0.00 0.59 1.00 0.21
+                                                                                                        0.28
+
+
+
+suggesting that some KU questions still depend on accurate evidence           IE MSR TR KU AR Avgstd
+
+selection. In contrast, MSR shows weak correlations with IE and Figure 5: Spearman rank correla-
+AR at all lengths, implying that its main challenge is aggregating tion (ρ) at 32K across all 34 evalu-
+information across multiple evidence pieces rather than retrieving ated LVLMs and memory agents.
+a single image. These findings reveal two complementary difficulty
+axes: an evidence-retrieval axis (IE, KU) and an aggregation axis (MSR). They also illustrate how
+a single aggregate score can obscure differences among long-context abilities, motivating per-type
+evaluation alongside overall performance [43, 44]. Full correlation matrices at 64K and 128K appear
+in Appendix G.5.
+
+Error analysis. Figure 4b decomposes wrong answers at 128K across the four answerable types,
+with the category definitions provided in Appendix G.4. On IE and KU, nearly 90% of errors fall in
+the Visual category, showing that the model fails to locate or read the evidence image; once the image
+is reached, the answer is usually extracted correctly. TR errors split between Mixed and Reasoning,
+reflecting both image-supportive grounding and small closed-set selection. MSR is the only type
+whose errors are dominated by the Reasoning category (73%). We further conduct an oracle-retrieval
+diagnostic in Appendix G.7, which restores accuracy on MSR to 90–100% when only the required
+sessions are supplied. This suggests a reasoning error at long context is largely induced by upstream
+retrieval failure, since the model computes correctly over the wrong subset of evidence. LVLMs
+and memory agents fail differently. LVLMs lose visual evidence as filler content accumulates in the
+context window, whereas memory agents lose it through lossy cross-modal compression at storage
+time. As memory grows, near-misses at 32K shift toward total misses at 128K (Appendix G.4) [75],
+suggesting that growth weakens evidence retrieval rather than reasoning. Future memory designs
+should therefore focus more on cross-modal evidence-retrieval fidelity instead of solely reasoning
+improvements.
+
+
+                                                    9
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_10.txt

@@ -0,0 +1,56 @@
+5   Conclusion
+We introduced M EM L ENS, the first benchmark for multimodal conversational memory to evaluate
+LVLMs and memory-augmented agents under a unified, length-controlled protocol (32K–256K
+tokens; five memory abilities). The task remains challenging: even the strongest LVLM reaches
+only 58.68% at 32K. LVLMs degrade sharply once input contexts grow, while memory agents stay
+length-stable but discard fine-grained visual evidence at storage time, and post-training on memory
+agent backbones can erode the agents’ abstention behavior. The five abilities show low cross-type
+correlation, confirming that per-type evaluation is necessary, and nearly 90% of information-extraction
+errors stem from visual perception rather than comprehension, indicating that scaling first harms
+evidence retrieval, not reasoning. Visual-evidence retention and retrieval, rather than raw scaling of
+either context or memory, therefore emerges as the principal bottleneck to address in the future.
+
+Acknowledgments and Disclosure of Funding
+The authors of this paper were supported by the National Key Research and Development Program of
+China (2025YFE0200500), the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong
+Kong, SAR, China, and the AoE (AoE/E-601/24-N), the RIF (R6021-20) and the GRF (16205322)
+from RGC of Hong Kong, SAR, China. We also thank the NVIDIA AI Technology Center (NVAITC)
+and OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. for their support and additional
+funding.
+
+Ethics Statement
+M EM L ENS is designed exclusively for evaluating multimodal long-term conversational memory
+in large vision-language models (LVLMs) and memory-augmented agents under a unified, length-
+controlled protocol. The benchmark is not intended as a training dataset for developing memory-
+equipped systems but rather as a controlled diagnostic for fair comparison across architectures. We
+release every artifact under frozen version tags so that any reported leaderboard cell remains traceable
+to the exact item set on which it was scored.
+The 4,695 source images underlying M EM L ENS are retrieved from public web image search via
+iCrawler against a non-person-centric topic ontology; candidates carrying watermarks, stock-photo
+logos, or copyright overlays are excluded at retrieval time, and the construction prompts never request
+the model to identify, name, or describe any depicted person. Author-produced artifacts (questions,
+evidence facts, prompts, and human-annotation records) are released under CC-BY-4.0 and the
+evaluation harness under MIT, while third-party images retain their original source-site licenses;
+per-image provenance metadata (source URL, retrieval timestamp, perceptual hash) accompanies the
+release, and a takedown contact in the project repository allows seven-day removal of any flagged
+image.
+Human review is conducted by project members rather than crowd-workers, so no third-party data
+collection or human-subjects protocol is involved. Three sequential rounds (Appendix E.1) audit
+cross-modal necessity at the question level, naturalness and recoverability at the session level, and a
+stratified sample of haystack sessions before any item reaches the released set.
+We caution that M EM L ENS is not intended as a training dataset: exposure of the 789 evaluation items
+as supervised data would compromise their diagnostic value, and the datasheet explicitly discourages
+such use. While the benchmark is grounded in multi-session multimodal conversation, the five
+memory abilities it isolates (information extraction, multi-session reasoning, temporal reasoning,
+knowledge update, and answer refusal) describe general functional requirements of any long-horizon
+conversational assistant. We therefore expect M EM L ENS to serve as a useful indicator of memory
+robustness in adjacent settings, such as document-grounded or voice-grounded assistants, where
+retrieval, integration, and calibrated abstention play comparable roles.
+
+Reproducibility Statement
+All closed-source LVLM evaluations and synthetic data construction were performed via the public
+APIs of OpenAI, Anthropic, and Google, with a total experimental cost of approximately 4,500 USD
+
+
+                                                  10
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_11.txt

@@ -0,0 +1,58 @@
+covering both pipeline construction and benchmarking. Open-weight LVLMs are served locally
+with vLLM v0.17–0.18 on 8×A100-80GB nodes with tensor parallelism for context windows at
+and above 128K. Per-model decoding budgets, judge protocol, retrieval depth for memory agents,
+and adapter details for the four text-only pipelines (Mem0, MemOS, MemAgent-7B, Memory-
+T1) are documented in Appendix B. The four-stage construction pipeline (multimodal session
+simulation, question construction, evidence session construction, conversation history assembly)
+is described in Section 3, with per-subtype generation routes in Appendix C.4 and full prompt
+templates in Appendix F. Quality control combines automated filtering with a three-round human
+review (Section 3.3, Appendix E.1) and an LLM-as-Judge protocol cross-validated by an independent
+judge family and a three-annotator human consensus; agreement statistics and per-question-type
+breakdowns appear in Appendix E.2. The 789-question benchmark, 4,695 source images with
+provenance metadata, evaluation harness, and prompt templates are publicly released under frozen
+version tags at https://huggingface.co/datasets/xiyuRenBill/MEMLENS (dataset) and
+https://github.com/xrenaf/MEMLENS (code) so that every reported leaderboard cell can be
+reproduced end-to-end.
+
+References
+  [1] Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complex-
+      ity. Technical report, Technical report, Bytedance, 2025. URL https://lf3-static. bytednsdoc.
+      com . . . .
+  [2] OpenAI. Openai gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267.
+  [3] OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774.
+  [4] Anthropic. Claude 3 model card. https://assets.anthropic.com/m/61e7d27f8c8f5
+      919/original/Claude-3-Model-Card.pdf, 2024. Accessed: 2026-04-30.
+  [5] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett
+      Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal
+      understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024.
+  [6] Bytedance Seed. Seed1. 8 model card: Towards generalized real-world agency. arXiv preprint
+      arXiv:2603.20633, 2026.
+  [7] Anthropic. Claude Sonnet 4.5 System Card. https://assets.anthropic.com/m/12f21
+      4efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf, 2025.
+  [8] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and
+      Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL https:
+      //arxiv.org/abs/2310.08560.
+  [9] Deshraj Yadav, Taranjeet Singh, and Prashant Srivastava. Mem0: Building production-ready
+      AI agents with scalable long-term memory, 2025. URL https://arxiv.org/abs/2504.1
+      9413.
+ [10] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag:
+      Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983, 2024.
+ [11] Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng,
+      Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, and Mark Steedman. MMLong-
+      Bench: Benchmarking long-context vision-language models effectively and thoroughly, 2025.
+      URL https://arxiv.org/abs/2505.10610.
+ [12] Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang
+      Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai,
+      Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024. URL https:
+      //arxiv.org/abs/2406.07230.
+ [13] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu
+      Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin
+      Cao, and Aixin Sun. Mmlongbench-doc: Benchmarking long-context document understanding
+      with visualizations, 2024. URL https://arxiv.org/abs/2407.01523.
+ [14] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long-
+      memeval: Benchmarking chat assistants on long-term interactive memory, 2025. URL
+      https://arxiv.org/abs/2410.10813.
+
+
+                                               11
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_12.txt

@@ -0,0 +1,57 @@
+[15] Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremen-
+     tal multi-turn interactions, 2026. URL https://arxiv.org/abs/2507.05257.
+[16] Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and
+     Yuwei Fang. Evaluating very long-term conversational memory of llm agents, 2024. URL
+     https://arxiv.org/abs/2402.17753.
+[17] Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu,
+     Hendrik Hamann, Jingrui He, and Hanghang Tong. Mem-Gallery: Benchmarking multimodal
+     long-term conversational memory for MLLM agents, 2026. URL https://arxiv.org/ab
+     s/2601.03515.
+[18] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing
+     large language models with long-term memory, 2023. URL https://arxiv.org/abs/23
+     05.10250.
+[19] Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun
+     Peng. MRAG-Bench: Vision-centric evaluation for retrieval-augmented multimodal models,
+     2025. URL https://arxiv.org/abs/2410.08182.
+[20] Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay
+     Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking
+     long-context capability of multimodal large language models, 2025. URL https://arxiv.
+     org/abs/2406.11230.
+[21] Greg Kamradt. Needle in a haystack — pressure testing LLMs. https://github.com/gka
+     mradt/LLMTest_NeedleInAHaystack, 2023. GitHub repository.
+[22] Yiming Du, Wenyu Huang, Danna Zheng, Zhaowei Wang, Sebastien Montella, Mirella Lapata,
+     Kam-Fai Wong, and Jeff Z. Pan. Rethinking memory in llm based agents: Representations,
+     operations, and emerging topics, 2025. URL https://arxiv.org/abs/2505.00675.
+[23] Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun
+     Ma, and Zhoujun Li. SCM: Enhancing large language model with self-controlled memory
+     framework, 2025. URL https://arxiv.org/abs/2304.13343.
+[24] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D.
+     Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval, 2024.
+[25] Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag:
+     Neurobiologically inspired long-term memory for large language models, 2025. URL https:
+     //arxiv.org/abs/2405.14831.
+[26] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM:
+     Agentic memory for LLM agents, 2025. URL https://arxiv.org/abs/2502.12110.
+[27] Yiming Du, Bingbing Wang, Yang He, Bin Liang, Baojun Wang, Zhongyang Li, Lin Gui,
+     Jeff Z. Pan, Ruifeng Xu, and Kam-Fai Wong. Memguide: Intent-driven memory selection for
+     goal-oriented multi-session llm agents, 2025. URL https://arxiv.org/abs/2505.20231.
+[28] Yiming Du, Baojun Wang, Yifan Xiang, Zhaowei Wang, Wenyu Huang, Boyang Xue, Bin
+     Liang, Xingshan Zeng, Fei Mi, Haoli Bai, Lifeng Shang, Jeff Z. Pan, Yuxin Jiang, and Kam-Fai
+     Wong. Memory-t1: Reinforcement learning for temporal reasoning in multi-session agents,
+     2025. URL https://arxiv.org/abs/2512.20092.
+[29] Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi,
+     Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen,
+     Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, Hongkang Yang, Zhi-Qin John
+     Xu, and Feiyu Xiong. Memos: An operating system for memory-augmented generation (mag)
+     in large language models, 2025. URL https://arxiv.org/abs/2505.22101.
+[30] Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin
+     Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. Memagent: Reshaping
+     long-context llm with multi-conv rl-based memory agent, 2025. URL https://arxiv.org/
+     abs/2507.02259.
+[31] Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi-
+     modal retrieval is what you need for multi-page multi-document understanding, 2024. URL
+     https://arxiv.org/abs/2411.04952.
+
+
+                                              12
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_13.txt

@@ -0,0 +1,58 @@
+[32] Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and
+     Pierre Colombo. Colpali: Efficient document retrieval with vision language models, 2025.
+     URL https://arxiv.org/abs/2407.01449.
+[33] Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen,
+     Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. VLM2Vec-V2:
+     Advancing multimodal embedding for videos, images, and visual documents, 2025. URL
+     https://arxiv.org/abs/2507.04590.
+[34] Junyu Feng, Binxiao Xu, Jiayi Chen, Mengyu Dai, Cenyang Wu, Haodong Li, Bohan Zeng,
+     Yunliu Xie, Hao Liang, Ming Lu, and Wentao Zhang. M2A: Multimodal memory agent
+     with dual-layer hybrid memory for long-term personalized interactions, 2026. URL https:
+     //arxiv.org/abs/2602.07624.
+[35] Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li.
+     Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory,
+     2025. URL https://arxiv.org/abs/2508.09736.
+[36] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu,
+     Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to
+     sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on
+     Computer Vision and Pattern Recognition, pages 18221–18232, 2024.
+[37] Jihyoung Jang, Minwook Bae, Minji Kim, Dilek Hakkani-Tur, and Hyounghun Kim. Enabling
+     chatbots with eyes and ears: An immersive multimodal conversation system for dynamic
+     interactions, 2025. URL https://arxiv.org/abs/2506.00421.
+[38] Bingbing Wang, Yiming Du, Bin Liang, Zhixin Bai, Min Yang, Baojun Wang, Kam-Fai
+     Wong, and Ruifeng Xu. A new formula for sticker retrieval: Reply with stickers in multi-
+     modal and multi-session conversation. Proceedings of the AAAI Conference on Artificial
+     Intelligence, 39(24):25327–25335, Apr. 2025. doi: 10.1609/aaai.v39i24.34720. URL
+     https://ojs.aaai.org/index.php/AAAI/article/view/34720.
+[39] Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented
+     generation with long-context llms. arXiv preprint arXiv:2406.15319, 2024.
+[40] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag:
+     Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International
+     Conference on Learning Representations, 2023.
+[41] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao
+     Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for
+     long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for
+     Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, 2024.
+[42] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng
+     Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning
+     on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024.
+[43] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia,
+     and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models?
+     In First Conference on Language Modeling, 2024.
+[44] Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe
+     Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effec-
+     tively and thoroughly. arXiv preprint arXiv:2410.02694, 2024.
+[45] Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng
+     Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language
+     models. In Proceedings of the 62nd Annual Meeting of the Association for Computational
+     Linguistics (Volume 1: Long Papers), pages 14388–14411, 2024.
+[46] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han,
+     Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. ∞Bench: Extending long context evaluation
+     beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for
+     Computational Linguistics (Volume 1: Long Papers), pages 15262–15277, 2024.
+[47] Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and
+     reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024.
+
+
+                                                 13
+

wizard-work/sphinx/archive-memlens-2026-10-22/page_14.txt

@@ -0,0 +1,58 @@
+[48] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin,
+     Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking
+     multi-task long video understanding, 2025. URL https://arxiv.org/abs/2406.04264.
+[49] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for
+     long-context interleaved video-language understanding, 2024. URL https://arxiv.org/
+     abs/2407.15754.
+[50] Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun
+     Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark
+     integrating understanding, reasoning, and locating. arXiv preprint arXiv:2412.18424, 2024.
+[51] Yew Ken Chia, Liying Cheng, Hou Pong Chan, CHAOQUN LIU, Maojia Song, Mahani
+     Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal
+     super-long document understanding and a retrieval-aware tuning framework. 2024.
+[52] Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman
+     Pan, Yangqiu Song, and Dong Yu. Divscene: Towards open-vocabulary object navigation with
+     large vision language models in diverse scenes. arXiv preprint arXiv:2410.02730, 2024.
+[53] Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang.
+     Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532, 2024.
+[54] Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava:
+     Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture. arXiv preprint
+     arXiv:2409.02889, 2024.
+[55] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and
+     Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large
+     language models. CoRR, 2024.
+[56] Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong
+     Wang, and Kam-Fai Wong. Perltqa: A personal long-term memory dataset for memory
+     classification, retrieval, and synthesis in question answering, 2024. URL https://arxiv.or
+     g/abs/2402.16288.
+[57] Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei
+     Xu. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on
+     Empirical Methods in Natural Language Processing, pages 8541–8565, 2024.
+[58] Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen,
+     Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say “I don’t know”.
+     In Proceedings of the 2024 Conference of the North American Chapter of the Association for
+     Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages
+     7113–7139, 2024.
+[59] Google DeepMind. Gemini 3 Pro Model Card. https://deepmind.google/models/mod
+     el-cards/gemini-3-pro/, 2026.
+[60] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton
+     Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics:
+     An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural
+     Information Processing Systems, 36:71683–71702, 2023.
+[61] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina
+     Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing
+     millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on
+     Computer Vision, pages 12065–12075, 2023.
+[62] Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi,
+     Xin Liu, and Yangqiu Song. Abspyramid: Benchmarking the abstraction ability of language
+     models with a unified entailment graph. In Findings of the Association for Computational
+     Linguistics: NAACL 2024, pages 3991–4010, 2024.
+[63] Zhaowei Wang, Wei Fan, Qing Zong, Hongming Zhang, Sehyun Choi, Tianqing Fang, Xin
+     Liu, Yangqiu Song, Ginny Wong, and Simon See. Absinstruct: Eliciting abstraction ability
+     from llms through explanation tuning with plausibility estimation. In Proceedings of the 62nd
+     Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
+     pages 973–994, 2024.
+
+
+                                              14
+

wizard-work/sphinx/archive-memlens-2026-10-22/quotes.json

@@ -0,0 +1,114 @@
+[
+ {
+  "q": "removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images",
+  "ok": true,
+  "pages": [
+   1
+  ]
+ },
+ {
+  "q": "the five memory abilities are largely independent: strong information extraction does not predict multi-session reasoning, and multi-session reasoning caps most evaluated systems below 30%",
+  "ok": true,
+  "pages": [
+   2
+  ]
+ },
+ {
+  "q": "long-context LVLMs deliver high short-context accuracy through direct visual grounding, but this advantage shrinks as conversations grow",
+  "ok": true,
+  "pages": [
+   2
+  ]
+ },
+ {
+  "q": "neither approach comes close to solving the task of long-term memory",
+  "ok": true,
+  "pages": [
+   2
+  ]
+ },
+ {
+  "q": "789 questions instantiated at four standardized input lengths",
+  "ok": true,
+  "pages": [
+   3
+  ]
+ },
+ {
+  "q": "Evidence sessions",
+  "ok": true,
+  "pages": [
+   5
+  ]
+ },
+ {
+  "q": "We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS",
+  "ok": true,
+  "pages": [
+   6
+  ]
+ },
+ {
+  "q": "65.7% of questions are image-essential",
+  "ok": true,
+  "pages": [
+   6
+  ]
+ },
+ {
+  "q": "GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%",
+  "ok": true,
+  "pages": [
+   6
+  ]
+ },
+ {
+  "q": "At 32K, the top eight LVLMs fall within a 6.34% band",
+  "ok": true,
+  "pages": [
+   7
+  ]
+ },
+ {
+  "q": "Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% drop)",
+  "ok": true,
+  "pages": [
+   7
+  ]
+ },
+ {
+  "q": "Per-type accuracy ceilings range from 97.78% on AR down to 44.06% on MSR (Kimi-K2.5) at 32K",
+  "ok": true,
+  "pages": [
+   7
+  ]
+ },
+ {
+  "q": "M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score 81.82% under direct inference",
+  "ok": true,
+  "pages": [
+   9
+  ]
+ },
+ {
+  "q": "nearly 90% of information-extraction errors stem from visual perception rather than comprehension",
+  "ok": true,
+  "pages": [
+   10
+  ]
+ },
+ {
+  "q": "even the strongest LVLM reaches only 58.68% at 32K",
+  "ok": true,
+  "pages": [
+   10
+  ]
+ },
+ {
+  "q": "The benchmark is not intended as a training dataset for developing memory-equipped systems",
+  "ok": true,
+  "pages": [
+   10
+  ]
+ }
+]

wizard-work/sphinx/archive-memlens-2026-10-22/verify_memlens_archive.py

@@ -0,0 +1,157 @@
+#!/usr/bin/env python3
+"""Verify the MemLens archive: every quotation must be verbatim in the archived paper.
+
+Exit code 0 = archive is sound (all quotations real, hashes intact, metadata matches).
+Exit code nonzero = something in this archive is wrong or fabricated.
+"""
+import hashlib
+import json
+import os
+import re
+import sys
+import xml.etree.ElementTree as ET
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+EXPECTED_ARXIV_ID = "2605.14906v1"
+EXPECTED_TITLE = "MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models"
+REQUIRED_SECTIONS = [
+    "## 1. Identity",
+    "## 2. Provenance",
+    "## 3. One-line verdict",
+    "## 4. What the paper contributes",
+    "## 5. Cross-modality validation",
+    "## 6. Key findings",
+    "## 7. Verified quotations",
+    "## 8. Caveats and limits",
+    "## 9. Verification",
+]
+N_PAGES = 14
+
+fails = []
+checks = 0
+
+
+def ok(msg):
+    global checks
+    checks += 1
+    print("  PASS  %s" % msg)
+
+
+def bad(msg):
+    global checks
+    checks += 1
+    fails.append(msg)
+    print("  FAIL  %s" % msg)
+
+
+def norm(s):
+    s = s.replace("\u00ad", "")
+    s = re.sub(r"-\s*\n\s*", "-", s)   # undo hyphenated line breaks
+    s = re.sub(r"\s+", " ", s)
+    return s.strip()
+
+
+def read(name):
+    with open(os.path.join(HERE, name), encoding="utf-8") as fh:
+        return fh.read()
+
+
+print("== 1. Quotations are verbatim in the archived paper ==")
+quotes = json.loads(read("quotes.json"))
+if len(quotes) != 16:
+    bad("expected 16 quotations in quotes.json, found %d" % len(quotes))
+else:
+    ok("quotes.json holds 16 quotations")
+
+pages = {n: norm(read("page_%02d.txt" % n)) for n in range(1, N_PAGES + 1)}
+full_text = norm(read("memlens.txt"))
+
+for entry in quotes:
+    q = entry["q"]
+    nq = norm(q)
+    label = q[:58].replace("\n", " ")
+    if nq not in full_text:
+        bad("QUOTE NOT IN PAPER TEXT: %s" % label)
+        continue
+    on_pages = [n for n, t in sorted(pages.items()) if nq in t]
+    if not on_pages:
+        bad("QUOTE NOT FOUND ON ANY PAGE: %s" % label)
+        continue
+    claimed = entry.get("pages") or []
+    if claimed and claimed != on_pages:
+        bad("PAGE MISMATCH for %s: claimed %s, actual %s" % (label, claimed, on_pages))
+        continue
+    ok("p%s  %s" % (on_pages, label))
+
+print("== 2. SHA256SUMS matches archived files ==")
+for line in read("SHA256SUMS").splitlines():
+    line = line.strip()
+    if not line:
+        continue
+    digest, name = line.split(None, 1)
+    name = name.strip()
+    path = os.path.join(HERE, name)
+    if not os.path.exists(path):
+        bad("missing archived file %s" % name)
+        continue
+    actual = hashlib.sha256(open(path, "rb").read()).hexdigest()
+    if actual == digest:
+        ok("%s  %s" % (digest[:16], name))
+    else:
+        bad("HASH MISMATCH %s: recorded %s, actual %s" % (name, digest[:16], actual[:16]))
+
+print("== 3. arXiv metadata ==")
+root = ET.parse(os.path.join(HERE, "metadata.xml")).getroot()
+ns = {"a": "http://www.w3.org/2005/Atom"}
+entry = root.find("a:entry", ns)
+meta_id = entry.find("a:id", ns).text.strip().split("/abs/")[-1]
+meta_title = " ".join(entry.find("a:title", ns).text.split())
+if meta_id == EXPECTED_ARXIV_ID:
+    ok("arXiv id %s" % meta_id)
+else:
+    bad("arXiv id mismatch: %s" % meta_id)
+if meta_title == EXPECTED_TITLE:
+    ok("title matches")
+else:
+    bad("title mismatch: %r" % meta_title)
+n_authors = len(entry.findall("a:author", ns))
+if n_authors == 14:
+    ok("14 authors")
+else:
+    bad("author count %d, expected 14" % n_authors)
+
+print("== 4. Research note structure ==")
+import glob
+notes = sorted(glob.glob(os.path.join(HERE, "REPORT-*.md")))
+if not notes:
+    notes = [os.path.join(HERE, "README.md")]
+note_path = notes[0]
+readme = read(os.path.relpath(note_path, HERE))
+print("  note file: %s" % os.path.basename(note_path))
+for sec in REQUIRED_SECTIONS:
+    if sec in readme:
+        ok("section present: %s" % sec)
+    else:
+        bad("section MISSING: %s" % sec)
+if "16/16" in readme:
+    ok("note records the 16/16 quotation count")
+else:
+    bad("note does not record the 16/16 count")
+
+print("== 5. Source PDF is intact ==")
+pdf = os.path.join(HERE, "memlens.pdf")
+head = open(pdf, "rb").read(8)
+if head.startswith(b"%PDF-"):
+    ok("memlens.pdf has a valid PDF header (%d bytes)" % os.path.getsize(pdf))
+else:
+    bad("memlens.pdf is not a PDF (header %r)" % head)
+
+print()
+print("checks run: %d   failures: %d" % (checks, len(fails)))
+if fails:
+    print("VERDICT: FAIL")
+    for f in fails:
+        print("  - %s" % f)
+    sys.exit(1)
+print("VERDICT: PASS — all quotations verbatim, hashes and metadata verified")
+sys.exit(0)