diff --git a/wizard-work/sphinx/README-archive-lane.md b/wizard-work/sphinx/README-archive-lane.md index bac18b6..490a10a 100644 --- a/wizard-work/sphinx/README-archive-lane.md +++ b/wizard-work/sphinx/README-archive-lane.md @@ -1,6 +1,32 @@ # sphinx — Archive lane work (multimodal, long-context research) -## Latest: `archive-mm-prm-2026-10-22` +## Latest: `archive-memlens-2026-10-22` + +Verifiable archive of arXiv 2605.14906v1 (*MemLens*), a multimodal long-term +conversational-memory benchmark — 789 questions, five memory abilities, 32K–256K +token lengths, evaluated on 27 LVLMs and 7 memory-augmented agents. + +Artifacts: +- `archive-memlens-2026-10-22/memlens.pdf` — the paper, 14 pages (sha256 recorded) +- `archive-memlens-2026-10-22/memlens.txt` + `page_01..14.txt` — full text and per-page text +- `archive-memlens-2026-10-22/metadata.xml` — raw arXiv Atom API response +- `archive-memlens-2026-10-22/quotes.json` — 16 anchor quotations + the page each was found on +- `archive-memlens-2026-10-22/build_memlens_archive.py` — stdlib-only builder (re-fetch + re-extract) +- `archive-memlens-2026-10-22/verify_memlens_archive.py` — offline verifier (exit 0 = correct) +- `archive-memlens-2026-10-22/REPORT-memlens-archive-2026-10-22.md` — the research note + +Grounding test: every one of the 16 quotations in the note is matched verbatim +(whitespace-normalised) against the extracted page text, and the page number cited must +actually contain that quotation. 16/16 pass; 33 checks, 0 failures. Negative control: +injecting one fabricated quotation makes the verifier exit 1 and print +`QUOTE NOT IN PAPER TEXT`. Reproduction re-ran the builder from scratch and reproduced +byte-identical digests (`b92a5b6d…` PDF, `628bd8d6…` text). + +Reproduce: + python3 build_memlens_archive.py + python3 verify_memlens_archive.py # exit 0 = correct + +## Previous: `archive-mm-prm-2026-10-22` Verifiable multimodal/long-context archive of arXiv 2505.13427 (*MM-PRM*), with a machine-checkable grounding test over an 18-chunk index. diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/REPORT-memlens-archive-2026-10-22.md b/wizard-work/sphinx/archive-memlens-2026-10-22/REPORT-memlens-archive-2026-10-22.md new file mode 100644 index 0000000..8b9cea3 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/REPORT-memlens-archive-2026-10-22.md @@ -0,0 +1,184 @@ +# REPORT — MemLens archive (arXiv 2605.14906v1) + +Lane: Archive (multimodal, long-context research) +Date: 2026-10-22 (UTC) · Archivist: sphinx (Hermes Agent) + +--- + +## 1. Identity (MEASURED — arXiv Atom API, saved as `metadata.xml`) + +| Field | Value | +|---|---| +| Title | MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models | +| arXiv | 2605.14906v1 [cs.CV] | +| Submitted | 2026-05-14 | +| Comment field | "Work in progress" | +| Authors | Xiyu Ren, Zhaowei Wang, Yiming Du, Zhongwei Xie, Chi Liu, Xinlin Yang, Haoyue Feng, Wenjun Pan, Tianshi Zheng, Baixuan Xu, Zhengnan Li, Yangqiu Song, Ginny Wong, Simon See (14) | +| Code | https://github.com/xrenaf/MEMLENS | +| Affiliations | HKUST CSE; CUHK; OmniMemory (Shenzhen); NVIDIA AI Technology Center | + +## 2. Provenance (MEASURED) + +| File | What | sha256 | +|---|---|---| +| `memlens.pdf` | Paper as published, 14 pages, PDF v1.7, 2,627,199 B | `b92a5b6d89999ba6ad77fbc32f8f6e9e86a39a49dfab0a8c9ee310edf380affd` | +| `memlens.txt` | Full text, `pdftotext -layout`, 293,612 B | `628bd8d62beb83c57b0dfe7792d3c1b25ed480adf04e37a8e93aa235f44c2bdb` | +| `page_01.txt` … `page_14.txt` | Per-page text (page-anchored citation) | — | +| `metadata.xml` | Raw arXiv API response | — | +| `quotes.json` | 16 quotations + the page each was found on | — | +| `SHA256SUMS` | Digests for the two large files | — | + +**Reproduction (MEASURED):** `build_memlens_archive.py` was re-run from scratch against the +live arXiv endpoints into a clean directory; it re-downloaded the PDF and re-extracted the text, +and produced byte-identical digests (`b92a5b6d…` / `628bd8d6…`). The archive is not a +hand-assembled copy — it rebuilds. + +**Every quotation in §7 was matched programmatically against the extracted page text +(whitespace-normalised) before being written down: 16/16 found. The check re-runs offline — see §9.** + +--- + +## 3. One-line verdict + +MemLens is a 789-question, length-controlled (32K–256K token) multimodal conversational-memory +benchmark — the first to put long-context vision-language models and memory-augmented agents on +the same questions — and its central measured result is that the task is nowhere near solved: +the best LVLM reaches 58.68% at 32K, multi-session reasoning caps most systems below 30%, and +nearly 90% of information-extraction errors come from visual perception failure rather than reasoning. + +## 4. What the paper contributes + +- **A benchmark, not a model.** It evaluates 27 LVLMs and 7 memory-augmented agents. +- **Cross-modal necessity enforced by construction.** Evidence facts are abstracted in the text + (e.g. "Golden Gate Bridge" → "the bridge shown in "), so only the image resolves the + reference; removing evidence images drops two frontier LVLMs below 2% accuracy. +- **Length control.** Four standard lengths (32K/64K/128K/256K tokens) under the MMLongBench + cross-modal token-counting scheme, so LVLMs and agents can be compared at equal nominal length. +- **Five memory abilities** — Information Extraction (IE), Multi-Session Reasoning (MSR), + Temporal Reasoning (TR), Knowledge Update (KU), Answer Refusal (AR) — across 9 subtypes. + +### Dataset statistics (Table 2, p.5 — MEASURED) + +| Statistic | Value | +|---|---| +| Questions | 789 | +| Types / Subtypes | 5 / 9 | +| Evidence sessions | 2,145 | +| Avg. turns/session | ~10 | +| Avg. images/session | ~1.5 | +| Tokens/image | ~2,000 | +| Sessions/instance | 14 → 93 | +| Images/instance | 20 → 138 | + +The paper also states 4,695 unique source images are released (p.10), and that an initial pool of +20k candidates was reduced to 789 by automated filters plus three rounds of human review (p.5). + +## 5. Cross-modality validation (p.6 — MEASURED) + +Image-necessity split: 65.7% image-essential, 14.7% image-supportive, 19.6% text-sufficient. + +Answerability test (n = 634, evidence supplied): GPT-5.4 reaches 93.13% overall, Gemini-3.1-Pro 89.42%. + +Ablation (evidence images removed): overall accuracy collapses below 2% for both models — +GPT-5.4 to 1.74% (Δ −91.39), Gemini-3.1-Pro to 1.89% (Δ −87.53), per Table 3. + +## 6. Key findings + +### 6.1 Headline findings (p.2 — quoted in §7) + +1. The five memory abilities are largely independent; multi-session reasoning caps most systems below 30%. +2. The two approaches fail in complementary ways: LVLMs win short context and degrade as it grows; + memory agents are length-stable but lose visual fidelity through storage-time compression. +3. Neither approach comes close to solving long-term memory. + +### 6.2 Main results (p.7 — MEASURED numbers) + +- At 32K the top eight LVLMs fall within a **6.34% band** — short-context accuracy no longer separates frontier systems. +- At 128K the picture inverts: several open-weight leaders lose more than 13%, while + **Gemini-3.1-Pro retains 51.99% and degrades least overall (a 2.11% drop)**. +- Per-type ceilings range from **97.78% on AR** down to **44.06% on MSR (Kimi-K2.5)** at 32K. +- Only Kimi-K2.5 (44.06%) and Gemini-3.1-Pro (32.17%) clear 30% on MSR — the shared capability ceiling. +- Answer refusal is the ability most exposed to context growth: GLM-4.6V AR falls 93.33% (32K) → 30.00% (128K). +- Memory agents trail LVLMs on nearly all types, with the largest gaps on IE, KU and AR. + +### 6.3 Analysis (p.8–10 — MEASURED numbers) + +- **Agents lose visual faithfulness.** Both text-only and multimodal pipelines compress image + evidence at storage time; the gap persists when BLIP captions replace raw images, so the + bottleneck is lossy cross-modality storage, not the answering model. +- **Post-training on agent backbones weakens abstention.** Frozen-backbone Mem0 (77.27%) and + MemOS (68.18%) keep much of the base model's refusal behaviour; the five backbone-finetuned + agents (M2A, M3-Agent, M3C, MemAgent-7B, Memory-T1) collapse to 9–22% AR. M2A reaches only + 22.73% on the same Qwen3-VL-8B backbone that scores 81.82% under direct inference. +- **Complementary scaling.** Six of seven agents stay within ±7% from 32K to 256K + (retrieve-then-reason is length-invariant); LVLMs degrade substantially (IE ~−20%, KU ~−12%, + AR ~75% at 32K → ~45% at 128K). MSR's apparent flatness is a floor effect near 30%. +- **Error decomposition at 128K.** Nearly 90% of IE and KU errors are in the "Visual" category; + MSR is the only type whose errors are dominated by "Reasoning" (73%). An oracle-retrieval + diagnostic restores MSR accuracy to 90–100% when only the required sessions are supplied, + indicating long-context reasoning failure is largely induced by upstream retrieval failure. +- **Correlations.** IE–KU is the strongest type pair at 32K (ρ = 0.65); MSR correlates weakly with + IE and AR, supporting an evidence-retrieval axis (IE, KU) versus an aggregation axis (MSR). + +### 6.4 Conclusion (p.10 — quoted in §7) + +"even the strongest LVLM reaches only 58.68% at 32K". The paper closes on visual-evidence +retention and retrieval — not raw scaling of context or memory — as the principal bottleneck. + +## 7. Verified quotations (16/16 matched against the extracted page text) + +| # | Page | Quotation | +|---|---|---| +| Q1 | 1 | "removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images" | +| Q2 | 2 | "the five memory abilities are largely independent: strong information extraction does not predict multi-session reasoning, and multi-session reasoning caps most evaluated systems below 30%" | +| Q3 | 2 | "long-context LVLMs deliver high short-context accuracy through direct visual grounding, but this advantage shrinks as conversations grow" | +| Q4 | 2 | "neither approach comes close to solving the task of long-term memory" | +| Q5 | 3 | "789 questions instantiated at four standardized input lengths" | +| Q6 | 5 | "Evidence sessions" (Table 2 row label, value 2,145) | +| Q7 | 6 | "We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS" | +| Q8 | 6 | "65.7% of questions are image-essential" | +| Q9 | 6 | "GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%" | +| Q10 | 7 | "At 32K, the top eight LVLMs fall within a 6.34% band" | +| Q11 | 7 | "Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% drop)" | +| Q12 | 7 | "Per-type accuracy ceilings range from 97.78% on AR down to 44.06% on MSR (Kimi-K2.5) at 32K" | +| Q13 | 9 | "M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score 81.82% under direct inference" | +| Q14 | 10 | "nearly 90% of information-extraction errors stem from visual perception rather than comprehension" | +| Q15 | 10 | "even the strongest LVLM reaches only 58.68% at 32K" | +| Q16 | 10 | "The benchmark is not intended as a training dataset for developing memory-equipped systems" | + +*(Q7/Q16 reproduce the paper's own typography, including the spaced small-caps "M EM L ENS" +and the original "that score" agreement; both are verbatim.)* + +## 8. Caveats and limits (CLAIMED — my reading, not the paper's words) + +- The arXiv comment field says "Work in progress"; no peer-review venue is stated. +- LVLMs are evaluated at only three lengths (32K/64K/128K) because many do not natively support + 256K; agents get all four. Overall numbers are therefore not a clean 4×4 matrix. +- Agents are scored on a 195-question canonical subset (a quarter of the dataset) while LVLMs are + scored on all 789 — cross-family per-system comparison inherits that asymmetry. +- Human review was conducted by project members, not crowd-workers. +- Scoring is LLM-as-Judge (Qwen3-VL-235B-A22B-Instruct), cross-validated by GPT-5.4-mini + re-judging (κ = 0.93) and a three-annotator human consensus. +- The benchmark is explicitly released as a diagnostic, not training data. + +## 9. Verification (offline, deterministic) + + python3 verify_memlens_archive.py + +Exit 0 only if **all** hold (33 checks in the passing run): + +1. All 16 quotations in `quotes.json` are present, verbatim (whitespace-normalised), in the + archived page text — i.e. no quotation in this report is invented. +2. Every quoted page number actually contains that quotation. +3. `SHA256SUMS` matches the archived PDF and text. +4. `metadata.xml` carries the expected arXiv ID, title and 14 authors. +5. This report contains all required section headers and records the 16/16 count. +6. `memlens.pdf` has a valid PDF header. + +**Negative control (MEASURED):** a copy of the archive with one fabricated quotation injected +("MemLens solves multi-session reasoning with 99.9% accuracy") makes the verifier exit **1** and +print `QUOTE NOT IN PAPER TEXT`. The check is a real test, not a tautology. + +Reproduce from source: + + python3 build_memlens_archive.py # exit 0 = built, 16/16 quotes matched diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/SHA256SUMS b/wizard-work/sphinx/archive-memlens-2026-10-22/SHA256SUMS new file mode 100644 index 0000000..5355109 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/SHA256SUMS @@ -0,0 +1,2 @@ +b92a5b6d89999ba6ad77fbc32f8f6e9e86a39a49dfab0a8c9ee310edf380affd memlens.pdf +628bd8d62beb83c57b0dfe7792d3c1b25ed480adf04e37a8e93aa235f44c2bdb memlens.txt diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/build_memlens_archive.py b/wizard-work/sphinx/archive-memlens-2026-10-22/build_memlens_archive.py new file mode 100644 index 0000000..222e227 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/build_memlens_archive.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Build the MemLens archive (arXiv 2605.14906v1) from scratch. + +Usage: python3 build_memlens_archive.py + +Steps (all network + local, no external deps): + 1. fetch the arXiv Atom metadata for the paper id + 2. fetch the paper PDF + 3. extract full text and per-page text with pdftotext + 4. match the 16 anchor quotations against the extracted text -> quotes.json + 5. write SHA256SUMS for the large artifacts + +Exit 0 = archive built and every anchor quotation found. +""" +import hashlib +import json +import os +import re +import subprocess +import sys +import urllib.request + +ARXIV_ID = "2605.14906v1" +UA = "Mozilla/5.0" + +ANCHOR_QUOTES = [ + "removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images", + "the five memory abilities are largely independent: strong information extraction does not predict multi-session reasoning, and multi-session reasoning caps most evaluated systems below 30%", + "long-context LVLMs deliver high short-context accuracy through direct visual grounding, but this advantage shrinks as conversations grow", + "neither approach comes close to solving the task of long-term memory", + "789 questions instantiated at four standardized input lengths", + "Evidence sessions", + "We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS", + "65.7% of questions are image-essential", + "GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%", + "At 32K, the top eight LVLMs fall within a 6.34% band", + "Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% drop)", + "Per-type accuracy ceilings range from 97.78% on AR down to 44.06% on MSR (Kimi-K2.5) at 32K", + "M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score 81.82% under direct inference", + "nearly 90% of information-extraction errors stem from visual perception rather than comprehension", + "even the strongest LVLM reaches only 58.68% at 32K", + "The benchmark is not intended as a training dataset for developing memory-equipped systems", +] + + +def norm(s): + s = s.replace("\u00ad", "") + s = re.sub(r"-\s*\n\s*", "-", s) + s = re.sub(r"\s+", " ", s) + return s.strip() + + +def fetch(url, dest): + req = urllib.request.Request(url, headers={"User-Agent": UA}) + with urllib.request.urlopen(req, timeout=120) as resp, open(dest, "wb") as fh: + fh.write(resp.read()) + return os.path.getsize(dest) + + +def pdftotext(args, pdf): + return subprocess.run(["pdftotext"] + args + [pdf, "-"], capture_output=True, text=True).stdout + + +def main(): + out = sys.argv[1] if len(sys.argv) > 1 else "memlens-archive" + os.makedirs(out, exist_ok=True) + + meta = os.path.join(out, "metadata.xml") + pdf = os.path.join(out, "memlens.pdf") + txt = os.path.join(out, "memlens.txt") + + print("[1/5] arXiv metadata -> %s" % meta) + n = fetch("https://export.arxiv.org/api/query?id_list=%s" % ARXIV_ID, meta) + print(" %d bytes" % n) + + print("[2/5] PDF -> %s" % pdf) + n = fetch("https://arxiv.org/pdf/%s" % ARXIV_ID, pdf) + print(" %d bytes" % n) + head = open(pdf, "rb").read(5) + if head != b"%PDF-": + print("FATAL: downloaded file is not a PDF", file=sys.stderr) + return 1 + + print("[3/5] text extraction") + open(txt, "w").write(pdftotext(["-layout"], pdf)) + raw = open(txt, encoding="utf-8").read() + pages = {} + for i in range(1, 15): + pages[i] = pdftotext(["-layout", "-f", str(i), "-l", str(i)], pdf) + open(os.path.join(out, "page_%02d.txt" % i), "w").write(pages[i]) + print(" %d chars full text, 14 page files" % len(raw)) + + print("[4/5] anchor quotation match") + nraw, npages = norm(raw), {k: norm(v) for k, v in pages.items()} + results, misses = [], 0 + for q in ANCHOR_QUOTES: + nq = norm(q) + if nq not in nraw: + misses += 1 + results.append({"q": q, "ok": False}) + print(" MISS: %s" % q[:70]) + continue + hits = [k for k in sorted(npages) if nq in npages[k]] + results.append({"q": q, "ok": True, "pages": hits}) + print(" OK p%s: %s" % (hits, q[:55])) + json.dump(results, open(os.path.join(out, "quotes.json"), "w"), indent=1) + + print("[5/5] SHA256SUMS") + lines = [] + for name in ("memlens.pdf", "memlens.txt"): + h = hashlib.sha256(open(os.path.join(out, name), "rb").read()).hexdigest() + lines.append("%s %s" % (h, name)) + open(os.path.join(out, "SHA256SUMS"), "w").write("\n".join(lines) + "\n") + for line in lines: + print(" " + line) + + found = sum(1 for r in results if r["ok"]) + print("\n%d/%d anchor quotations found" % (found, len(ANCHOR_QUOTES))) + return 1 if misses else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/memlens.pdf b/wizard-work/sphinx/archive-memlens-2026-10-22/memlens.pdf new file mode 100644 index 0000000..c430c75 Binary files /dev/null and b/wizard-work/sphinx/archive-memlens-2026-10-22/memlens.pdf differ diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/memlens.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/memlens.txt new file mode 100644 index 0000000..4e6a76c --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/memlens.txt @@ -0,0 +1,4332 @@ + M EM L ENS: Benchmarking Multimodal Long-Term + Memory in Large Vision-Language Models + + + Xiyu Ren1 Zhaowei Wang1 Yiming Du2 Zhongwei Xie1 + Chi Liu1 Xinlin Yang1 Haoyue Feng1 Wenjun Pan1 Tianshi Zheng1 + Baixuan Xu1 Zhengnan Li3 Yangqiu Song1 Ginny Wong4 Simon See4 +arXiv:2605.14906v1 [cs.CV] 14 May 2026 + + + + + 1 + CSE Deparment, HKUST 2 CUHK + 3 + OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. + 4 + NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA + {xrenaf, zwanggy, yqsong}@cse.ust.hk + ydu@se.cuhk.edu.hk lzhengnan389@gmail.com + + + + Abstract + + Memory is essential for large vision-language models (LVLMs) to handle long, + multimodal interactions, with two method directions providing this capability: long- + context LVLMs and memory-augmented agents. However, no existing benchmark + conducts a systematic comparison of the two on questions that genuinely require + multimodal evidence. To close this gap, we introduce M EM L ENS, a comprehensive + benchmark for memory in multimodal multi-session conversations, comprising + 789 questions across five memory abilities (information extraction, multi-session + reasoning, temporal reasoning, knowledge update, and answer refusal) at four + standard context lengths (32K–256K tokens) under a cross-modal token-counting + scheme. An image-ablation study confirms that solving M EM L ENS requires vi- + sual evidence: removing evidence images drops two frontier LVLMs below 2% + accuracy on the 80.4% of questions whose evidence includes images. Evaluating + 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs + achieve high short-context accuracy through direct visual grounding but degrade + as conversations grow, whereas memory agents are length-stable but lose visual fi- + delity under storage-time compression. Multi-session reasoning caps most systems + below 30%, and neither approach alone solves the task. These results motivate hy- + brid architectures that combine long-context attention with structured multimodal + retrieval. Our code is available at https://github.com/xrenaf/MEMLENS. + + + + 1 Introduction + + Memory is essential for enabling large vision-language models (LVLMs) [1, 2] to maintain con- + sistency across extended multimodal interactions and to continuously incorporate new information + over time [3–5]. When LVLMs are deployed as agents, the inputs they handle accumulate incre- + mentally rather than arriving all at once. As interaction histories expand, the agent must recall past + content and reason over the growing context to remain consistent with prior turns. At the same time, + it must integrate new facts as they arrive and revise outdated information to keep its knowledge + current. Two directions have emerged to provide this capability, namely long-context LVLMs and + memory-augmented agents. Long-context LVLMs enlarge the native context window so that the + complete dialogue history, including interleaved images, can be processed directly by the model [5–7]. + Memory-augmented agents, building on retrieval-augmented generation, instead compress, index, + and selectively retrieve past content from an external store [8–10]. + Table 1: Left: Comparison of M EM L ENS with existing long-context and conversational memory +benchmarks. Type — Dial (Dialogue) or Doc (Document); Max L — maximum context length +reported (“—” if not applicable); L Control — standardized token length control; Comp. Eval. — +paper benchmarks both LVLMs and memory-augmented agents on the same data; Memory Tasks — +dot pattern showing coverage of the five evaluation tasks (• = supported, ◦ = not supported) in fixed +order: IE (Information Extraction), MSR (Multi-Session Reasoning), TR (Temporal Reasoning), +KU (Knowledge Update), AR (Answer Refusal). Right: Distribution of the 789 evaluation questions +in M EM L ENS across the five major task types (inner ring) and nine reporting subtypes (outer ring). + + Max Multi- Multi- L Comp. Memory +Benchmark Type Refuse Entity + L modal Sess. Control Eval. Tasks 11.4% 15.2% +MemoryBank [18] Dial — ✗ ✓ ✗ ✗ •◦◦◦◦ AR +LoCoMo [16] Dial 10K ✓ ✓ ✗ ✗ •••◦◦ Update 11.4% + 14.7% IE +MM-NIAH [12] Doc 128K ✓ ✗ ✓ ✗ •◦◦◦◦ KU 31.2% + 14.7% PrevInfo + 16.0% +MMLongBench-Doc [13] Doc 128K ✓ ✗ ✗ ✗ •◦◦◦◦ +MRAG-Bench [19] Doc — ✓ ✗ ✗ ✗ •◦◦◦◦ + Count. +LongMemEval [14] Dial 1.5M ✗ ✓ ✓ ✗ ••••• 5.8% MSR +MMLongBench [11] Doc 128K ✓ ✗ ✓ ✗ •◦◦◦◦ 18.1% TR + Ent. Res. 24.6% Temp. Gnd. +Multimodal NIAH [20] Doc 128K ✓ ✗ ✓ ✗ •◦◦◦◦ 6.0% + Arith. 13.1% +MemAgentBench [15] Dial 115K ✗ ✓ ✗ ✓ •◦◦◦◦ 6.3% Dur. Cmp. + 11.5% +M EM L ENS (Ours) Dial 256K ✓ ✓ ✓ ✓ ••••• + + + + +Despite progress along both directions, no current benchmark conducts a length-controlled compari- +son of long-context LVLMs and memory-augmented agents on questions that genuinely require visual +evidence. Long-context multimodal benchmarks measure context-length scaling on long documents +and retrieval tasks [11–13], but cover mainly LVLMs and do not place them alongside memory- +augmented agents for direct comparison. Text-only conversational memory benchmarks such as +LongMemEval [14] and MemoryAgentBench [15] overlook the visual modality entirely, treating +memory as a single-modality problem. Multimodal conversational benchmarks such as LoCoMo [16] +and Mem-Gallery [17] retain both visual and text modalities, yet most of their questions admit a +text-only shortcut, rendering the visual modality redundant. As Table 1 summarizes, no existing +benchmark requires visual evidence to answer its questions while supporting a length-controlled +comparison of long-context LVLMs and memory-augmented agents. +To bridge the gap, we introduce M EM L ENS, a comprehensive benchmark for assessing memory in +multimodal multi-session conversations. M EM L ENS consists of 789 questions covering five core +memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge +update, and answer refusal. We design questions that demand cross-modal reasoning over both +visual and textual evidence, requiring the retrieval of multimodal information hidden within one +or more conversations between a user and an assistant. To verify that solving M EM L ENS requires +visual evidence, we conduct an image-ablation study on the 80.4% of questions whose evidence +includes images: when these images are removed, the accuracy of two frontier LVLMs collapses +below 2% (§3.4). Following the needle-in-a-haystack construction paradigm [21], we design a +four-stage pipeline that builds a coherent multi-session chat history for each question, distributing +the evidence across one or more user-assistant sessions alongside topically related distractor turns. +Because distractor turns can be added independently of the evidence, the chat length is freely +extensible. We release four standard context lengths (32K, 64K, 128K, and 256K tokens) under a +cross-modal token-counting scheme [11] that aligns text and vision tokens. M EM L ENS thus enables a +length-controlled comparison of long-context LVLMs and memory-augmented agents on multimodal +questions spanning five comprehensive memory abilities. +Using M EM L ENS, we evaluate 27 LVLMs and 7 memory-augmented agents across all four context +lengths (32K, 64K, 128K, and 256K). Our evaluation yields three key findings. First, the five +memory abilities are largely independent: strong information extraction does not predict multi- +session reasoning, and multi-session reasoning caps most evaluated systems below 30%. Second, +the two approaches exhibit complementary failure modes: long-context LVLMs deliver high short- +context accuracy through direct visual grounding, but this advantage shrinks as conversations grow; +memory agents, in contrast, are length-stable but lose visual fidelity under storage-time compression, +and memory-oriented post-training of agent backbones can additionally weaken their abstention +behavior. Third, neither approach comes close to solving the task of long-term memory. These + + + 2 + results point to a clear next step: architectures that combine long-context attention with structured +multimodal retrieval, rather than scaling either component in isolation. + +2 Related Work +Memory-Augmented LLM Agents. Recent surveys systematize memory representations and oper- +ations across LLM-based agents [22]. Text-only memory agents span structured-fact stores [18, 23], +OS-inspired paging [8], tree-summarized retrieval [24], neurobiological graphs [25], relational em- +beddings [9], agentic self-organizing notes [26], intent-driven memory selection [27], RL-selected +time-aware memory [28], layered memory tiers [29], and sliding-window RL agents [30]. Mul- +timodal extensions add ColPali-style document retrieval [31, 32], multimodal embeddings [33], +dual-layer semantic memory [34], entity-centric audio-visual memory [35], sparse video mem- +ory [36], LoRA-tuned session retrieval [37], and intent-guided multimodal response retrieval over +multi-session conversations [38]. Recent work probes the intersection of long-context LLMs and +retrieval-augmented generation [10, 39, 40], suggesting retrieval and long attention are complemen- +tary rather than competing. + +Long-Context and Conversational Memory Benchmarks. Most current long-context benchmarks +established core protocols for retrieval and length scaling [41–47], but neglect the visual modality. +Multimodal extensions, such as MMLongBench [11], scale context length over documents [11, 13], +needle-style retrieval [12, 20], long-form video [48, 49], super-long documents [50, 51], and multi- +image inputs [52–55]. Yet their inputs are documents or videos rather than multi-session conver- +sations. Memory under conversational interaction therefore remains unexercised. Conversational +memory benchmarks restore the multi-session structure but rarely require visual perception: Long- +MemEval [14] (500 questions over five memory abilities, up to 1.5M tokens), PerLTQA [56], +and MemoryAgentBench [15] (retrieval, test-time learning, long-range understanding, selective +forgetting) discard images entirely, while LoCoMo [16] and Mem-Gallery [17] embed images in +persona-grounded dialogue but allow most questions to be answered from text alone, so neither +stress-tests multimodal memory. + +3 The M EM L ENS Benchmark +We propose M EM L ENS, a multimodal long-term conversational memory benchmark that comprises +789 questions instantiated at four standardized input lengths (32K/64K/128K/256K tokens). In +contrast to prior benchmarks restricted to text-only conversations and questions, M EM L ENS provides +multimodal conversation sessions in which text and images are interleaved, together with questions +that require cross-modal reasoning over evidence images and the surrounding textual context. §3.1 +defines the five memory abilities, §3.2 describes the construction pipeline (Figure 1), §3.3 presents +the quality control, and §3.4 describes the cross-modality validation. + +3.1 Memory Abilities + +To comprehensively reflect the retrieval, reasoning, and update that a conversational assistant must +perform over a long multimodal history, M EM L ENS formulates five core memory abilities. We break +each ability into subtypes that target specific reasoning operations. + + • Information Extraction (IE) tests the recall of a specific fact from a single evidence session, + with two subtypes. Entity questions form a two-hop chain: the model first identifies an + abstracted entity in the evidence image, then retrieves the associated information from the + surrounding text. PrevInfo (previous information) questions instead abstract the session + reference, asking the model to recall a visual detail from an image shared in an earlier + session. The visual identification hop draws on image-understanding skills such as fine- + grained recognition, object counting, spatial reasoning, and numerical reasoning. + • Multi-Session Reasoning (MSR) evaluates aggregation across three to eight sessions: count- + ing tallies unique items identified only by their evidence images, arithmetic sums values + stated in text or embedded in visual artifacts, and entity resolution determines co-reference + by comparing images across sessions. + + + 3 + Multimodal Session Simulation Conversation History Assembly + 1 Topic Ontology 2 Web Image 3 Two-agent 4 Multimodal Session + M M: Multimodal Session + Retrieval + Filtering Dialogue + E E: Evidence Session + VLM VLM F F: Filler Session (Text-only) + User Assistant ··· + ··· Assembled Multimodal Conversation History + ··· + 11/02 M ··· + ··· + Topic Pool + 12/02 F ··· + + ✓ image selected before dialogue → structural image-text alignment 。 E ··· + 。 F ··· + Question-Answer Pair Construction Evidence Session Construction 。 + 。 M ··· + 1 Background Generation 2 Visual Anchor Retrieval 3 Entity Ambiguity VLM / Agent + 。 E ··· + Sampled Topic Ontology replace name with generic placeholder 。 + Anchor Entity Golden BEFORE (named entity) AFTER (generic placeholder) + Gate Bridge : What time did I meet Emma F ··· + Emma and I Emma and I at the Golden Gate Bridge last + finally met finally met Saturday? 14/04 E ··· + crossed the crossed the + Golden Gate bridge : 6:42 AM + Bridge together together last Fillers (text-only sessions) + Emma and I finally met last Saturday ··· Saturday ··· Atomic Facts (evidence units) + crossed the Golden Gate + Bridge together last F1 F2 + My sister Emma My sister and I + Saturday, three years after + and I crossed the started crossing + From ShareGPT / UltraChat + we first planned to. We met bridge in + Golden Gate Bridge the bridge the bridge at 6:42 + on the the south side of the last Saturday AM that morning. + bridge at at 6:42 AM to morning. + catch the sunrise, and I + wore Dad’s red windbreaker + Cross with Saturday Cross with Saturday Fixed text/image ratio + from his 1998 photo. + Emma 6:42 AM Emma 6:42 AM indirectly mention the fact in conversation Context Budgets + Halfway across, near the + S1 S2 + second tower, Marcus + proposed to Emma. We text-only path (unsolvable) + celebrated afterward at Pier 32K 64K 128K 256K + Anchor Image + 39. + ✗ Requires image evidence + + + Figure 1: M EM L ENS construction pipeline. + + + • Temporal Reasoning (TR) assesses joint reasoning over temporal references, including both + natural-language expressions and session timestamps, together with visual content: duration + comparison compares two intervals derived from textual or visual cues, and temporal + grounding either orders events chronologically or extracts the specific date of an event. + Beyond entity abstraction, TR also replaces temporal expressions with visual artifacts such + as clock faces and calendar pages. + • Knowledge Update (KU) tests the ability to track an evolving user attribute across a chain of + four successive updates [57] (e.g., “I used to like apples” → “now I prefer kiwi”), requiring + the model to reason from the final state of the chain rather than earlier superseded ones. + • Answer Refusal (AR) removes all supporting evidence from an otherwise answerable ques- + tion, so it can no longer be answered from the remaining context; a correct model must + decline rather than hallucinate [58]. AR serves as a calibration check for hallucination + detection, not a core memory retrieval task. + +A single cross-modal principle unifies the four answerable types: the evidence image carries infor- +mation that the text deliberately withholds, so correct answers require joint visual–textual reasoning +(§3.2); §3.4 validates this dependency empirically. The formal problem specification appears in +Appendix C.1. Appendix I grounds this taxonomy in established memory research, and Appendix C.3 +provides the complete subtype breakdown with representative examples. + +3.2 Data Curation + +The construction proceeds in four components (Figure 1): (i) first, multimodal session simulation +generates topic-grounded multimodal dialogue sessions that form the conversation background; +(ii) next, question construction produces evaluation questions whose answers require visual content +through entity abstraction; (iii) evidence session construction wraps each evidence fact into a complete +session that matches its topical and stylistic profile; and (iv) conversation history assembly interleaves +evidence, haystack, and text-only filler sessions in timestamp order and compiles each question into +four standardized input lengths. + +Multimodal session construction. Each session begins with topic sampling from a hierarchical +ontology. For each topic, we generate an image query to retrieve a batch of candidate images from the +web1 . We then filter these candidates for visual relevance. From this filtered set, we select the final + 1 https://github.com/hellock/icrawler; sourcing, licensing, privacy, and release detailed in Appendix C.7. + + + + + 4 + image. Given this image, we generate sessions through a dual-model simulation with GPT-5.1 [2] +as the user and Gemini-3-Pro [59] as the assistant, producing multi-turn dialogues that incorporate +the selected images into interleaved image-text sequences [60]. Because image selection precedes +dialogue generation, image–text alignment is a structural property of the data rather than a post-hoc +filter. The image retrieval pipeline (§C.5), full topic ontology (§C.2), and image diversity statistics +(§C.6) are detailed in the Appendix. + +Question-Answer Pair. Question generation follows a four-step pipeline. (i) A topic is sampled +from the hierarchical ontology, and a LVLM generates a background paragraph containing salient +named-entities, for instance, a paragraph about San Francisco landmarks that mentions the Golden +Gate Bridge. (ii) One entity is selected as the visual anchor for the question. A text query derived +from the entity is issued to the same web-crawling and multi-model scoring pipeline used for haystack +images (§C.5), yielding a high-relevance image [61]; the anchor may range from a famous landmark +to a specific product model. (iii) Cross-modal dependency is enforced through entity abstraction [62– +64]: the entity in the background paragraph is replaced with a higher level concept drawn from +a dictionary spanning 55 entity categories (e.g., “Golden Gate Bridge” → “the bridge shown in +”). After replacement, the paragraph no longer names the entity, and only the evidence +image can resolve the reference. (iv) The abstracted paragraph, evidence image, and original entity +name are provided to Gemini-3-Pro [59], which generates a (question, answer) pair together with +atomic evidence facts. The generation prompt constrains the question to require both the image and +the surrounding text, closing the text-only shortcut documented in prior work [16]. The full pipeline +and per-subtype generation routes appear in Appendix C.4. + +Evidence session. Directly inserting evidence statements into the conversation history creates +abrupt semantic shifts that make the evidence trivially locatable by similarity-based retrieval. Prior +work has shown that increasing contextual similarity between evidence and distractors raises retrieval +difficulty, a design principle also adopted by LongMemEval [14]. Accordingly, each evidence fact +is wrapped in a complete evidence session generated using the same pipeline as haystack sessions +but grounded on the evidence fact rather than a sampled topic. Consequently, the evidence facts +match the topical and stylistic profile of the surrounding haystack, so that evidence cannot be located +by surface-level shortcuts. To further increase the difficulty, we also prompt the user model to +mention the facts indirectly without stressing them: To embed the fact “I started a new job last +month,” the user turn might open by asking about updating the tax withholding and mention the job +change incidentally later. When an evidence fact carries an evidence image, the image is placed +adjacent to the corresponding textual mention within the session, preserving unambiguous image-text +co-reference after entity abstraction. + +Conversation history assembly. For each question, evidence Table 2: Dataset statistics. +sessions are inserted into a timestamp-ordered history of haystack +sessions, with positions chosen uniformly at random (for KU, we Statistic Value +preserve the evidence-session order; Appendix C.8). Haystack Questions 789 +sessions are curated to be contextually related but uninformative Types / Subtypes 5/9 +for the question, and never include answer-relevant details. We Evidence sessions 2,145 +vary the number of haystack sessions to produce four standardized Avg. turns/session ∼10 +context lengths (32K/64K/128K/256K tokens) using the cross- Avg. images/session ∼1.5 +modal counting scheme of MMLongBench [11] (Table 2). To Tokens/image ∼2,000 +avoid revealing evidence positions via image clustering, we keep 32K → 256K +a fixed text-per-image ratio across the history, padding with text- Sessions/instance 14 → 93 +only filler sessions from ShareGPT and UltraChat [65]. A post- Images/instance 20 → 138 +hoc classifier achieves only marginally above-chance accuracy +when separating evidence from haystack text (Appendix G.8), and the generator scales beyond 256K. + +3.3 Quality Control + +Automated filtering. Each candidate question is screened by two automated checks. A rule-based +pre-filter removes images from the question and evidence and drops cases where the remaining text +already determines the answer. An LLM judge (GPT-5.1 [2]) then sees only the question text (without +evidence or conversation history) and removes items solvable from parametric knowledge alone [66]. + + + 5 + Table 3: Cross-modality ablation on the image-essential and image-supportive questions. With +evidence: question + evidence facts + evidence images, no haystack (n = 634). W/o evidence: +question + evidence facts, no evidence images. + Model Input Overall IE MSR TR KU ∆ + With evidence image 93.13 94.31 100.00 96.91 75.86 — + GPT-5.4 + W/o evidence image 1.74 0.41 0.00 5.15 0.00 −91.39 + With evidence image 89.42 89.02 90.21 96.19 82.24 — + Gemini-3.1-Pro + W/o evidence image 1.89 0.00 0.00 6.19 0.00 −87.53 + + + +Remaining questions therefore require using the evidence image, which is a prerequisite for the +multimodal analyses below. + +Human review. Three rounds of human review operate as a cumulative quality gate on the filtered +questions. Round 1 verifies that the evidence image carries answer-critical information; for AR +questions, which are constructed by removing the evidence facts, this round instead confirms that the +removed facts was answer-critical. Round 2 checks that evidence facts are naturally embedded and +recoverable from their sessions, and that each session reads as a plausible user–assistant exchange +with natural dialogue flow; sessions that fail the naturalness criterion are subsequently refined by +humans. Round 3 reviews the haystack sessions to assess both the quality of the images and the +naturalness of the dialogues. Together, the automated filters and three-round review reduce the initial +pool of 20k candidates to the final 789 questions. The filler sessions drawn from ShareGPT and +UltraChat (§3.2) further ground the surrounding context in authentic user–AI conversational patterns. +Annotation guidelines and inter-annotator agreement appear in Appendix E.1. + +3.4 Cross-modality Validation + +The construction pipeline targets image-necessary questions: across M EM L ENS, 65.7% of questions +are image-essential (the answer is unrecoverable without the evidence image), 14.7% are image- +supportive (the image confirms or disambiguates a textual fact), and 19.6% are text-sufficient (all AR +questions plus a subset of MSR items that test cross-session reasoning over purely textual evidence). +We verify this cross-modal dependency with two empirical checks (Table 3). An answerability test +supplies each image-essential and image-supportive question (n = 634) with its full evidence (textual +facts and evidence images) and confirms that the questions are answerable: GPT-5.4 reaches 93.13% +overall and Gemini-3.1-Pro 89.42%. A multimodal ablation then removes all evidence images: +overall accuracy collapses below 2% for both models. Two frontier proprietary LVLMs converge on +near-identical collapses, showing our questions are highly multimodal. + + +4 Evaluation and Analysis + +4.1 Experimental Setup + +We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS. The LVLMs span +closed-source systems including GPT-5.4 [2], Claude Sonnet 4.5 [7], and Gemini-3.1-Pro [67], +alongside major open-source families such as Kimi-K2.5 [68], Qwen3.5 [69], GLM-4.6V [70], +and Gemma3 [71]; the complete list is provided in Appendix B. The memory-augmented agents +comprise three multimodal pipelines (M3-Agent [35], M2A [34], and M3C [37]) and four text-only +pipelines (Mem0 [9], MemOS [29], MemAgent-7B [30], and Memory-T1 [28]), with backbone +and adapter details reported in Appendix G. Since the four text-only memory agents do not accept +visual inputs, we follow standard practice and replace the image inputs with captions generated by +BLIP-2 [72]; the remaining input adaptations are documented in Appendix B. For comparability, +the LVLMs are evaluated at three context lengths (32K, 64K, and 128K), as many of them do not +natively support 256K, whereas the agents are evaluated across all four context lengths (32K, 64K, +128K, and 256K). We report LLM-as-Judge accuracy [73] using Qwen3-VL-235B-A22B-Instruct +as the judge, cross-validated by GPT-5.4-mini re-judging (κ = 0.93) and a three-annotator human +consensus (Appendix E.2). + + + 6 + IE MSR TR KU AR Overall + Claude Sonnet 4.5 28.0 26.4 19.5 13.3 9.1 2.8 40.2 35.1 33.0 29.3 21.6 16.4 97.8 94.4 93.3 36.5 32.4 27.8 + + Gemini-3.1-Pro 57.3 58.9 55.8 32.2 31.5 29.4 40.9 42.3 41.2 49.1 48.6 46.0 97.8 95.6 96.7 54.1 53.0 52.0 + + GPT-5.4 69.5 63.0 60.2 28.2 27.3 21.7 39.2 42.3 39.7 47.4 43.1 43.1 97.8 96.7 94.4 52.7 52.3 49.6 + + Kimi-K2.5 54.8 52.4 51.6 44.1 35.7 28.7 53.1 52.1 48.2 50.9 49.1 48.6 97.8 97.8 93.3 54.9 54.0 52.0 + +Qwen3.5-122B-A10B 74.4 67.1 43.1 30.1 27.3 22.4 51.5 51.5 41.8 49.1 48.6 46.0 88.9 84.4 83.3 58.7 55.9 45.5 + + Qwen3.5-27B 70.3 62.2 54.1 29.4 21.0 14.0 39.2 35.6 34.0 42.2 44.0 38.8 86.7 67.8 63.3 53.0 46.1 40.7 + +Nemotron-Nano-12B 63.8 56.1 — 16.8 21.0 — 44.8 42.8 — 39.7 34.5 — 45.6 35.6 — 45.0 40.9 — + +Cosmos-Reason2-8B 55.7 46.3 37.0 23.8 20.3 20.3 48.5 45.9 43.8 31.9 24.1 12.9 68.9 54.4 35.6 46.1 39.2 31.9 + + Phi4-Multimodal 31.7 28.0 22.8 20.3 21.0 22.4 43.8 39.2 30.9 19.8 24.1 14.7 43.3 21.1 15.6 32.2 28.1 22.7 + + GLM-4.6V 59.3 61.8 52.8 20.3 23.8 23.1 53.1 53.1 43.8 44.0 34.5 28.4 93.3 64.4 30.0 52.3 49.0 39.0 + + Gemma3-27B 35.0 26.4 19.1 23.8 22.4 18.2 41.2 44.3 37.1 30.2 27.6 13.8 68.9 61.1 48.9 37.6 34.2 26.0 + + Qwen3-VL-235B 51.6 38.6 29.3 25.2 18.2 15.4 49.5 42.8 43.3 36.2 35.3 30.2 88.9 75.6 67.8 48.3 39.7 34.7 + + Qwen3-VL-30B 41.5 39.0 33.7 11.9 11.9 10.5 35.6 37.1 36.6 30.2 29.3 20.7 71.1 65.6 60.0 36.4 35.2 31.3 + + Mem0 13.1 11.5 8.2 25.7 22.9 20.0 50.0 45.8 50.0 17.2 24.1 24.1 77.3 77.3 77.3 31.8 31.3 30.3 + + MemOS 18.0 18.0 16.4 22.9 22.9 22.9 39.6 39.6 43.8 24.1 24.1 17.2 68.2 68.2 81.8 30.3 30.8 31.3 + + MemAgent-7B 18.0 13.1 14.8 25.7 28.6 28.6 62.5 45.8 60.4 41.4 24.1 20.7 13.6 4.5 9.1 32.8 25.6 28.2 + + Memory-T1 18.0 19.7 21.3 25.7 22.9 22.9 62.5 56.2 56.2 13.8 17.2 27.6 9.1 18.2 13.6 28.7 28.7 30.3 + + M3-Agent 18.0 18.0 22.9 22.9 17.1 20.0 29.2 37.5 22.9 6.9 3.5 6.9 13.6 4.5 18.2 19.5 19.0 19.5 + + M2A 14.8 18.0 21.3 8.6 14.3 11.4 2.1 25.0 27.1 0.0 0.0 0.0 22.7 13.6 13.6 15.4 15.9 16.9 + + 32K 64K128K 32K 64K128K 32K 64K128K 32K 64K128K 32K 64K128K 32K 64K128K + +Figure 2: Per-type accuracy (%) by context length for representative 13 LVLMs and 6 memory- +augmented agents. Each panel shows one question type; cells use a green colormap. Missing +cells indicate models that exceed their usable context budget at 128K. LVLMs are evaluated on the +full 789-question benchmark, agents on the 195-question canonical subset. Full rosters appear in +Tables 13 and 14. + + +4.2 Main Results + +Overall performance. Figure 2 reports per-type accuracy across context lengths for a represen- +tative 13-LVLM cohort and six memory agents; the full 27-LVLM and seven-agent rosters are in +Appendix G.1. At 32K, the top eight LVLMs fall within a 6.34% band, so short-context accuracy +no longer separates frontier systems. The picture inverts at 128K: several open-weight leaders lose +more than 13%, while Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% +drop). AR shows the steepest context-driven decline in the open-weight LVLM family, suggesting +that hallucination control is the ability most exposed to evidence dilution at long contexts. Memory +agents occupy a narrower range, with the top four text-only systems clustering within 5% of each +other, while M3-Agent, M3C, and M2A fall substantially lower. Memory agents trail LVLMs across +nearly all types, with the largest gaps on visually grounded retrieval (IE, KU) and on answer refusal +(AR). + +Type-specific difficulty. Per-type accuracy ceilings range from 97.78% on AR down to 44.06% +on MSR (Kimi-K2.5) at 32K. AR, which serves as a calibration check for hallucination detection, +is the easiest type as expected, although its apparent ease erodes substantially at long contexts +for open-weight LVLMs (e.g., GLM-4.6V AR drops from 93.33% at 32K to 30.00% at 128K). +TR follows (60.82% ceiling), because timestamps and dates in session metadata provide explicit +retrieval anchors. IE (74.39% ceiling) and KU (50.86%) form an intermediate band: IE requires +visual grounding, with Entity questions demanding two-hop reasoning through the abstracted image +reference, while KU requires tracking a four-fact update chain in which missing a single image anchor + + + 7 + Visual Textual Output + Mixed Reasoning + IE TR AR 100 + 8% + 80 MSR KU 14% + + + + + % of wrong answers + 70 80 11% + 44% + 60 +Accuracy (%) + + + 60 + 50 73% + + 40 40 + 89% + 75% + 30 42% + + + 20 20 + + 10 10% 11% + 0 + 0 IE MSR TR KU + 32K 64K 128K Memory type + Context length + (b) Wrong-answer error decomposition at 128K con- +(a) Per-type accuracy vs. input length. Solid: LVLM text across the four answerable memory types, split +average (27 models, n=789); dashed: agent average by failure modality (visual, textual, mixed, reasoning, +(7 systems, n=195). Bands: 95% CI. output). Category definitions appear in Table 19. + + +flips the predicted state. MSR is the hardest, as cross-session aggregation over three to eight sessions +defeats every evaluated system: only Kimi-K2.5 (44.06%) and Gemini-3.1-Pro (32.17%) clear 30% +by margin, exposing this as the shared capability ceiling of M EM L ENS. Within types, IE-Entity is +consistently harder than single-hop IE-PrevInfo, and MSR-Arithmetic is the most difficult subtype +overall (Appendix C.3). + +No model dominates all memory abilities. No single model fam- IE MSR TR KU AR +ily dominates across all types (Figure 3). GLM-4.6V [70] leads GPT-5.4 69.5 28.2 39.2 47.4 97.8 +TR but collapses on KU, while Qwen3.5 [69] inverts the pattern. Gemini-3.1-Pro 57.3 32.2 40.9 49.1 97.8 +Kimi-K2.5 [68] is relatively strongest on MSR at 32K, though this + Kimi-K2.5 54.8 44.1 53.1 50.9 97.8 +advantage fades at longer contexts. Gemini-3.1-Pro is the only + Qwen3.5 (2) 72.4 29.7 45.4 45.7 87.8 +model simultaneously competitive on IE, KU, and MSR at 128K. + GLM-4.6V 59.3 20.3 53.1 44.0 93.3 +Memory agents exhibit an inverted profile: Memory-T1 reaches + Mem0 13.1 25.0 50.0 17.2 77.3 +high TR accuracy through BM25 date matching but falls well below +direct LVLMs on IE, substituting keyword retrieval for the visual MemAgent-7B 17.2 25.0 61.5 40.6 13.6 + +grounding that IE demands. Memory-T1 18.0 25.0 63.5 13.8 9.1 + + M3-Agent 19.0 22.2 30.6 6.2 13.6 + +4.3 Analysis + Figure 3: Memory-ability spe- +Current memory pipelines lose faithfulness to original visual cialization across representative + LVLMs and memory agents. +evidence. The gap between agents and LVLMs is largest on the +visually grounded types (IE and KU). Despite their different input formats (§4.1), both text-only +and multimodal pipelines compress evidence visual information into a fixed memory representation +at storage time, leaving the original image pixels inaccessible at query time. Retrieval over these +compressed encodings is also less reliable than retrieval over the raw conversation text. The loss +is sharpest on IE and KU because captions and embeddings retain only the gist of an image and +discard fine-grained visual cues, such as counts, attributes, and spatial relations, that these two +types specifically probe. The same gap appears when blip-captions replace raw images for text-only +agents, indicating that the bottleneck lies in lossy cross-modality storage rather than in the answering +model. Closing this gap requires memory architectures that preserve image-level evidence rather than +caption-based compression. + +Post-training on memory agent backbones weakens abstention. Memory agents fall far below +their direct-inference counterparts on AR. We evaluate two agents that keep the backbone frozen, +Mem0 (77.27%) and MemOS (68.18%), and five that further finetune the backbone via RL or LoRA + + + 8 + for memory management (M2A, M3-Agent, M3C, MemAgent-7B, and Memory-T1). While the +frozen-backbone agents preserve much of the abstention behavior of the underlying LLM, the others +collapse to 9–22% AR. M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score +81.82% under direct inference, and the backbone ablation in Appendix G.6 confirms that stronger +backbones alone do not close this gap. This suggests that the reward design of current RL/SFT +fine-tuning on memory agent backbones optimizes mainly answer correctness and retrieval success, +providing no signal that refusing an unanswerable question is correct, so abstention degrades after +training. Future agent designs should therefore optimize memory access, answer accuracy, and +evidence-sensitive abstention jointly [74], rather than treating memory management as independent +of hallucination control. + +LVLMs and memory agents show complementary scaling trade-offs. Figure 4a plots accuracy +across five abilities from 32K to 128K for both LVLMs and memory agents. The two approaches +respond to context scaling in structurally different ways. LVLMs degrade substantially on retrieval- +heavy types, with IE and KU losing ∼20% and ∼12% respectively as evidence images become harder +to locate among growing filler content. MSR shows apparent flatness that reflects a floor effect near +30% rather than genuine robustness. AR shows the steepest LVLM decline (∼75% at 32K to ∼45% +at 128K), suggesting that growing filler content erodes abstention and pushes LVLMs to hallucinate +on unanswerable questions rather than refuse them. Six of seven memory agents, by contrast, stay +within ±7% from 32K to 256K because their retrieve-then-reason pipeline is length-invariant by +construction. The two failure modes are orthogonal: LVLMs lose to context length, while memory +agents lose to lossy multimodal compression at storage time. Because each architecture covers only +one axis, scaling along that axis leaves the other failure mode untreated, motivating hybrid designs +that span both axes. + +Memory ability correlations reveal distinct sources of difficulty. +We analyze pairwise Spearman correlations among the five question IE 1.00 -0.19 0.12 0.65 0.36 0.23 + 0.31 + + + +types at 32K (Figure 5). Correlations vary substantially across type MSR -0.19 1.00 0.33 0.21 -0.11 0.060.21 + +pairs, indicating that the types do not measure a single capability. IE +and KU form the most consistent retrieval-oriented pair: they have TR 0.12 0.33 1.00 0.29 -0.00 0.18 + 0.13 + + + +the strongest correlation at 32K and remain significantly correlated KU 0.65 0.21 0.29 1.00 0.59 0.43 + 0.19 + + +across context lengths, reflecting their shared need to first locate +the relevant evidence image. KU also correlates with AR at 32K, AR 0.36 -0.11-0.00 0.59 1.00 0.21 + 0.28 + + + +suggesting that some KU questions still depend on accurate evidence IE MSR TR KU AR Avgstd + +selection. In contrast, MSR shows weak correlations with IE and Figure 5: Spearman rank correla- +AR at all lengths, implying that its main challenge is aggregating tion (ρ) at 32K across all 34 evalu- +information across multiple evidence pieces rather than retrieving ated LVLMs and memory agents. +a single image. These findings reveal two complementary difficulty +axes: an evidence-retrieval axis (IE, KU) and an aggregation axis (MSR). They also illustrate how +a single aggregate score can obscure differences among long-context abilities, motivating per-type +evaluation alongside overall performance [43, 44]. Full correlation matrices at 64K and 128K appear +in Appendix G.5. + +Error analysis. Figure 4b decomposes wrong answers at 128K across the four answerable types, +with the category definitions provided in Appendix G.4. On IE and KU, nearly 90% of errors fall in +the Visual category, showing that the model fails to locate or read the evidence image; once the image +is reached, the answer is usually extracted correctly. TR errors split between Mixed and Reasoning, +reflecting both image-supportive grounding and small closed-set selection. MSR is the only type +whose errors are dominated by the Reasoning category (73%). We further conduct an oracle-retrieval +diagnostic in Appendix G.7, which restores accuracy on MSR to 90–100% when only the required +sessions are supplied. This suggests a reasoning error at long context is largely induced by upstream +retrieval failure, since the model computes correctly over the wrong subset of evidence. LVLMs +and memory agents fail differently. LVLMs lose visual evidence as filler content accumulates in the +context window, whereas memory agents lose it through lossy cross-modal compression at storage +time. As memory grows, near-misses at 32K shift toward total misses at 128K (Appendix G.4) [75], +suggesting that growth weakens evidence retrieval rather than reasoning. Future memory designs +should therefore focus more on cross-modal evidence-retrieval fidelity instead of solely reasoning +improvements. + + + 9 + 5 Conclusion +We introduced M EM L ENS, the first benchmark for multimodal conversational memory to evaluate +LVLMs and memory-augmented agents under a unified, length-controlled protocol (32K–256K +tokens; five memory abilities). The task remains challenging: even the strongest LVLM reaches +only 58.68% at 32K. LVLMs degrade sharply once input contexts grow, while memory agents stay +length-stable but discard fine-grained visual evidence at storage time, and post-training on memory +agent backbones can erode the agents’ abstention behavior. The five abilities show low cross-type +correlation, confirming that per-type evaluation is necessary, and nearly 90% of information-extraction +errors stem from visual perception rather than comprehension, indicating that scaling first harms +evidence retrieval, not reasoning. Visual-evidence retention and retrieval, rather than raw scaling of +either context or memory, therefore emerges as the principal bottleneck to address in the future. + +Acknowledgments and Disclosure of Funding +The authors of this paper were supported by the National Key Research and Development Program of +China (2025YFE0200500), the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong +Kong, SAR, China, and the AoE (AoE/E-601/24-N), the RIF (R6021-20) and the GRF (16205322) +from RGC of Hong Kong, SAR, China. We also thank the NVIDIA AI Technology Center (NVAITC) +and OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. for their support and additional +funding. + +Ethics Statement +M EM L ENS is designed exclusively for evaluating multimodal long-term conversational memory +in large vision-language models (LVLMs) and memory-augmented agents under a unified, length- +controlled protocol. The benchmark is not intended as a training dataset for developing memory- +equipped systems but rather as a controlled diagnostic for fair comparison across architectures. We +release every artifact under frozen version tags so that any reported leaderboard cell remains traceable +to the exact item set on which it was scored. +The 4,695 source images underlying M EM L ENS are retrieved from public web image search via +iCrawler against a non-person-centric topic ontology; candidates carrying watermarks, stock-photo +logos, or copyright overlays are excluded at retrieval time, and the construction prompts never request +the model to identify, name, or describe any depicted person. Author-produced artifacts (questions, +evidence facts, prompts, and human-annotation records) are released under CC-BY-4.0 and the +evaluation harness under MIT, while third-party images retain their original source-site licenses; +per-image provenance metadata (source URL, retrieval timestamp, perceptual hash) accompanies the +release, and a takedown contact in the project repository allows seven-day removal of any flagged +image. +Human review is conducted by project members rather than crowd-workers, so no third-party data +collection or human-subjects protocol is involved. Three sequential rounds (Appendix E.1) audit +cross-modal necessity at the question level, naturalness and recoverability at the session level, and a +stratified sample of haystack sessions before any item reaches the released set. +We caution that M EM L ENS is not intended as a training dataset: exposure of the 789 evaluation items +as supervised data would compromise their diagnostic value, and the datasheet explicitly discourages +such use. While the benchmark is grounded in multi-session multimodal conversation, the five +memory abilities it isolates (information extraction, multi-session reasoning, temporal reasoning, +knowledge update, and answer refusal) describe general functional requirements of any long-horizon +conversational assistant. We therefore expect M EM L ENS to serve as a useful indicator of memory +robustness in adjacent settings, such as document-grounded or voice-grounded assistants, where +retrieval, integration, and calibrated abstention play comparable roles. + +Reproducibility Statement +All closed-source LVLM evaluations and synthetic data construction were performed via the public +APIs of OpenAI, Anthropic, and Google, with a total experimental cost of approximately 4,500 USD + + + 10 + covering both pipeline construction and benchmarking. Open-weight LVLMs are served locally +with vLLM v0.17–0.18 on 8×A100-80GB nodes with tensor parallelism for context windows at +and above 128K. Per-model decoding budgets, judge protocol, retrieval depth for memory agents, +and adapter details for the four text-only pipelines (Mem0, MemOS, MemAgent-7B, Memory- +T1) are documented in Appendix B. The four-stage construction pipeline (multimodal session +simulation, question construction, evidence session construction, conversation history assembly) +is described in Section 3, with per-subtype generation routes in Appendix C.4 and full prompt +templates in Appendix F. Quality control combines automated filtering with a three-round human +review (Section 3.3, Appendix E.1) and an LLM-as-Judge protocol cross-validated by an independent +judge family and a three-annotator human consensus; agreement statistics and per-question-type +breakdowns appear in Appendix E.2. The 789-question benchmark, 4,695 source images with +provenance metadata, evaluation harness, and prompt templates are publicly released under frozen +version tags at https://huggingface.co/datasets/xiyuRenBill/MEMLENS (dataset) and +https://github.com/xrenaf/MEMLENS (code) so that every reported leaderboard cell can be +reproduced end-to-end. + +References + [1] Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complex- + ity. Technical report, Technical report, Bytedance, 2025. URL https://lf3-static. bytednsdoc. + com . . . . + [2] OpenAI. Openai gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267. + [3] OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. + [4] Anthropic. Claude 3 model card. https://assets.anthropic.com/m/61e7d27f8c8f5 + 919/original/Claude-3-Model-Card.pdf, 2024. Accessed: 2026-04-30. + [5] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett + Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal + understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. + [6] Bytedance Seed. Seed1. 8 model card: Towards generalized real-world agency. arXiv preprint + arXiv:2603.20633, 2026. + [7] Anthropic. Claude Sonnet 4.5 System Card. https://assets.anthropic.com/m/12f21 + 4efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf, 2025. + [8] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and + Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL https: + //arxiv.org/abs/2310.08560. + [9] Deshraj Yadav, Taranjeet Singh, and Prashant Srivastava. Mem0: Building production-ready + AI agents with scalable long-term memory, 2025. URL https://arxiv.org/abs/2504.1 + 9413. + [10] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag: + Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983, 2024. + [11] Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, + Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, and Mark Steedman. MMLong- + Bench: Benchmarking long-context vision-language models effectively and thoroughly, 2025. + URL https://arxiv.org/abs/2505.10610. + [12] Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang + Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, + Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024. URL https: + //arxiv.org/abs/2406.07230. + [13] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu + Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin + Cao, and Aixin Sun. Mmlongbench-doc: Benchmarking long-context document understanding + with visualizations, 2024. URL https://arxiv.org/abs/2407.01523. + [14] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long- + memeval: Benchmarking chat assistants on long-term interactive memory, 2025. URL + https://arxiv.org/abs/2410.10813. + + + 11 + [15] Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremen- + tal multi-turn interactions, 2026. URL https://arxiv.org/abs/2507.05257. +[16] Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and + Yuwei Fang. Evaluating very long-term conversational memory of llm agents, 2024. URL + https://arxiv.org/abs/2402.17753. +[17] Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, + Hendrik Hamann, Jingrui He, and Hanghang Tong. Mem-Gallery: Benchmarking multimodal + long-term conversational memory for MLLM agents, 2026. URL https://arxiv.org/ab + s/2601.03515. +[18] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing + large language models with long-term memory, 2023. URL https://arxiv.org/abs/23 + 05.10250. +[19] Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun + Peng. MRAG-Bench: Vision-centric evaluation for retrieval-augmented multimodal models, + 2025. URL https://arxiv.org/abs/2410.08182. +[20] Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay + Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking + long-context capability of multimodal large language models, 2025. URL https://arxiv. + org/abs/2406.11230. +[21] Greg Kamradt. Needle in a haystack — pressure testing LLMs. https://github.com/gka + mradt/LLMTest_NeedleInAHaystack, 2023. GitHub repository. +[22] Yiming Du, Wenyu Huang, Danna Zheng, Zhaowei Wang, Sebastien Montella, Mirella Lapata, + Kam-Fai Wong, and Jeff Z. Pan. Rethinking memory in llm based agents: Representations, + operations, and emerging topics, 2025. URL https://arxiv.org/abs/2505.00675. +[23] Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun + Ma, and Zhoujun Li. SCM: Enhancing large language model with self-controlled memory + framework, 2025. URL https://arxiv.org/abs/2304.13343. +[24] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. + Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval, 2024. +[25] Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: + Neurobiologically inspired long-term memory for large language models, 2025. URL https: + //arxiv.org/abs/2405.14831. +[26] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: + Agentic memory for LLM agents, 2025. URL https://arxiv.org/abs/2502.12110. +[27] Yiming Du, Bingbing Wang, Yang He, Bin Liang, Baojun Wang, Zhongyang Li, Lin Gui, + Jeff Z. Pan, Ruifeng Xu, and Kam-Fai Wong. Memguide: Intent-driven memory selection for + goal-oriented multi-session llm agents, 2025. URL https://arxiv.org/abs/2505.20231. +[28] Yiming Du, Baojun Wang, Yifan Xiang, Zhaowei Wang, Wenyu Huang, Boyang Xue, Bin + Liang, Xingshan Zeng, Fei Mi, Haoli Bai, Lifeng Shang, Jeff Z. Pan, Yuxin Jiang, and Kam-Fai + Wong. Memory-t1: Reinforcement learning for temporal reasoning in multi-session agents, + 2025. URL https://arxiv.org/abs/2512.20092. +[29] Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, + Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, + Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, Hongkang Yang, Zhi-Qin John + Xu, and Feiyu Xiong. Memos: An operating system for memory-augmented generation (mag) + in large language models, 2025. URL https://arxiv.org/abs/2505.22101. +[30] Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin + Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. Memagent: Reshaping + long-context llm with multi-conv rl-based memory agent, 2025. URL https://arxiv.org/ + abs/2507.02259. +[31] Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi- + modal retrieval is what you need for multi-page multi-document understanding, 2024. URL + https://arxiv.org/abs/2411.04952. + + + 12 + [32] Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and + Pierre Colombo. Colpali: Efficient document retrieval with vision language models, 2025. + URL https://arxiv.org/abs/2407.01449. +[33] Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, + Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. VLM2Vec-V2: + Advancing multimodal embedding for videos, images, and visual documents, 2025. URL + https://arxiv.org/abs/2507.04590. +[34] Junyu Feng, Binxiao Xu, Jiayi Chen, Mengyu Dai, Cenyang Wu, Haodong Li, Bohan Zeng, + Yunliu Xie, Hao Liang, Ming Lu, and Wentao Zhang. M2A: Multimodal memory agent + with dual-layer hybrid memory for long-term personalized interactions, 2026. URL https: + //arxiv.org/abs/2602.07624. +[35] Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li. + Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory, + 2025. URL https://arxiv.org/abs/2508.09736. +[36] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, + Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to + sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on + Computer Vision and Pattern Recognition, pages 18221–18232, 2024. +[37] Jihyoung Jang, Minwook Bae, Minji Kim, Dilek Hakkani-Tur, and Hyounghun Kim. Enabling + chatbots with eyes and ears: An immersive multimodal conversation system for dynamic + interactions, 2025. URL https://arxiv.org/abs/2506.00421. +[38] Bingbing Wang, Yiming Du, Bin Liang, Zhixin Bai, Min Yang, Baojun Wang, Kam-Fai + Wong, and Ruifeng Xu. A new formula for sticker retrieval: Reply with stickers in multi- + modal and multi-session conversation. Proceedings of the AAAI Conference on Artificial + Intelligence, 39(24):25327–25335, Apr. 2025. doi: 10.1609/aaai.v39i24.34720. URL + https://ojs.aaai.org/index.php/AAAI/article/view/34720. +[39] Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented + generation with long-context llms. arXiv preprint arXiv:2406.15319, 2024. +[40] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: + Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International + Conference on Learning Representations, 2023. +[41] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao + Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for + long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for + Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, 2024. +[42] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng + Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning + on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024. +[43] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, + and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? + In First Conference on Language Modeling, 2024. +[44] Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe + Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effec- + tively and thoroughly. arXiv preprint arXiv:2410.02694, 2024. +[45] Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng + Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language + models. In Proceedings of the 62nd Annual Meeting of the Association for Computational + Linguistics (Volume 1: Long Papers), pages 14388–14411, 2024. +[46] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, + Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. ∞Bench: Extending long context evaluation + beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for + Computational Linguistics (Volume 1: Long Papers), pages 15262–15277, 2024. +[47] Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and + reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024. + + + 13 + [48] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, + Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking + multi-task long video understanding, 2025. URL https://arxiv.org/abs/2406.04264. +[49] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for + long-context interleaved video-language understanding, 2024. URL https://arxiv.org/ + abs/2407.15754. +[50] Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun + Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark + integrating understanding, reasoning, and locating. arXiv preprint arXiv:2412.18424, 2024. +[51] Yew Ken Chia, Liying Cheng, Hou Pong Chan, CHAOQUN LIU, Maojia Song, Mahani + Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal + super-long document understanding and a retrieval-aware tuning framework. 2024. +[52] Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman + Pan, Yangqiu Song, and Dong Yu. Divscene: Towards open-vocabulary object navigation with + large vision language models in diverse scenes. arXiv preprint arXiv:2410.02730, 2024. +[53] Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. + Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532, 2024. +[54] Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: + Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture. arXiv preprint + arXiv:2409.02889, 2024. +[55] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and + Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large + language models. CoRR, 2024. +[56] Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong + Wang, and Kam-Fai Wong. Perltqa: A personal long-term memory dataset for memory + classification, retrieval, and synthesis in question answering, 2024. URL https://arxiv.or + g/abs/2402.16288. +[57] Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei + Xu. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on + Empirical Methods in Natural Language Processing, pages 8541–8565, 2024. +[58] Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, + Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say “I don’t know”. + In Proceedings of the 2024 Conference of the North American Chapter of the Association for + Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages + 7113–7139, 2024. +[59] Google DeepMind. Gemini 3 Pro Model Card. https://deepmind.google/models/mod + el-cards/gemini-3-pro/, 2026. +[60] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton + Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: + An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural + Information Processing Systems, 36:71683–71702, 2023. +[61] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina + Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing + millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on + Computer Vision, pages 12065–12075, 2023. +[62] Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, + Xin Liu, and Yangqiu Song. Abspyramid: Benchmarking the abstraction ability of language + models with a unified entailment graph. In Findings of the Association for Computational + Linguistics: NAACL 2024, pages 3991–4010, 2024. +[63] Zhaowei Wang, Wei Fan, Qing Zong, Hongming Zhang, Sehyun Choi, Tianqing Fang, Xin + Liu, Yangqiu Song, Ginny Wong, and Simon See. Absinstruct: Eliciting abstraction ability + from llms through explanation tuning with plausibility estimation. In Proceedings of the 62nd + Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), + pages 973–994, 2024. + + + 14 + [64] Paul Lerner, Olivier Ferret, and Camille Guinaudeau. Cross-modal retrieval for knowledge- + based visual question answering, 2024. URL https://arxiv.org/abs/2401.05736. +[65] Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, + Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality + instructional conversations. In Proceedings of the 2023 Conference on Empirical Methods in + Natural Language Processing, pages 3029–3051, 2023. +[66] Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh + Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and + non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for + Computational Linguistics (Volume 1: Long Papers), pages 9802–9822, 2023. +[67] Google DeepMind. Gemini 3.1 pro, February 2026. URL https://storage.googleap + is.com/deepmind-media/Model-Cards/Gemini-3-1-Pro-Model-Card.pdf. Model + card. +[68] Kimi Team. Kimi k2.5: Visual agentic intelligence, 2026. URL https://arxiv.org/abs/ + 2602.02276. +[69] Qwen Team. Qwen3.5: Towards Native Multimodal Agents. https://www.alibabacloud + .com/blog/qwen3-5-towards-native-multimodal-agents_602894, 2026. +[70] Zhipu AI. GLM-4.6V Model Card. https://huggingface.co/zai-org/GLM-4.6V, + 2025. +[71] Gemma Team. Gemma 3 technical report, 2025. URL https://arxiv.org/abs/2503.1 + 9786. +[72] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- + image pre-training with frozen image encoders and large language models. In Proceedings + of the 40th International Conference on Machine Learning, volume 202 of Proceedings of + Machine Learning Research, pages 19730–19742. PMLR, 2023. +[73] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao + Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and + Ion Stoica. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. In Advances in + Neural Information Processing Systems, volume 36, 2023. +[74] Skylar Zhai, Jingcheng Liang, and Dongyeop Kang. Abstain-R1: Calibrated abstention and + post-refusal clarification via verifiable RL, 2026. URL https://arxiv.org/abs/2604.1 + 7073. +[75] Mosh Levy, Alon Jacoby, and Yoav Goldberg. Same task, more tokens: the impact of input + length on the reasoning performance of large language models. In Proceedings of the 62nd + Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), + pages 15339–15353, 2024. +[76] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao + Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie + Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin + Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng + Liu, Shixuan Liu, Dunjie Lu, Ruilin Luo, Chenxu Lv, Rui Men, Lingchen Meng, Xuancheng + Ren, Xingzhang Ren, Sibo Song, Yuchong Sun, Jun Tang, Jianhong Tu, Jianqiang Wan, Peng + Wang, Pengfei Wang, Qiuyue Wang, Yuxuan Wang, Tianbao Xie, Yiheng Xu, Haiyang Xu, Jin + Xu, Zhibo Yang, Mingkun Yang, Jianxin Yang, An Yang, Bowen Yu, Fei Zhang, Hang Zhang, + Xi Zhang, Bo Zheng, Humen Zhong, Jingren Zhou, Fan Zhou, Jing Zhou, Yuanzhi Zhu, and + Ke Zhu. Qwen3-vl technical report, 2025. URL https://arxiv.org/abs/2511.21631. +[77] V Team. Glm-4.5v and glm-4.1v-thinking: Towards versatile multimodal reasoning with + scalable reinforcement learning, 2026. URL https://arxiv.org/abs/2507.01006. +[78] Microsoft. Phi-4-mini technical report: Compact yet powerful multimodal language models + via mixture-of-loras, 2025. URL https://arxiv.org/abs/2503.01743. +[79] NVIDIA. Cosmos-Reason2-8B Model Card. https://huggingface.co/nvidia/Cosmos + -Reason2-8B, 2025. +[80] NVIDIA. Nvidia nemotron nano v2 vl, 2025. URL https://arxiv.org/abs/2511.03929. + + + 15 + [81] An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, + Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, + Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei + Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin + Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, + Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi + Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang + Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu + Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL + https://arxiv.org/abs/2505.09388. +[82] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, + Dayiheng Liu, Fei Huang, et al. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115, + 2024. +[83] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing + Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui + Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing + vision-language model’s perception of the world at any resolution, 2024. URL https: + //arxiv.org/abs/2409.12191. +[84] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, + Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang, + Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong, + Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-vl2: + Mixture-of-experts vision-language models for advanced multimodal understanding, 2024. + URL https://arxiv.org/abs/2412.10302. +[85] Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: + Enhanced transformer with rotary position embedding, 2024. URL https://arxiv.org/ab + s/2104.09864. +[86] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context + window of large language models via positional interpolation, 2023. URL https://arxiv. + org/abs/2306.15595. +[87] Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, + Fan Yang, and Mao Yang. Longrope: Extending LLM context window beyond 2 million + tokens, 2024. URL https://arxiv.org/abs/2402.13753. +[88] Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. Yarn: Efficient context + window extension of large language models, 2023. URL https://arxiv.org/abs/2309 + .00071. +[89] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. + URL https://arxiv.org/abs/2307.08691. +[90] Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, and Ming- + Wei Chang. Can pre-trained vision and language models answer visual information-seeking + questions?, 2023. URL https://arxiv.org/abs/2302.11713. +[91] Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan + Bisk. Webqa: Multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on + computer vision and pattern recognition, pages 16495–16504, 2022. +[92] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini + Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, + and Ilya Sutskever. Learning transferable visual models from natural language supervision. + In Proceedings of the 38th International Conference on Machine Learning, volume 139 of + Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 2021. +[93] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for + language image pre-training. In Proceedings of the IEEE/CVF International Conference on + Computer Vision (ICCV), pages 11941–11952, 2023. +[94] Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on + document images. In Proceedings of the IEEE/CVF winter conference on applications of + computer vision, pages 2200–2209, 2021. + + + 16 + [95] OpenAI. Gpt-4v(ision) system card. https://cdn.openai.com/papers/GPTV_System_ + Card.pdf, 2023. Accessed: 2026-04-30. + [96] Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna + Wallach, Hal Daumé III, and Kate Crawford. Datasheets for datasets. Communications of the + ACM, 64(12):86–92, 2021. doi: 10.1145/3458723. URL https://doi.org/10.1145/3458 + 723. + [97] Jacob Cohen. A coefficient of agreement for nominal scales. Educational and Psychological + Measurement, 20(1):37–46, 1960. + [98] Jianjie Zheng, Zhichen Liu, Zhanyu Shen, Jingxiang Qu, Guanhua Chen, Yile Wang, Yang + Xu, Yang Liu, and Sijie Cheng. Evaluating memory capability in continuous lifelog scenario, + 2026. URL https://arxiv.org/abs/2604.11182. Findings of ACL 2026. + [99] Zhaowei Wang, Quyet V Do, Hongming Zhang, Jiayao Zhang, Weiqi Wang, Tianqing Fang, + Yangqiu Song, Ginny Wong, and Simon See. Cola: Contextualized commonsense causal + reasoning from the causal inference perspective. In Proceedings of the 61st Annual Meeting of + the Association for Computational Linguistics (Volume 1: Long Papers), pages 5253–5271, + 2023. +[100] Zhaowei Wang, Hongming Zhang, Tianqing Fang, Yangqiu Song, Ginny Wong, and Simon + See. Subeventwriter: Iterative sub-event sequence generation with coherence controller. In + Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, + pages 1590–1604, 2022. +[101] Shaohang Wei, Wei Li, Feifan Song, Wen Luo, Tianyi Zhuang, Haochen Tan, Zhijiang Guo, + and Houfeng Wang. TIME: A multi-level benchmark for temporal reasoning of LLMs in + real-world scenarios, 2025. + + + + + 17 + A Use of Large Language Models +During the preparation of this paper, we made controlled use of LLMs (specifically ChatGPT and +Claude) as auxiliary writing tools. The LLMs were employed solely for stylistic refinement— +improving the fluency, grammar, and readability of paragraphs that were originally drafted by the +authors. Importantly, the scientific content, methodology, experimental design, and main narrative +of the paper were fully conceived, written, and validated by the authors without reliance on LLMs. +Therefore, LLMs served purely in a supportive role for polishing author-written text, and their +contribution does not rise to the level of co-authorship. + +B Evaluation Setup +B.1 Models + +We evaluate 27 LVLMs and seven memory-augmented agent systems. Because compute is limited +and not all LVLMs support a 256K-token context, we evaluate the LVLMs at 32K, 64K, and 128K +for a fair comparison, and the memory-augmented agents at all four lengths. + +Closed-source API models. GPT-5.4 [2] (OpenAI), Claude Sonnet 4.5 [7] (Anthropic), and +Gemini-3.1-Pro [67] (Google). + +Open-source LVLMs. Kimi-K2.5 [68] (Moonshot), Qwen3-VL family [76] (235B-A22B and 30B- +A3B MoE; 8B, 4B, 2B dense) in both Instruct and Thinking modes, GLM-4.6V [70], GLM-4.5V [77], +Gemma3 [71] (27B, 12B, 4B), Phi4-Multimodal [78], Cosmos-Reason2-8B [79], Nemotron-Nano- +12B [80], and the Qwen3.5 family [69] (122B-A10B MoE; 27B, 9B, 4B, 2B dense). + +Memory-augmented agents. We evaluate seven memory-augmented agents, split into three mul- +timodal agents and four text-only agents. The multimodal agents are M3-Agent [35] (ColPali [32] +retrieval + RL-trained Qwen2-VL-7B), M2A [34] (dual-layer SQLite + SigLIP2 + Qwen3-VL- +8B), and M3C [37] (LoRA-adapted Qwen2-VL-2B session retrieval). The text-only agents are +Mem0 [9] (FAISS vector store + Qwen3-8B), MemOS [29] (layered memory architecture + Qwen3- +8B), MemAgent-7B [30] (recurrent sliding-window + RL-trained Qwen2.5-7B), and Memory-T1 [28] +(BM25 text retrieval + RL-trained Qwen2.5-3B). +For agents that include RL-finetuned or LoRA-adapted models (Memory-T1, MemAgent-7B, M2A, +M3C, M3-Agent), we use their released checkpoints to reflect each system’s published capability. +For Mem0 and MemOS, which are framework-based systems without custom-trained models, we +use Qwen3-8B [81] as the backbone for both, providing a matched-scale comparison against the +open-weight Qwen3 family in our roster. To disentangle backbone quality from architecture, we +additionally evaluate both frameworks with alternative backbones, including the original gpt-4.1-mini +for Mem0 and Qwen2.5-7B [82] for MemOS (Table 21). + +Agent evaluation protocol. Because agent pipelines are substantially slower than direct LVLM +inference (M2A takes roughly 60× longer per question), we evaluate all agents on a stratified 195- +question subset (∼1/4 of the benchmark; derivation in Appendix G.2). LVLMs are evaluated at +32K, 64K, and 128K; agents are evaluated at all four context lengths including 256K. The four +text-only agents receive BLIP-2 [72] generated image captions in place of actual images as text input. +Among the three multimodal agents, M3-Agent is a video-based model that does not natively support +interleaved image-text conversations; we render each session as a composite image and feed sessions +as a sequence of images. M2A and M3C process the multimodal input directly. Table 4 lists the full +model specifications. + +Agent input-format adapters and protocol asymmetry. Memory agents and direct LVLMs do +not consume the same input. Each agent ingests the conversation through an adapter that depends on +its architecture: the four text-only agents see BLIP-2 [72] captions in place of every evidence image, +M3-Agent sees one composite image per session because its video-style backbone does not natively +accept interleaved image-text sequences, and only M2A and M3C process the original interleaved +messages directly. At answer time the asymmetry persists: the text-only agents have no path back +to pixel evidence, M3-Agent re-attends a session-level composite, while M2A and M3C retrieve + + + 18 + Table 4: Model specifications for all 27 evaluated LVLMs. Length means the training length (default) +or claimed context window (denoted by † ). All models are instruction-tuned. “Image Proc.” stands +for Image Processing, which is mainly Dynamic Resolution ViT [83] or Dynamic Tiling [84]. Most +models extend context length via RoPE [85] with position interpolation techniques [86–88]. + Name Length Image Proc. # Params + Proprietary + Claude Sonnet 4.5 200,000† ? ? + Gemini-3.1-Pro 1,048,576† ? ? + GPT-5.4 1,000,000† ? ? + Qwen3.5 + Qwen3.5-122B-A10B 262,144 Dynamic-Resolution ViT 122B + Qwen3.5-27B 262,144 Dynamic-Resolution ViT 27B + Qwen3.5-9B 262,144 Dynamic-Resolution ViT 9B + Qwen3.5-4B 262,144 Dynamic-Resolution ViT 4B + Qwen3.5-2B 262,144 Dynamic-Resolution ViT 2B + Other Open-Source + Nemotron-Nano-12B 131,072 Dynamic Tiling 12B + Cosmos-Reason2-8B 262,144 Dynamic-Resolution ViT 8B + Phi4-Multimodal 131,072 Dynamic Tiling 5.6B + Kimi-K2.5 262,144† Dynamic-Resolution ViT 1T + GLM + GLM-4.6V 131,072 Dynamic Resolution ViT 106B + GLM-4.5V 65,536 Dynamic Resolution ViT 106B + Gemma3 + Gemma3-27B 131,072† Dynamic Tiling 27B + Gemma3-12B 131,072† Dynamic Tiling 12B + Gemma3-4B 131,072† Dynamic Tiling 4B + Qwen3-VL + Qwen3-VL-235B (T) 262,144 Dynamic-Resolution ViT 235B + Qwen3-VL-235B (I) 262,144 Dynamic-Resolution ViT 235B + Qwen3-VL-30B (T) 262,144 Dynamic-Resolution ViT 30B + Qwen3-VL-30B (I) 262,144 Dynamic-Resolution ViT 30B + Qwen3-VL-8B (T) 262,144 Dynamic-Resolution ViT 8B + Qwen3-VL-8B (I) 262,144 Dynamic-Resolution ViT 8B + Qwen3-VL-4B (T) 262,144 Dynamic-Resolution ViT 4B + Qwen3-VL-4B (I) 262,144 Dynamic-Resolution ViT 4B + Qwen3-VL-2B (T) 262,144 Dynamic-Resolution ViT 2B + Qwen3-VL-2B (I) 262,144 Dynamic-Resolution ViT 2B + + + + +embedding-based memory entries (no raw pixels at query time). Direct LVLMs, in contrast, attend +over the original conversation pixel-for-pixel within the model’s context window. Table 5 lists each +adapter explicitly. We do not normalize this asymmetry because the adapter is part of the system being +evaluated—released checkpoints assume the input format their authors trained on, and any uniform +substitute would either degrade architectures that depend on caption-only memory (Mem0, MemOS, +MemAgent-7B, Memory-T1) or block agents whose backbones cannot accept interleaved input +(M3-Agent). Reported deficits relative to direct LVLMs therefore conflate adapter-induced visual +information loss with retrieval and reading quality. The matched-backbone contrast in Appendix G.6 +(M2A vs. direct Qwen3-VL-8B-Instruct on identical weights) and the backbone ablations for Mem0 +and MemOS (Table 21) isolate the architectural component, while the BLIP-2 captioning step bounds +the visual-information ceiling for text-only agents from above. + + + 19 + Table 5: Per-agent input adapters. “Write-time visual” is the form in which evidence images enter +the memory store; “answer-time visual” is the form available to the backbone when the question +is asked. Direct LVLMs (omitted from the table) attend over the original interleaved conversation +pixel-for-pixel within the model’s context window. + Agent Backbone Write-time visual Answer-time visual + Multimodal-backbone agents + M3-Agent Video LVLM (Qwen2-VL-7B) Composite per-session image Retrieved session composite(s) + M2A Native LVLM (Qwen3-VL-8B) Original images Stored embeddings + M3C Native LVLM (Qwen2-VL-2B) Original images Stored embeddings + Text-only-backbone agents + Mem0 Text LLM (Qwen3-8B) BLIP-2 captions only Captions only + MemOS Text LLM (Qwen3-8B) BLIP-2 captions only Captions only + MemAgent-7B Text LLM (Qwen2.5-7B) BLIP-2 captions only Captions only + Memory-T1 Text LLM (Qwen2.5-3B) BLIP-2 captions only Captions only + + +B.2 Metrics + +LLM-as-Judge (J). LLM-as-Judge accuracy [73] is our primary metric; a single canonical judge, +Qwen3-VL-235B-A22B-Instruct with thinking disabled, scores every accuracy number reported +in this paper. String-match metrics such as substring exact match fail across M EM L ENS because +answers span heterogeneous formats—binary choice, counts, currency and date values, ranked +orderings, short fill-ins, and explicit refusals—and LVLMs frequently wrap the correct answer in a +multi-sentence rationale or a thinking trace. The judge reads the question, reference answer, and raw +model output and emits a binary correct/incorrect verdict under task-specific criteria (Appendix B). +To prevent degenerate outputs from contaminating scores, the pipeline tail-truncates outputs beyond +6,000 characters, auto-zeros responses exceeding 500 parsed words, and instructs the judge to reject +circular reasoning. Judge reliability is quantified in Appendix E.2: the Qwen3-VL-235B judge +agrees with GPT-5.4-mini on 96.40% of a stratified 800-item re-judge sample (Cohen’s κ = 0.93; +Spearman ρ = 0.97 at the model level) and with a three-annotator human consensus on 93.60% +of 484 items (κ = 0.86), and the residual gap in either direction does not reorder the leaderboard. +A format-dependent leniency on very short (1–3 word) answers is documented and corrected in +Appendix E.2. + +Substring Exact Match (SE). The SE metric checks whether the normalized reference answer +appears as a substring of the model output. This rule is generous to verbose models but inflates scores +when the output contains reasoning traces that happen to mention the reference string. + +Coverage and Per-Answer Accuracy. Beyond overall accuracy, we decompose performance +into Coverage (Cov; fraction of the 699 answerable questions the model attempts) and Per-Answer +Accuracy (PA; accuracy on attempted answers only). The two components recover overall accuracy +via J ≈ (Cov × PA × 699 + ARcorrect ) / 789, where 699 is the answerable subset and 789 is the full +benchmark; the decomposition exposes the coverage–accuracy trade-off discussed in Appendix G.3. + +B.3 Infrastructure + +Local models are served via vLLM (v0.17–0.18) with FlashAttention-2 [89] on 8×A100-80GB nodes +with tensor parallelism for 128K inputs. API models use provider endpoints with concurrent requests +(4–8 threads). Generation length is set to 2,048 tokens for direct models and 16,384 for thinking +models to accommodate reasoning traces. + +C Dataset Construction Details +C.1 Problem Formulation + +An evaluation instance in M EM L ENS is the 4-tuple (S, q, I, a), where S = [(t1 , M1 ), . . . , (tN , MN )] +is a sequence of N time-stamped multi-turn sessions (t1 < · · · < tN ), each interleaving text and + + + 20 + images; V(S) denotes the set of all images appearing in S, and I ⊆ V(S) is the subset that carries +answer-critical visual information not recoverable from the surrounding text; q is a query targeting +one of five memory abilities; and a is the gold answer, or the literal string NOT_MENTIONED for +answer-refusal items. A correct system must (i) localize the relevant evidence sessions within a long, +distractor-heavy history and (ii) ground its answer in cross-modal reasoning over I. + +C.2 Topic Ontology + +Each of the four answerable memory abilities (IE, MSR, TR, KU) is backed by a dedicated ontology +of ∼100 topics organized into three complementary tracks: identification (recognizing real-world +entities such as products, landmarks, and dishes), experience (everyday activities and lifestyle +moments), and document (text-rich artifacts such as receipts, menus, and schedules). Each topic +is further expanded into ∼30 fine-grained subtopics, yielding more than 12,000 subtopics across +the four answerable types. Table 6 groups the de-duplicated topic titles into 19 sub-attributes, each +illustrated by representative leaves. + +Table 6: M EM L ENS topic ontology. Three tracks decompose into 19 sub-attributes; each row lists a + small sample of the de-duplicated topic titles drawn from the underlying ∼400-title pool. +Sub-attribute Representative topic titles +Track 1: Identification — recognizing real-world entities (∼40%) +1.1 Retail & commerce supermarket shelves; bakery counters; pharmacy aisles; hardware/tool shops; music + and sports stores; thrift stores; street food carts; fish and flower markets; secondhand + electronics stalls; warehouse club bulk aisles +1.2 Home objects & belongings kitchen appliances; knife and utensil drawers; pantry; spice jars; bookshelves; + CD/vinyl collections; gaming desk; TV streaming setup; closet; jewelry/watch layout; + makeup vanity; perfume shelf; skincare bottles; pet accessories; board games +1.3 Vehicles & mobility personal cars; scooters; motorcycles; bicycle helmets and locks; bike-sharing docks; + parked bicycles; license plates; parking lot vehicles; public transit vehicles; e-scooter + models on sidewalks +1.4 Urban environment & landmarks street signs; shopfront names; building numbers; mailboxes and doorbells; landmarks + and tourist attractions; street murals and sculptures; residential houses; recycling bins + and trash trucks; playgrounds; basketball courts; public swimming pools +1.5 Workspace & institutions office kitchen; supply closet; lobby directories; vending machines; public library + shelves; gym equipment; pharmacy windows; temple/church exteriors +Track 2: Experience — everyday activities and lifestyle moments (∼40%) +2.1 Dining & food activities casual restaurants; fast-food counters; food courts; cafes; street food walks; late-night + noodle shops; home cooking; weekend brunch; batch cooking; baking experiments; + cooking classes; asian grocery hunts; farmer market pickups +2.2 Fitness & outdoor recreation indoor climbing; badminton; trampoline park; futsal; table tennis; jogging routes; + soccer at local field; pickup basketball; public swimming; sunrise yoga; weekend + cycling; camping; hiking; beach outings; ski trips +2.3 Social & community gatherings house parties; potluck dinners; game and movie nights; karaoke rooms; casual bar + meetups; community festivals; charity runs; language exchanges; quiz nights; board + game cafés; dance socials; volunteer shifts; sports stadium visits +2.4 Cultural & entertainment outings art museum visits; science museum visits; gallery visits; historic house tours; concerts + and small gigs; movie theater; outdoor summer cinema; theme park rides; indoor + arcades; cosplay meetups; comic store browsing; photography walks +2.5 Routines & errands morning bathroom and kitchen routines; commuting by bus or train; weekend laun- + dry folding; apartment cleaning; grocery top-up trips; salon and barber visits; pet + grooming; ride-hailing pickups; fuel stops; DIY home repair; content drafting on + phone +2.6 Travel & mobility outings weekend road trips; vacation rental stays; hotel check-ins; airport and station transit; + riverside walks; coastal promenades; neighborhood evening jogs; weekend tram rides; + day-trip walks + + Continued on next page + + + + 21 + Table 6: M EM L ENS topic ontology. Three tracks decompose into 19 sub-attributes; each row lists a + small sample of the de-duplicated topic titles drawn from the underlying ∼400-title pool. + (Continued) +Sub-attribute Representative topic titles +2.7 Home life & indoor hobbies morning balcony coffee; evening living-room TV; evening reading corner; family + breakfast; hobby painting sessions; sewing and knitting; pottery class; home garden + balcony; video recording for reels +Track 3: Document — text-rich artifacts (∼20%) +3.1 Receipts & bills grocery self-checkout receipts; restaurant takeout receipts; monthly utility bills; taxi + and ride receipts; parking garage receipts; car service invoices; veterinarian receipts +3.2 Tickets, passes & itineraries event tickets and seating plans; museum day passes; transit monthly passes; intercity + bus tickets; hotel booking confirmations; train reservations; travel itineraries; gym + class punch cards; university club ID cards +3.3 Schedules & calendars school and work weekly schedules; printed school lunch menus; cinema film sched- + ules; fitness class timetables; chore rotation charts; carpool rotation sheets; weekly + meal plans; fitness challenge calendars +3.4 Contracts & application forms rental leases; job offer letters; gym membership contracts; international visa packets; + school field trip permission slips; gym class registration forms; university club + membership forms; workplace sign-in sheets +3.5 Personal records & logs personal budget sheets and bank statements; fitness tracker logs; meal tracking + notebooks; handwritten to-do lists; travel packing checklists; recipe cards; phone + contact lists; household grocery expense notebooks +3.6 Subscriptions & digital records streaming subscription billing emails; digital movie rental history; online shopping + order screenshots; streaming watch history lists; social media profile/settings screens; + social media content calendars +3.7 Health & service records medical prescriptions and pharmacy labels; clinic appointment cards; home appliance + warranty cards; smartphone repair invoices; household appliance repair job cards; + product warranty manuals + + + +C.3 Subtype Detail + +Table 7 provides the complete subtype taxonomy of M EM L ENS, listing the 8 answerable subtypes +plus Answer Refusal with per-subtype question counts, the visual skill or reasoning operation each +subtype isolates, and a representative example. Across the benchmark, 65.7% of questions are +image-essential (the evidence image is required to recover the answer), 14.7% are image-supportive +(the image confirms or disambiguates a textual fact that a strong text-only model could otherwise +guess), and 19.6% are text-sufficient (all AR questions plus a subset of MSR items retained by design). +The image-essential share is highest in IE and MSR, substantial in KU, and lower in TR, where a +portion of items renders the temporal cue as explicit textual dates or session-boundary timestamps. +The cross-subtype correlation structure (Figure 6) confirms that the nine subtypes do not consistently +correlate with each other, supporting per-type evaluation rather than a single aggregate. +Table 8 presents representative examples from each question type, illustrating the cross-modal +reasoning chain required to reach the correct answer. + +C.4 Question Construction Pipeline + +The question construction pipeline produces cross-modal evaluation questions from a topic ontology +through four steps: background generation, entity selection with image retrieval, entity abstraction, +and question generation. This pipeline is shared across all four answerable question types (IE, MSR, +TR, KU), with type-specific adaptations described in the per-subtype routes at the end of this section. + +Background generation. For each question, a topic is sampled from the hierarchical ontology +(§C.2), and Gemini-3-Pro generates a background paragraph of three to five sentences grounded in that +topic. The paragraph is required to contain at least two named entities drawn from recognizable real- +world referents, such as landmarks, commercial products, biological species, or cultural institutions, +whose visual appearance is distinctive and web-searchable. For example, a paragraph about San + + + 22 + Table 7: Complete subtype taxonomy of M EM L ENS (n = 789). Each subtype isolates a distinct +visual skill or reasoning operation. Example questions are shortened for space; full versions appear +in the released dataset. + Subtype Type n Skill Tested / Representative Example + Entity IE 120 Identify a visually grounded entity via two-hop + reasoning (disambiguation, alignment, counting, + spatial, arithmetic). “What phrase does that mes- + sage spell out?” + PrevInfo IE 126 Recall a visual detail from an earlier session’s + image (screenshot, app interface, photo). “What + color is the engine bay painted?” + Arithmetic MSR 50 Sum or compute over prices/quantities scattered + across 3–8 sessions. “How much total have I spent + on weights?” + Counting MSR 46 Count entities matching a criterion across sessions. + “How many cats do I have?” + Entity MSR 47 Resolve whether two cross-session references de- + note the same entity, either by counting distinct + entities or by Y/N identity matching. “Is the new + bird the same species as Rio?” + Duration Cmp TR 91 Compare two durations derived from session + timestamps and visual cues (clocks, calendars). + “Which duration is longer?” + Temporal Grounding TR 103 Locate when an event occurred—either by sort- + ing events chronologically or extracting a specific + date—using session timestamps, textual dates, + and clock/calendar images. “Sort these facts in + chronological order.” / “When did I complete X? + (YYYY/MM/DD)” + Update KU 116 Track a 4-fact preference chain and report the cur- + rent state, distinguishing it from outdated values. + “What do I prefer now?” + Refusal AR 90 Decline to answer when the evidence image + has been deliberately removed from the context. + “How many cables are visible?” (image absent) + Total 789 + + + +Francisco landmarks might mention the Golden Gate Bridge, Alcatraz Island, and Fisherman’s +Wharf, providing both visual anchors and textual context from which questions can be derived. The +background paragraph serves a dual purpose: it supplies the factual context that the question will +target, and it introduces the named entity that will undergo abstraction in the next stage. + + +Entity selection and image retrieval. From the background paragraph, one named entity is selected +as the visual anchor for the question. The selection requires that the entity be visually identifiable in +a photograph, sufficiently specific that a web image search returns relevant results, and critical to at +least one fact stated in the paragraph. A text query derived from the entity name is then issued to +the web-crawling pipeline, which retrieves a batch of candidate photographs using iCrawler. Each +candidate is scored by the multi-model filter described in §C.5, combining CLIP (ViT-L/14), SigLIP +(ViT-SO400M), and a text–text cosine similarity channel between the query and a BLIP-2 caption of +the candidate image. The highest-scoring candidate that passes negative-content filtering and global +URL deduplication is selected as the evidence image. This retrieval pipeline is identical to the one +used for haystack images (§3.2), ensuring uniform visual quality across evidence and background +content. The selected entity may range from a famous landmark to a specific product model or a +biological species. + + + 23 + Table 8: Representative evaluation questions from M EM L ENS, one per question type. Each question +requires joint visual and textual reasoning: the evidence image carries the discriminative information +that the surrounding text deliberately withholds. Fact sketches are abbreviated; full evidence sessions +appear in the released dataset. + Type Question Answer Why Multimodal + IE What shape is the Mount Fuji detail in Triangular Resolve “the artwork” from + the artwork my Tokyo friend was study- of The Great Wave off Kanagawa, then + ing? extract visual shape. + MSR How much total have I spent on Arkham $124.94 Sum $59.99 + $64.95 from two ses- + Horror expansions recently? sions; one price is visible only on the + box . + TR Which duration is longer: my time living A London start date is on a boarding + in London vs. my stint as a barista? pass ; barista dates are tex- + tual. Compare two durations. + KU Back when I liked it mainly for quick, Farro Track a 4-image preference chain; + filling meals, what was my favorite? each update is anchored by a differ- + ent grain . + AR How many vertical cables are visible on NOT_MENTIONED The bridge photograph has been delib- + one side of the bridge? erately removed; a correct model must + refuse. + + + +Entity abstraction. The surface form of the selected entity is then masked in the background +paragraph and replaced with a natural anaphor that references the evidence image [19, 64, 90]. +The replacement proceeds in two stages. The entity is first classified into one of 55 semantic +categories spanning places (museums, parks, restaurants, bridges, temples), organizations (companies, +foundations, institutes), objects (books, paintings, vehicles, instruments), and generic fallbacks. A +replacement phrase is then sampled from a type-aware dictionary of ∼170 entries, with each category +providing three to four paraphrase variants to ensure lexical diversity across questions. For instance, +“Golden Gate Bridge” may be replaced with “the bridge shown in ,” while “Portland Art +Museum” might become “the gallery I visited, shown in .” When the entity type cannot be +matched to any dictionary category, a generic fallback such as “the place shown in ” is used. +After replacement, the background paragraph no longer contains the entity name. The anaphor is +deliberately under-specified: “the bridge” could refer to any of thousands of bridges worldwide, +and only the accompanying evidence image allows the reader to identify the specific referent. This +design enforces a cross-modal dependency in which the text provides the factual context (e.g., year +of construction, architectural style) while the image provides the entity identity, following the visual +information-seeking paradigm established in prior work [64, 90]. +For KU questions, dictionary-based replacement alone is insufficient because the evolving attribute +chain tracks concrete items within a single category (e.g., successive favorite fruits). In this case, an +LLM rewrites each evidence fact with a short sensory or visual descriptor of at most five words that +does not name the category. For instance, “blood orange” becomes “this tangy round thing ” +and “blueberries” becomes “these tiny purple spheres .” The descriptor is constrained +to be plausible for multiple items within the same category, preserving the ambiguity needed for +cross-modal dependency. + + +Question generation and quality verification. The abstracted background paragraph, the evidence +image, and the original entity name are provided to Gemini-3-Pro, which generates a (question, +answer) pair together with one or more atomic evidence facts. The generation prompt enforces two +constraints: the question must be answerable only when both the image and the surrounding text +are available, and the answer must be derivable from the evidence facts without requiring external +knowledge beyond the provided context. For MSR questions, a three-layer text-hackability defense +additionally verifies that no textual cue leaks the entity identity: anti-leakage prompt rules prevent +the generation model from naming the entity, a rule-based pre-filter rejects facts containing the entity +name or close synonyms, and an LLM text-only judge confirms that the answer cannot be derived + + + 24 + Subtype Spearman Rank Correlation (32k) + + IE: Entity 1.00 0.87 0.41 0.04 -0.03 0.05 0.38 0.68 0.40 0.350.30 + + + + IE: PrevInfo 0.87 1.00 0.32 -0.05 -0.04 0.05 0.37 0.62 0.34 0.310.30 + + + + MSR: Arithmetic 0.41 0.32 1.00 0.30 -0.13 -0.28 0.34 0.68 0.55 0.270.30 + + + + MSR: Counting 0.04 -0.05 0.30 1.00 0.44 0.45 0.04 0.41 -0.05 0.200.21 + + + + MSR: Entity Resolution -0.03 -0.04 -0.13 0.44 1.00 0.60 0.27 0.09 -0.26 0.120.27 + + + + TR: Duration Cmp 0.05 0.05 -0.28 0.45 0.60 1.00 0.06 0.02 -0.44 0.060.32 + + + + TR: Temporal Grounding 0.38 0.37 0.34 0.04 0.27 0.06 1.00 0.54 0.66 0.330.20 + + + + KU: Update 0.68 0.62 0.68 0.41 0.09 0.02 0.54 1.00 0.59 0.460.24 + + + + AR: Refusal 0.40 0.34 0.55 -0.05 -0.26 -0.44 0.66 0.59 1.00 0.220.39 + c + y + + + + + ion + + + + + ing + + + ate + + + + l + fo + + + + + ng + + + + + mp + + + + + td + sa + eti + tit + + + + + gs + vIn + + + + + nti + + + + + efu + lut + + + nC + + + nd + + + pd + + + + + Av + En + + + + + hm + Pre + + + + + u + + + + + ou + + + :U + so + + + + + :R + tio + Co + IE: + + + + + rit + + + + + Re + + + + + Gr + + + KU + + + AR + ura + IE: + + + A + + + R: + + + + + ral + R: + + + + + ty + MS + + + + + :D + MS + + + + + nti + + + + + po + TR + E + + + + + em + R: + + + + + :T + MS + + + + + TR + + + +Figure 6: Cross-subtype Spearman rank correlation across the evaluated models (n = 789, 9 reporting +subtypes). IE Entity and IE PrevInfo form the only near-ceiling pair (ρ = 0.87 at 32K, 0.94 at 128K), +reflecting their shared retrieval skill. MSR-internal correlation is weak at 32K (mean pairwise +ρ = 0.20) and rises to 0.38 at 128K as a shared-failure artifact of MSR collapsing toward the floor; +TR-internal correlation stays near zero at both contexts (ρ = 0.06). The heterogeneity supports +evaluating all five major types separately rather than reporting a single aggregate (§4.3). + + +from the textual evidence alone. All generated questions then pass through the automated filtering +and human review pipeline described in §3.3. + +Per-subtype generation routes. The five question types follow different generation paths, each +tailored to the visual skill under test: + + • IE – Entity two-hop (5 subtypes, n = 120). An LLM generates a two-hop chain [91]: + the first hop resolves the entity from the evidence image, and the second hop retrieves a + property of that entity from the surrounding text. The five subtypes vary the visual skill + required for the first hop: disambiguation (distinguishing similar-looking entities), alignment + (matching an image to a textual description), counting (enumerating items in the image), + spatial reasoning (locating objects relative to each other), and arithmetic (computing from + visually presented numbers). + • IE – PrevInfo (3 subtypes, n = 126). The question asks about a visual detail from an image + shared in an earlier conversation session. The three subtypes correspond to the image source: + a screenshot of a chat interface, an app or web interface, or a natural photograph. Entity + abstraction is applied to the session reference rather than the entity itself. + • KU – Update (n = 116). A 4-fact atomic chain is generated: each fact updates a user + attribute (e.g., favorite drink changes from tea → coffee → matcha → espresso martini). + The question asks for the current state, requiring the model to locate all four updates and + identify the latest. Entity abstraction masks the attribute’s anchoring entity so the evidence + image is needed to identify which preference chain is being queried. + • MSR (3 subtypes, n = 143). Facts are distributed across 3–8 sessions. Arithmetic (n = 50) + requires summing prices or quantities; Counting (n = 46) requires enumerating entities + + + 25 + matching a criterion; Entity (n = 47) requires determining whether two cross-session + references denote the same entity, either by counting distinct entities or by Y/N identity + matching. A 3-layer text-hackability defense (anti-leakage prompt rules, rule-based pre- + filter, LLM text-only judge) ensures that the answer cannot be derived without the evidence + images. + • TR (2 subtypes, n = 194). Duration Comparison (n = 91) derives two durations from + session timestamps and visual cues (clocks, calendars) and asks which is longer. Temporal + Grounding (n = 103) bundles two operations: order ranking (n = 24, sort events chronolog- + ically) and date extraction (n = 79, answer “When did X happen?” in YYYY/MM/DD). Three + generation modes cross with these operations—Mode B renders the temporal cue itself as a + visual artifact (clock, calendar, receipt), Mode C pairs an entity image with explicit textual + dates, and Mode D pairs an entity image with session-level timestamps that serve as implicit + temporal anchors—providing comprehensive coverage of temporal–visual integration. + +C.5 Image Filtering + +A unified cross-modal image filter is applied to every image in M EM L ENS—both needle images in +evidence sessions and filler images in haystack sessions—to maintain consistent visual quality across +the benchmark. The filter operates in two stages. + +Stage 1: Multi-channel relevance scoring. Each candidate image is scored against its associated +textual query on three independent channels: CLIP [92] (ViT-L/14), SigLIP [93] (ViT-SO400M), and +a text–text cosine between the query and a BLIP-2 [72] caption of the image. The CLIP threshold set +to 0.30. A candidate must exceed the CLIP threshold and at least one of the two secondary channels +to pass. + +Stage 2: Negative-content filtering. Candidates that pass relevance scoring undergo a negative- +keyword filter that rejects images containing watermarks, stock-photo logos, copyright overlays, or +resolution artifacts. For DocVQA-style images [94] (receipts, menus, forms), a separate multimodal +judge (GPT-4V) inspects each image for watermark presence and rejects any image flagged as +containing artifacts that could distract from the document content. + +Deduplication. A persistent URL registry tracks every image URL used across all generation +batches, enforcing global image uniqueness: no two questions in M EM L ENS share the same source +image. Within each question, duplicate detection uses perceptual hashing (pHash) with a Hamming +distance threshold of 6 to reject near-duplicate candidates from the same web-retrieval session. + +C.6 Image Diversity + +Unlike prior multimodal long-context benchmarks that categorize visual content by overlapping +semantic topics (mixing scenes, media types, and tasks on a single axis), M EM L ENS partitions +images by their dominant visual format, with categories selected to cover distinct perceptual regimes +(Table 9). +The first and largest category, natural photographs, covers real-world scenes, objects, food, and +lifestyle items retrieved through a CLIP-filtered web pipeline. This category tests open-world +object and scene recognition, spatial understanding, and visual attribute extraction in unstructured +environments, and supplies the visual evidence for most IE, MSR, and KU questions. The second +category, text-rich documents, includes receipts, menus, invoices, posters, price tags, and forms in +which dense text and spatial layout carry the answer; reasoning over these images requires OCR +together with layout grounding and text–visual alignment, the regime targeted by IE Entity alignment +questions. The third category, digital and symbolic interfaces, covers app and web screenshots, chat +interfaces, and synthetic clock and calendar renderings. These test interface-layout parsing and +reading of synthetic symbols (clock hands, calendar grids, UI widgets) rather than natural-image +recognition: an analog clock face or a settings panel exercises different perceptual regimes than a +photograph. IE PrevInfo questions draw on screenshots, while TR Duration Comparison relies on +synthetic clocks and calendars. +This distribution emerges from the CLIP-filtered web-retrieval pipeline rather than from manual +balancing. Each category is anchored by its own topic subset in the ontology (§C.2), and images are + + + 26 + retrieved per topic rather than per target proportion. As a result, M EM L ENS evaluates multimodal +evidence understanding across complementary perceptual regimes, and avoids the narrow object- +centric or OCR-only focus of prior benchmarks. + +Table 9: Image categories in M EM L ENS, partitioned along a single capability-driven axis (dominant +visual format). Each row corresponds to a distinct perceptual regime that the benchmark exercises. +Category Description Primary Types +Natural Photographs Scenes, places, objects, IE, MSR, KU + food, and lifestyle items. +Text-Rich Documents Receipts, menus, invoices, IE Entity (alignment, receipt), MSR + posters, price tags, and + forms where dense text and + spatial layout carry the an- + swer. +Digital & Symbolic Interfaces App and web screenshots, IE Assistant PrevInfo, TR Duration Comparison + chat interfaces, and syn- + thetic clock and calendar + renderings. + + + +C.7 Image Sourcing, Licensing, and Release + +Sourcing. Every image in M EM L ENS originates from a public web image search issued through +iCrawler, with queries derived from the topic ontology (§C.2) or the entity-abstraction slot for +evidence images. iCrawler queries are issued against general-purpose image search rather than +commercial stock aggregators, and the relevance filter described in Appendix C.5 is applied uniformly +to evidence and haystack images. + +Negative-content filtering. The negative-content filter rejects candidates carrying watermarks, +stock-photo logos, copyright overlays, or resolution artifacts before any image is admitted into the +benchmark, so that images exhibiting commercial-source markers are excluded at retrieval time +rather than redistributed downstream. For DocVQA-style images (receipts, menus, posters, forms), +an additional multimodal judge [95] inspects each image and rejects any flagged as containing +watermarks or layout-distorting artifacts. + +Privacy and identifiability. The topic ontology that drives image retrieval (§C.2) covers objects, +places, products, text-rich documents, screenshots, and synthetic interfaces; no topic and no entity- +abstraction slot is person-centric. iCrawler queries are derived from these topic and entity names +rather than from any individual’s name, and the retrieval pipeline performs no face- or identity- +based search. Natural-photograph queries (e.g., “Times Square at dusk”) may include incidental +human figures in the background, but the benchmark never targets identifiable individuals, and the +construction prompts do not request the model to identify, name, or describe any depicted person. +A takedown contact in the project repository allows seven-day removal of any image that, on user +report, is found to reveal an identifiable individual. + +Per-image metadata and datasheet. For each admitted image, the construction pipeline records +its source URL, retrieval query, retrieval timestamp, BLIP-2 caption [72], CLIP and SigLIP relevance +scores, and perceptual hash, and a persistent URL registry enforces global uniqueness across genera- +tion batches so that no two questions in M EM L ENS share the same source image. A datasheet for +datasets [96] accompanies the release at the same URL, documenting motivation, collection process, +intended uses (evaluation only; see Appendix H), and licensing. + +Release, license, and takedown. The 4,695 unique images referenced across the four context- +length datasets are distributed alongside the dataset files at https://huggingface.co/datasets/ +xiyuRenBill/MEMLENS, together with the per-image metadata listed above (source URL, retrieval +query, retrieval timestamp, and perceptual hash). The release is versioned with frozen tags so that any +specific evaluation run remains reproducible. The author-produced artefacts of M EM L ENS—dataset +annotations (questions, evidence facts, abstracted paragraphs, consensus labels), per-image metadata, + + + 27 + prompt templates, and human-annotation records—are released under CC-BY-4.0; the evaluation +harness and supporting code at https://github.com/xrenaf/MEMLENS are released under MIT. +Third-party images retrieved from public web search are not relicensed by the authors and remain +governed by their original source-site licenses; we redistribute them solely to support reproducibility +of the evaluation, and the per-image provenance metadata enables downstream users to independently +verify or re-fetch the original source. A takedown contact is provided in the project repository, and +any flagged image is removed within seven days. + + + +C.8 Conversation History Assembly + +Evidence sessions are inserted into the full conversation history with positions chosen uniformly at +random, except for KU questions where, while the position remains random, the relative order of +the evidence sessions is kept the same since it is the user preference update order and critical to the +answer. + + + +D Data Examples + +D.1 Information Extraction + + + + + Example Reasoning Chain — IE - Entity + + Main Question. Examine the readout and provide the temperature value. + Answer with a short phrase only, no explanation. + + Final Answer. 24 + + + + Evidence Facts: + 1.(Image) I’m trying to make sense of which thermostat details actually matter before I start changing + settings, especially since I don’t want comfort fixes that quietly raise my bill. I’ve been looking at the + thermostat setting on the left side of the display to make sure the AC isn’t working harder than it needs to. + Here’s a photo of me checking smart home impact on my electricity bill in Singapore + . Based on that kind of setup, what would you pay closest attention to on the screen or in the menu + to tell whether the fan and cooling behavior are set efficiently? + + + + +Figure 7: Sampled IE-Entity questions. The visually grounded entity is abstracted in the question text, +so the agent must first identify the entity from the evidence image before retrieving the relevant fact. + + + + + Example Reasoning Chain — IE - PrevInfo + + Main Question. What does the large text on the green wall of the supermarket say? + Answer with a short phrase only, no explanation. + + Final Answer. Produce + + + + + Evidence Facts: + + 1. (Image) Okay, I'm staring at this wall of fruits and veggies, but what I actually want to understand is the + labels at the vegetable counter . Can you walk me through how those stickers usually break down + price per pound, weight, and total cost so I can check that the register charge matches? + + + + +Figure 8: Sampled IE-PrevInfo questions. The answer is a visual detail (color, count, layout, on- +screen text) of an image shared in an earlier session, requiring multi-session image recall. + + + 28 + D.2 Multi-Session Reasoning + + + + + Example Reasoning Chain — MSR - Arithmetic + + Main Question. How much total have I spent on German + language board games this month? + Answer with the exact amount (e.g., "$45.00") only. + Final Answer. $80 + Evidence Facts: + + 1. (Text) Those examples are exactly the kind of thing I’m trying + to internalize so I don’t freeze mid-sentence during a heated + turn. I’m especially interested in getting the nuances right + between friendly teasing and something that sounds too harsh or + formal, since tone feels tricky to judge in a second + language. Speaking of which, I was organizing my + games on my little Vienna kitchen table and + noticed I’d dropped about 35 dollars on a German copy + of Die Siedler von Catan, but it actually feels worth it just + to have everything written in the original German. Could you + walk me through some core vocabulary for components and + actions in both German and Spanish (like pieces, tiles, draw a + card, roll the dice, shuffle, discard)? + + 2. (Image) I’d definitely like to dig into the sustainable + materials angle, especially where there’s tension between “luxury + feel” and “eco” cues in what people keep. It’s interesting because + I catch myself doing this with games: if the box art feels special + or a bit different, I’m way more inclined to keep that version. I + just added another board game to the “stay” stack while deciding + which copies to sell, and the one I’m keeping has slightly + different cover art that feels more collectible to me. I actually + have a picture of the stack with that game on + top here, and I’m curious how much subtle artwork variants + like this play into perceived long-term value or “display- + worthiness.” + + 3. (Text) The level of detail you're giving on what to look/listen + for is exactly the kind of thing I worry I'm missing + when I'm tempted by a "deal." Speaking of which, I've noticed + how easy it is for me to rationalize prices when I'm in "collector + brain" mode — like that game sitting on top of the stack + in the picture I showed you earlier. I grabbed it earlier + this month for about $45, which felt steep, but I convinced + myself the German edition was worth it to round out the set + alongside my Catan copy. How would you go about building + amental cheat sheet for common repair ranges on the gear side + so I can factor that into my buying decisions in real time? + + + + +Figure 9: Sampled MSR-Arithmetic questions. The agent sums or computes over prices, durations, +or quantities scattered across sessions; at least one operand is visible only in an image. + + + 29 + Example Reasoning Chain — MSR - Counting + + Main Question. How many educational board games do I + currently have for the kids? + Answer with a single number only. + + Final Answer. 4 + Evidence Facts: + 1. (Text) I’m really liking this whole idea of turning classic logic + puzzles into things the kids can literally walk around inside; it + feels way more likely to keep them engaged after a long school day. + A life-sized Rush Hour setup sounds like exactly that kind of thing, + and I’m curious how you’d structure the “cars” and rules so it stays + strategic rather than devolving into chaos-tag. Speaking of which, I + recently bought a Funskool Brainvita peg-jump puzzle + for my oldest to stretch his logic muscles a bit, + and it’s already sitting on the playroom shelf waiting for its turn in + the rotation. How would you adapt the Rush Hour-style puzzle for, + say, two kids plus an adult in a small living room space? + + 2. (Text) I really like the idea of turning the older kids into + “teachers” so they’re stretched instead of just racing to answer first; + that feels like it would shift the mood from competition to + collaboration. I’m especially interested in game formats that we can + set up quickly after dinner without a lot of prep or printing. By the + way, with my youngest I’ve been using a homemade “sight + word bingo” board pretty often, where she has to read + and cover simple English words, and it’s been one of the few + things she’ll willingly stick with. Could you suggest a few + concrete word lists or themes that would work well for these + mixed-level games—maybe by age band or difficulty level? + + 3. (Text) I like the idea of badges and a ship’s log because my kids + really respond when something feels “official” and ongoing, not + just a one-off game. I’m curious how a reward store might fit into + this without making it feel like they’re only doing the missions for + prizes instead of the story. Speaking of which, my sister gave + them a board game called Sum Swamp for their + birthday, and it’s been a surprisingly big hit for + practicing adding and subtracting without them + noticing it’s “math.” How could I borrow the feel of that kind of + game—lots of turns, simple decisions, a bit of luck—and blend + it into a points or reward system for these campaigns? + + 4. (Image) That comparison between legacy-style campaigns and + something like Hogwarts Battle is really helpful, especially the part + about kids worrying less if “failure” is framed as part of the + story. I’m still trying to balance that desire for meaningful + consequences with not turning game night into a source of stress, + especially for my more anxious kid. I just added another game + to our collection on the playroom mat today, and it + seems like a solid fit for their age range—I actually have a + picture of it here on the mat with the other + games . Based on what you can see in the photo and + what we’ve been talking about, how would you ease + kids into campaign/legacy elements so they stay excited about the + ongoing story without fixating on “messing up” permanently? + + + +Figure 10: Sampled MSR-Counting questions. The agent counts how many sessions or items match a +given criterion across the conversation history. + + + + + 30 + Example Reasoning Chain — MSR - Entity Resolution + + Main Question. Is the new creepy plant the same type of plant + as Monty? + Answer with "Yes" or "No" only. + Final Answer. Yes + Evidence Facts: + 1. (Text) I love these archetypes; they make it so much easier to + “read” what a plant is asking for without turning it into a chore + chart in my head. The idea of the respectful nod for snake + plants especially makes me laugh because it’s exactly how I already + treat them without really thinking about it. Speaking of + dramatic plants, I’ve got a big Monstera Deliciosa in my + NYC place that I’ve been growing for years called Monty, + and he’s just started putting out those fenestrated leaves and feels + like this slowly emerging prima donna in the corner. Could you go + a bit deeper into example names/rituals for a few other common + types—like ZZ plants, philodendrons, or prayer plants—and then + maybe circle back to what you mentioned about handling the “end + of life” phase with named plants? + 2. (Text) I'm trying to design a really immersive, horror-themed + apartment setup for a classic scary movie night, and I want it to feel + more like an experience than just people sitting on the couch. For + example, I'm turning the living room into a mini Overlook Hotel + for our Halloween screening of "The Shining," and I even picked + up this oversized, ominous-looking plant with big holes + in its leaves that matches all the Spirit Halloween props I put + around the room — I'll show you a photo of it in a sec so you + can see what I mean. I'd love help thinking through how to tie + all the elements together so it feels cohesive and not just like a pile + of random decorations. Can you suggest ways to structure + the space, lighting, and small details so the whole evening feels like + stepping into the movie's world? + 3. (Image) The way certain textures feel like they "don't belong" + in a supposedly cozy space is exactly what I'm trying to play with + for this same gathering. I'm especially curious how a single object + with a strange texture or silhouette can tip a room's mood toward + uncanny even if everything else is pretty + normal. Here's that creepy Shining-room plant I was just + describing — you can seethe slightly sinister, + otherworldly vibe I was talking about. I'd love your take on how + something like this might read in a photo depending on lighting + and background textures around it. + + 4. (Text) I'm really into this idea of using the white walls almost + like a projection screen for shadows instead of just blasting the + room with overhead light. I'd love to get more specific about bulb + choices and color temperatures, especially for keeping things cozy + but still bright enough to read or work at night. Speaking of + which, that same ominous Shining-room plant from the + photo has earned a permanent spot in the center of the + living room now, because when I dim the lights its outline on + the wall turns into this really dramatic, sculptural shadow— those + big fenestrated leaves throw such a recognizable silhouette. How + would you mix warm vs. neutral bulbs in a small space like that so + the silhouettes stay strong but the room doesn't feel gloomy or + yellowed? + + +Figure 11: Sampled MSR-Entity Resolution questions. The agent decides whether two cross-session +references denote the same entity, either via Yes/No identity matching or by counting distinct entities. + + + + + 31 + 32 + D.3 Temporal Reasoning + + + Example Reasoning Chain — TR - Duration (Comparison Explicit + Temporal Information) + Main Question. Which of the following two durations is + longer? + *Duration 1:* The time I lived in New York City + *Duration 2:* The time I spent working on my freelance writing + project + A. Duration 1 is longer. + B. Duration 2 is longer. + Answer with "A" or "B" only. + Final Answer. A + Evidence Facts: + 1. (Text) That distinction between restless avoidance and actual + fatigue really lands for me, because my evenings tend to flip + between those two and I don't always know which one I'm in + until I've already lost a couple hours. I also like the idea of a five- + minute test, but I think I'd need something even more concrete in + the moment when my brain is foggy. Speaking of which, I came to + New York City on June 9, 2009, feeling like I was stepping + into an entirely new phase of my life, so this whole “unsettled after + work” feeling is hitting something familiar for me. Could you give + me maybe 3 really simple self-check questions I can ask myself at + the end of the workday to decide whether I need rest, grounding, or + a little structure? + 2. (Image) I like the idea of smaller intentional moments more + than one big send-off, because I think I'd get overwhelmed by a + formal farewell party. I'm especially drawn to something that + leaves people feeling connected instead of emotional in a stiff or + awkward way. My own big departure felt very real when my long + city adventure finally ended and I went to the airport for my + flight back to London, and I actually have a image of the + boarding pass from that day. How would you turn + something like that into a simple farewell ritual or keepsake + without it feeling too sentimental or forced? + 3. (Text) That makes sense, especially the part about shifting from + urgency to impact, because I tend to blur those together + when I'm looking at a crowded list. I can usually + tell what's important in theory, but in the moment every task starts + making its own case and I end up second-guessing myself. + Speaking of which, during a stay away from home I took on a + freelance writing assignment and formally kicked it off + on March 6, 2013, and I remember having the same problem of + not knowing whether to protect the writing time or clear the + smaller obligations first. How would you set up a simple decision + rule for days when paid work, personal creative goals, and admin + all feel like they need attention right now? + 4. (Text) I think the part I struggle with most is whether I need a + little more time for the pattern to become obvious, or whether + that’s just me rationalizing extra effort. I’d like something even + more concrete, maybe a very short scoring rule or threshold I can + use at the end so I’m not relying on mood in the moment. Come to + think of it, I had a small project once that didn’t last long, + and I officially closed it out on March 22, 2014. Could you + turn your checklist into a tiny yes/no decision tree I could reuse for + habit tests and workflow experiments? + + +Figure 12: Sampled TR-Duration Comparison questions. The agent compares two time spans whose +endpoints come from a mixture of textual dates, session timestamps, and visual cues. + + 33 + Example Reasoning Chain — TR – Temporal Grounding (Implicit Temporal Information) + + Main Question. When did I start living near Lake Zurich? + Answer with a date in YYYY/MM/DD format only. + + Final Answer. 2019/03/01 + + Reasoning Chain: + 1. (Image) That ridge-above-the-water idea is exactly the kind of tradeoff I'm trying to picture, because I + want the views without ending up on a route that floods or gets jammed every holiday weekend. I'm also + wondering how much difference there is between a town where the lake is kind of beside your commute versus + one where the main road hugs the shoreline the whole way. Starting that spring, I really loved living near the + water because it made my drive to work feel a lot more scenic, and I have a picture I took attached here + . When you're comparing places like that, what signs usually tell you the scenery is worth it without + the day-to-day hassle becoming a pain? + 2. (System Timestamp) + Session Date: 2019/03/01 + + + + +Figure 13: Sampled TR-Temporal Grounding questions, including chronological ordering and +absolute date extraction. The temporal cue is sometimes available only as an image of a clock face or +a calendar page. + + + + + 34 + D.4 Knowledge Update + + + + + Example Reasoning Chain — KU - Update + + Main Question. What is my favorite hat now? + Answer with only the name or phrase, no explanation. + Final Answer. baseball cap + Evidence Facts: + + 1. (Image) I think the hard part for me is keeping the system + short enough that I’ll actually look at it when I’m rushing out the + door. Since I switch bags so much, I’m realizing I probably + need one visual cue for the pouch and a separate reminder for + the weather-dependent stuff. At first I was really into this + kind of hat that my son wore one time. Here’s a photo + of it —would you put something like that in the “add + fresh each night” category, or is there a smarter way to build it + into the checklist without making the list longer? + + 2. (Image) I’ve been thinking about the small accessories that + anchor a daily ritual, like how one object can make the start of + the day feel more settled and intentional. For me it’s less about + fashion and more about having a familiar thing that + signals I’m in my usual rhythm. At one point, I began + carrying a felt hat with a brim around with me, and it + oddly became part of the routine I liked the + most. Here’s a photo of this kind of + hat,and I’m curious what kinds of everyday + accessories tend to create that same ritual feeling for other + people. + 3. (Image) I’ve been thinking a lot about how my clothes kind of + mirror my mood, and I’m curious whether that’s actually + a common thing or if I’m just reading too much into it. Here’s a + photo of the hat that I currently like the most, + since accessories seem to be a big part of this for + me. Lately, that big hat of mine has become one of those pieces I + pick out more intentionally depending on how I’m feeling. + Can you help me understand why fashion choices can feel so tied + to emotional state? + 4. (Image) I’ve been realizing that certain comfort items make + me feel way more put-together and confident when I’m out, even + if they seem small. For me, it’s mostly hats and a couple of + familiar accessories, and here’s a photo of my favorite type + of hat now so you can see the style I mean. + Honestly, that hat has become such a go-to that I feel a little off + leaving the house without it. Can you help me understand + why something like that can feel so reassuring and confidence- + boosting? + + + + +Figure 14: Sampled KU-Update questions. A four-step preference chain is anchored by a different +image at each step; the gold answer is always the most recent state. + + + 35 + D.5 Answer Refusal + + + Example Reasoning Chain — AR - Refusal + + Main Question. Which fast food brand has its logo displayed on the black sign directly above the store + where I bought my jacket? + + Final Answer. Insufficient information + + + + +Figure 15: Sampled AR-Refusal questions. The supporting evidence has been deliberately removed +from the conversation history, so the gold answer is a refusal phrase rather than a content answer. + + +E Quality Assurance +E.1 Annotation Guidelines and Quality Assurance + +Human review is organized in three sequential rounds, each targeting a different granularity of the +benchmark construction pipeline. + +Round 1: Question-level review. Four annotators participate in Round 1; each of the n = 20k +generated candidates is independently reviewed by two of the four. For each question, both reviewers +verify: + + 1. The evidence image carries information vital to the answer—the question cannot be answered + from text alone. + 2. The question text does not leak the answer (e.g., the entity name does not appear in the + question after abstraction). + 3. The answer is unambiguous given the evidence. + 4. The difficulty is calibrated: trivially easy items (e.g., “What color is the sky?”) and + impossibly hard items (requiring domain expertise beyond the conversation) are flagged for + revision or removal. + +Each of the two reviewers marks the item as accept, revise, or reject. Items marked revise by either of +the two reviewers are rewritten to address the flagged issue and re-reviewed. Items marked reject +by both reviewers are removed. Inter-annotator agreement on the accept/reject decision is κ = 0.78 +(Cohen’s κ [97]; n = 200 items sampled from the double-coded pool), indicating substantial +agreement. + +Round 2: Session-level review. Two annotators read every evidence session end-to-end (n = 2,145 +sessions after Round 1 filtering, an average of 2.7 evidence sessions per question; see Table 2) and +verifies: + + 1. All needle facts are present in the session dialogue and recoverable from it. + 2. Needle facts are distributed across diverse conversational positions—not concentrated in a + single turn. + 3. The session reads naturally as a plausible user–assistant conversation. + 4. The evidence image is placed adjacent to the corresponding textual mention, ensuring + unambiguous image–text co-reference. + +Sessions that fail any criterion are returned to the generation pipeline for regeneration with adjusted +constraints. + +Round 3: Haystack auditing. Two annotators inspect a stratified random sample of 500/689 +multimodal haystack sessions (stratified by topic track: identification, experience, document) and +verifies: + + + 36 + 1. The haystack image is relevant to the topic of the conversation and the surrounding textual + context. + 2. The haystack session does not accidentally contain information that could serve as a spurious + answer to any needle question. + 3. Image quality meets a minimum standard (no broken images, no extreme blur, no primarily + text-on-white stock images). + 4. The dialogue progresses in a natural way and closely resembles real user–assistant interac- + tions. + +Multimodal sessions that fail quality checks are further rewritten and refined by human annotators +manually. + +Conversational naturalness. Naturalness is maintained through two complementary mechanisms. +Round 2 review directly targets dialogue quality: evidence sessions that read as stilted or overly +formal in register are returned to the generation pipeline for revision until the annotator is satisfied that +the exchange is a plausible user–AI interaction. This criterion covers colloquial phrasing, turn-taking +coherence, and the indirect embedding of factual content, following practices established in prior +conversational memory benchmarks [14]. Separately, the filler sessions that constitute the majority +of each assembled context are drawn from ShareGPT and UltraChat [65], providing real user–AI +conversations as the surrounding conversational frame rather than additional synthetic material. A +post-hoc text classifier trained to separate evidence sessions from haystack sessions achieves only +marginally above-chance accuracy (DeBERTa F1: 57.92%; Appendix G.8), confirming that the +human-curated evidence sessions carry negligible stylistic fingerprint relative to the surrounding +haystack sessions at the text level. + +Overall Quality Control. The three rounds collectively reduce the candidate pool from 20k to +the final 789 questions. The primary reasons for removal are: answer leakage in the question text +(23% of rejections), evidence image not carrying answer-critical information (31%), ambiguous or +multi-interpretable answers (18%), and difficulty calibration failures (28%). + +E.2 Judge Validation Details + +A natural concern with LLM-as-Judge evaluation is systematic bias, particularly when the judge +model belongs to the same family as evaluated models. We address this through cross-family +validation and format-dependent bias correction. + +Cross-family agreement is high. We re-evaluate a stratified sample of 800 items, drawn from +73,784 total judge calls (≈1.08% of the population), with GPT-5.4-mini as an independent judge; the +sample combines 200 random, 250 targeted, and 350 GPT-only extended items to ensure coverage +across model family, context length, question type, and judge score. Item-level agreement reaches +96.40% (Cohen’s κ = 0.93), and the model-level ranking correlation is Spearman ρ = 0.97 (p < +10−6 ), with a mean per-model accuracy delta of 3.70%. Judge choice does not reorder the leaderboard. + +The judge agrees with human consensus on 93.60% of items. Three annotators independently +labeled 484 items; disagreements were resolved to consensus. Against these consensus labels, our +Qwen3-VL-235B judge reaches 93.60% raw agreement with Cohen’s κ = 0.86. The errors are +leniency-biased: 29 false positives versus 2 false negatives, meaning the judge credits borderline +answers more often than rejecting correct ones. The per-question-type breakdown appears in Table 10 +below. + +The judge does not favor Qwen-family outputs. Because the Qwen3-VL-235B-A22B-Instruct +judge also appears as a benchmarked model (Table 13, row Qwen3-VL-235B (I)), we guard against +self-favoritism with two independent oracles. First, the pre-specified cross-family test compares the +Qwen-vs-GPT leniency gap on Qwen-family outputs (+3.00%) against non-Qwen outputs (+2.70%), +giving a difference of +0.33%—an order of magnitude below the 3% practical-significance threshold. +Second, on the 484 human-annotated items, the judge’s false-positive pattern is type-dependent +(IE partial matches, AR hedge phrases) rather than family-dependent, confirming that the leniency +is a judge-personality trait rather than a family-bias artifact. Beyond family bias, we identify a + + + 37 + format-dependent bias: the judge evaluates very short answers (1–3 words) more leniently. We +correct the resulting false positives for 6 affected models; all scores in this paper use corrected values. + +Sampling design and human annotation protocol. The LLM-as-Judge validation uses a two-tier +sample over the population of 73,784 total judge calls, spanning roughly 92 model×context runs (each +LVLM run on the full 789-question benchmark and each agent run on the 195-question canonical +subset, plus auxiliary backbone-ablation runs; some LVLM runs do not support the 128K context). A +core of 450 items (200 random + 250 targeted on hard cells) is annotated by both human raters and +GPT-5.4-mini; an extended 350 items is re-judged by GPT-5.4-mini only, yielding 800 cross-judge +items in total. Stratification spans model family (Qwen vs non-Qwen), context length (32K / 64K / +128K), question type, and judge score, so that rare cells—MSR arithmetic, KU stale-retrieval, and +thinking-mode degenerate outputs—are represented at a minimum floor rather than by chance. All 484 +items released for human verification received a consensus verdict; a further 120 are tagged for a future +dedicated inter-annotator reliability study (item IDs in double_annotation_item_ids.json). The +human annotation protocol used three annotators labeling each item across three rounds; whenever +the three labels disagreed, annotators discussed the case and converged on a single consensus verdict, +which is what we release as the human reference. + +Cross-judge agreement by question type. Table 10 reports Qwen3-VL-235B judge vs GPT-5.4- +mini accuracy on the 585 cross-judged items (after excluding 200 auto-zero cases and 15 empty +outputs). The Qwen3-VL-235B judge is equally or slightly more lenient across every question type; +the gap is largest on IE (−6.50%), driven by partial-match acceptances, and zero on MSR. No type +reverses the ranking direction between judges. + + +Table 10: Cross-judge agreement by question type on the 585 cross-judged items. Negative ∆ +indicates the Qwen3-VL-235B judge is more lenient than GPT-5.4-mini. IE: Information Extraction; +MSR: Multi-Session Reasoning; TR: Temporal Reasoning; KU: Knowledge Update; AR: Answer +Refusal. + Type Qwen acc (%) GPT acc (%) ∆ (%) n + IE 45.60 39.10 −6.50 169 + MSR 34.40 34.40 +0.00 128 + TR 47.70 46.30 −1.30 149 + KU 30.30 27.60 −2.60 76 + AR 63.50 60.30 −3.20 63 + + + +Human-vs-judge false-positive diagnosis. Table 11 breaks down the 29 false positives (judge=1, +human consensus=0) from Appendix E.2 by question type and disagreement pattern. Two observations +matter. First, 9 of the 10 AR false positives live in pre-retest runs that no longer feed any production +leaderboard; on canonical retest runs, a deterministic substring rule for the canonical refusal phrase +reaches 95.90% agreement with the human consensus (consistent with the AR row of the deterministic +typed-accuracy audit; Table 12). Second, the 11 IE false positives reflect the same partial-match +leniency observed in the aggregate cross-judge analysis—the Qwen3-VL-235B judge accepts short +factual answers that GPT and the human consensus both reject—which is a judge-personality trait +rather than a family-specific bias. + + +Table 11: False-positive diagnosis on the 484-item human-annotated subset. FP = judge scored 1 +where human consensus scored 0. Only 2 FN cases exist, so the asymmetry is strongly leniency- +biased. + Type FP count Disagreement pattern + Information Extraction (entity + previnfo) 11 Partial match on short factual answers + MSR / TR 3 Edge cases + Knowledge Update 5 Verbose-correct vs literal mismatch + Answer Refusal 10 Hedge phrases credited as refusal + + + + + 38 + Position relative to published reliability bars. κ = 0.86 exceeds the κ ≥ 0.80 reliability tier +reported for short-answer NLP judges [97] and the 80–85% raw-agreement bar reported for LLM-as- +judge on MT-Bench-style tasks [73]. We intentionally do not push agreement beyond the ≈90–96% +human-human ceiling typical for these tasks, since “super-consistent” judges can reflect overfitting to +annotator idiolect rather than improved reliability. The per-type breakdown in Table 10 and the FP +diagnosis in Table 11 together scope the residual uncertainty: the judge is slightly lenient, uniformly +across families, and most so on partial-match IE—none of which reorders model rankings in §4.2. + +Deterministic typed-accuracy audit. We complement the 484-item human-consensus audit with a +large-scale rule-based rescoring of every judge call whose reference answer is closed-form. Seven +of the nine reporting subtypes admit deterministic normalization: MSR Counting (integer match), +MSR Arithmetic (currency-normalized scalar), MSR YesNo (Yes/No), TR Order Ranking (tuple +of session indices), TR Duration Comparison (A/B label), TR Date Extraction (multi-format date +canonicalization), and AR Answer Refusal (canonical refusal-phrase set). IE Entity, IE PrevInfo, and +KU produce free-form short answers and remain LLM-judged. We apply the rules to all 32K runs of +the 27 LVLMs and the seven memory agents (34 evaluation rosters; 12,234 deterministic items), and +compare each per-item rule outcome against the LLM-judge label. Item-level agreement ranges from +87.4% on TR Order Ranking to 98.5% on MSR Arithmetic (Table 12); the count-weighted mean is +93.6%, in line with the 93.6% raw agreement on the human-consensus subset. The disagreement is +systematically leniency-biased: the LLM judge credits a deterministically-wrong answer (J-FP) on a +weighted 5.4% of items, against only 1.0% deterministically-correct answers rejected (J-FN). The +largest leniency channels are TR Order Ranking (12.6% J-FP, partial credit on near-correct tuples), +TR Date Extraction (8.6% J-FP, format-flexible date matching), and AR (5.7% J-FP, hedge phrases +credited as refusal)—the same partial-match and hedge-phrase patterns identified on the 484-item +human subset, but estimated here at 25× the sample size. At the model-leaderboard level, the +Spearman rank correlation between the LLM- and the deterministic-aggregated per-model accuracy +across the 34 rosters is ρ = 0.78, the top-10 sets overlap on 7 of 10 entries, and within the LLM-top- +10 the rank correlation is ρ = 0.82. The top-5 (Kimi-K2.5, Qwen3-VL-30B-Instruct, Qwen3.5-122B, +Qwen3-VL-235B-Instruct, Qwen3-VL-8B-Instruct) is preserved under both metrics. The three rosters +that drop out of the top-10 under deterministic rescoring—GLM-4.6V, Qwen3-VL-235B-Thinking, +and Mem0 (GPT-4.1-mini backbone)—all produce verbose justifications that the LLM judge credits +but the rule-based check rejects, consistent with the format-dependent leniency already corrected for +short outputs in this section. The takeaway is that LLM-judge leniency inflates closed-form accuracy +by approximately 5% in absolute terms but does not reorder the leaderboard top, which is the same +conclusion reached by the cross-family and human-consensus audits above on a much smaller sample. + + +Table 12: Deterministic typed-accuracy audit across 34 evaluation rosters at 32K. Agree is the per-item +agreement between the LLM judge and the rule-based label. J-FP marks LLM-judge over-credits +(judge=1, deterministic=0); J-FN marks LLM-judge under-credits (judge=0, deterministic=1). n +is the count of (model, item) pairs; not all rosters cover every subtype, since memory agents are +evaluated on the 195-question canonical subset. + Subtype n LLM (%) Det (%) Agree (%) J-FP (%) J-FN (%) + MSR Counting 1,551 11.7 10.6 97.6 1.7 0.7 + MSR Arithmetic 1,440 6.8 5.5 98.5 1.4 0.1 + MSR YesNo 1,082 47.3 42.2 94.7 5.2 0.1 + TR Order Ranking 661 27.5 15.0 87.4 12.6 0.0 + TR Duration Comparison 2,632 36.8 31.8 95.0 5.0 0.0 + TR Date Extraction 2,285 53.0 47.3 88.3 8.6 3.1 + AR Answer Refusal 2,583 78.1 73.8 92.9 5.7 1.4 + Aggregate (weighted) 12,234 42.3 37.8 93.6 5.4 1.0 + + + +F Prompt Templates + +This appendix documents the prompt templates that drive the construction and evaluation of M EM - +L ENS. The selection covers one canonical template per pipeline stage: the user prompt and judge +rubric used at evaluation time, the persona-driven dialogue prompt used for haystack sessions, the + + + 39 + question-generation prompt for each of the five major question types, the assistant template that +builds evidence sessions, and the text-only judge that gates anti-shortcut filtering. Helper prompts for +image search, and intermediate validators share the same skeleton as the canonical templates and are +omitted here; the full set is available in the code release. Within each template, identifiers in curly +braces (for example {context}, {question}, {theme}) are runtime placeholders that the pipeline +substitutes before the call is issued. + +F.1 Evaluation Prompts + +This subsection lists the two prompts that drive the evaluation pipeline of M EM L ENS. The first +prompt is the user message sent to every Vision-Language Model in the main evaluation. The con- +versation history, with images already inlined as tokens, is substituted into the {context} +placeholder, and the instruction line is fixed to Directly output the answer with no extra +output for every result reported in the paper (the chain-of-thought and structured-reasoning variants +present in the codebase are not part of the reported numbers). The second prompt is the rubric +followed by the LLM-as-Judge that produces the per-question accuracy scores in the main results +table. The system block encodes a final-answer-extraction policy that is robust to long thinking +traces and to circular reasoning, after which a task-specific criterion is appended at runtime; the +Information Extraction criterion is reproduced below as a representative example, while the remaining +seven criteria (MSR Yes/No, MSR Counting, MSR Arithmetic, TR Duration Comparison, TR Order +Ranking, TR Date Extraction, KU Knowledge Update, AR Answer Refusal) follow the same format. +Each judge call is cached in a SQLite store keyed by question, reference, prediction, and judge +identifier, so that re-runs over the same outputs remain deterministic. + + LVLM Evaluator User Prompt + Provide answers based on the given conversation history. If the question + cannot be answered based on the given conversation, respond with + "Insufficient information". + Conversation: + {context} + + Directly output the answer with no extra output. + Question Date: {question_date} + Question: {question} + + +For answer-refusal items, the dataset stores the literal string NOT_MENTIONED as the gold answer (Ap- +pendix F.3), while the eval prompt above instructs the model to output Insufficient information +as its surface refusal phrase. The judge maps any prediction in a canonical refusal-phrase set to a +successful abstention, so the gold token in the dataset and the natural-language phrase emitted by the +model are intentionally distinct strings. + + LLM-as-Judge Prompt (system block + IE task criterion) + Now your role is a grading teacher. Your task is to review and score student + answers based on reference standard answers for a question-answering + benchmark. You need to notice the following key points: + - First, extract the final answer from the student’s solution, then analyze + and judge whether the answer is correct. + - Scoring should only refer to the final answer obtained by the student; + there is no need to examine whether the intermediate problem-solving + steps are correct. + - If the response contains both hesitation and a clear answer, judge the + answer itself. + - If the response contains both a refusal and a guessed answer, judge the + final committed answer. + - If the response gives multiple inconsistent answers, assign 0 points. + - If the student’s response shows circular reasoning - repeatedly revisiting + the same evidence, flip-flopping between answers (e.g., "yes... wait + no... actually yes... let me reconsider... no"), with no clear final + commitment - assign 0 points. A correct answer mentioned during reasoning + does NOT count unless it is the clearly stated final conclusion. + - Only the student’s LAST clearly stated answer counts. If the student + initially says "yes" but later concludes "no" (or vice versa), score + based on the LAST position only. Intermediate answers during reasoning + are not final answers. + - If the student’s output is entirely a reasoning trace (e.g., "let me + check... scanning evidence... re-reading session 5...") with no clear + + + + + 40 + concluding answer statement, assign 0 points - even if the correct answer + appears somewhere within the reasoning. +- If the student’s output begins with "[...truncated earlier reasoning...]", + this means only the final portion of a longer response is shown. Focus on + extracting the answer from this final portion. + +Below are examples of problematic student outputs that apply to ALL task +types. Study these before grading. + +[Universal Example A - Circular Reasoning -> 0 points] +: Is the cat in Mark and Jenny’s 5-year anniversary post named +Mittens? +: Yes +: let me check session 5 mark has cat named mittens session +7 mark and jenny anniversary post there is no link wait let me re-read +session 5 says mittens session 7 says anniversary actually looking at this +again wait is it possible let me reconsider mark has mittens in session 5 +the post is about couple therefore answer is likely no wait let me double +check session 5 mark has cat mittens actually maybe yes but text doesnt say +[Scoring Rationale]: The output flip-flops between "yes" and "no" multiple +times with no clear final commitment. The last position is ambiguous +("maybe yes but text doesnt say"). This is circular reasoning. +In summary, the student’s answer deserves 0 points. +[JSON]: {"answer_score": 0} + +[Universal Example B - Redundant But Committed -> Score Normally] +: Is the trailing plant in the cafe safe for my kitten? +: No +: the plant looks like pothos pothos is toxic to cats +therefore answer is no let me double check is pothos safe for cats no is +answer no yes wait let me verify image 3 is pothos image 9 is pothos pothos +is toxic therefore answer must be no +[Scoring Rationale]: Despite excessive self-verification, the student +consistently commits to "no" throughout and never wavers to a different +answer. The final answer is clearly "No", which matches the standard +answer. +In summary, the student’s answer deserves 1 point. +[JSON]: {"answer_score": 1} + +[Universal Example C - Reasoning Trace With No Answer -> 0 points] +: How many sessions mentioned the coffee shop? +: 3 +: scanning session 1 yes coffee mentioned session 2 no +session 3 yes coffee again session 4 no session 5 maybe let me re-read +session 5 it mentions cafe is that same as coffee shop need to check +session 6 no session 7 unclear let me look at session 5 again +[Scoring Rationale]: The student walks through sessions but never states a +final count. The output is entirely a reasoning trace with no concluding +answer. +In summary, the student’s answer deserves 0 points. +[JSON]: {"answer_score": 0} + +Now proceed with grading. Remember: the universal examples above apply +regardless of task type. + +- When analyzing and judging whether the answer is correct, you need to + write down the scoring rationale, organize it into clear statements that + follow the logical flow. The summary of the scoring rationale should be + placed at the end, using the following format: "In summary, the student’s + answer deserves x points" (where x represents the student’s specific + score). +- Keep the whole process concise, within 150 words. +- Provide the score based on your analysis and display it in a code block + in "JSON" format. +- An item is covered if it is strictly mentioned or unambiguously implied + by a semantic equivalence. This includes numerical equivalence (e.g., + 10% and 0.1), synonyms (e.g., UK and United Kingdom), plural/singular + forms (e.g., "apple" and "apples"), and equivalent date formats (e.g., + 2024-01-15 and January 15, 2024). However, do not accept loosely related + concepts. +- Ignore minor formatting differences, capitalization, punctuation, and + equivalent wording when meaning is unchanged. + +Your output format is: +[Scoring Rationale]: +[Score]: x points +[JSON]: +{"answer_score": } + + + + + 41 + Below is the grading rubric: + [Scores]: + The scoring scale consists of 2 levels in total, from highest to lowest: + 1 point, 0 points (the minimum is 0 points). + [Tier Details]: + 1 point: Assign 1 point if the student’s final answer matches the standard + answer under the task-specific criteria below. + 0 points: Assign 0 points if the student’s final answer does not match the + standard answer, the student refuses to answer, claims insufficient + information (except for AR tasks), or does not clearly answer. + + [Task-Specific Criteria] + [IE - Information Extraction] + This is an information extraction question. + - Assign 1 point if the student’s response contains the core information + from the standard answer. Minor wording differences are acceptable, but + the essential information must be present and correct. + - Assign 0 points if the core information is missing, contradicted, too + vague, refused, or incorrect. + + [Remaining seven task-specific criteria (MSR Yes/No, MSR Counting, + MSR Arithmetic, TR Duration Comparison, TR Order Ranking, TR Date + Extraction, KU Knowledge Update, AR Answer Refusal) follow the same + format and are bundled with two worked examples per type in the + codebase release.] + + + +F.2 Haystack Generation + +The haystack pipeline turns a persona profile, the conversation summary up to that point, and a +list of recent life events into the next message of the conversation. The box below shows the +canonical prompt used at this step. The same template generates both user and assistant turns by +swapping the persona slot; image sharing is encouraged in the body of the message itself, so that +the resulting haystack sessions remain authentically multimodal rather than text-only sessions with +images appended as side annotations. Persona descriptions, event tables, and the previous-session +digest are filled into the slots marked with %s, in the order listed at the bottom of the prompt. + Haystack: Persona-Driven Dialogue Turn + Use a given PERSONALITY to write the next message in this ongoing, + friendly chat. + + PERSONALITY: %s + + STYLE: + - No hard word limit - write fluidly but keep it conversational, not like + a letter. + - Focus on real emotions, relationships, regrets, or inspirations tied to + life events. + - Reference real people and times ("this morning", "last week", "when I + turned ten"). + - Ask thoughtful follow-up questions when feels natural. + - IMPORTANT: Share photos when discussing experiences, memories, or current + activities. + Good examples: + - "The view was incredible! [shares a photo from the overlook]" + - "This is what I’ve been dealing with all week. [shares a photo of the + messy project]" + - "Look what I found in the attic! [shares an old family photo]" + - Make the photo sharing feel conversational and relevant to what you’re + saying. + - Avoid talking about outdoor activities or sports. + + EVENT MARKING RULES: + If you mention an event, place **EVENT:ID** at the end of the sentence, + after punctuation. + Multiple events: **EVENT:EA1** **EVENT:EA2** + Never in mid-sentence or as a heading. + + %s last chatted with %s on %s. Today is %s. You are %s. + + Conversation summary: + %s + + Recent life events (with IDs): + %s + + + + + 42 + Background known to both: + %s + + %s + Write a heartfelt, realistic continuation to %s. + Discuss how the relevant EVENTS have influenced you - show joy, sadness, + or frustration honestly. + + + +F.3 Question Generation + +This subsection collects one canonical generator prompt per major question type. Each prompt +receives topic context, image metadata, and event-table information, and returns a strict JSON object +containing a question, an answer, and supporting facts. Generated outputs then pass through a +downstream rule-based pre-filter and the text-only judge described in subsection F.5 before being +admitted to the candidate pool that goes to human review. Variants of these prompts (one per subtype +within a major type) share the same skeleton with only minor changes to the rule list and the JSON +schema; only the canonical entry is reproduced here. + +Information Extraction. The two-hop alignment prompt below is the canonical IE generator. The +first hop is a textual cue that identifies which visual element to inspect, and the second hop is a +visual extraction from that element. The same template handles OCR, attribute, counting, and spatial +variants by varying the visual_type field; an analogous template (omitted) handles the PrevInfo +subtype that grounds the visual hop in screenshots from earlier sessions. + Question Generation: IE (Two-Hop Alignment) + [SYSTEM] + You create two-hop visual questions testing alignment (knowing WHERE to + look). + + Structure: + - Hop1: A first-person fact directs attention to a specific element/region + in the image + - Hop2: The answer comes ONLY from visually inspecting that element + + Rules: + - Without the fact, it’s ambiguous which element to examine + - Vary the visual skill: read text, check color/attribute, count items, + spatial relations + - Never reference photos ("in the photo", "in the image I shared") + + + { + "question": "Since what year does it state the brand has been trusted?", + "answer": "1967", + "target_element": "Circular seal in top right corner", + "alignment_cue": "checking the seal on the package", + "hop1_reasoning": "Fact directs attention to the circular seal in the + top right corner.", + "hop2_reasoning": "The seal reads ’Trusted since 1967’.", + "visual_type": "ocr", + "fact": {"fact_id": "F1", + "text": "I’m checking the circular seal in the top right corner + to see how long the brand’s been around."}, + "rationale": "1) Fact points to circular seal. 2) Seal displays 1967.", + } + + + [USER] + {image_description} + {visual_elements} + {context} + + + Generate a two-hop alignment question: + 1. Write a first-person fact that identifies which element to examine + (no token in the fact) + 2. Ask a question whose answer requires visually inspecting that element + 3. The answer must be impossible to determine without examining the image + + Phrasing: ask naturally - never say "in the photo". + Vary structure - don’t always start with "What". + + + + + 43 + Return JSON with the same schema as the example above. + + + +Temporal Reasoning. The order-ranking prompt asks the model to produce eight first-person facts +with one timestamp each, of which one is a needle whose timestamp is encoded by a clock image +(Mode B) rather than by text. Modes B-date, C, and D follow the same skeleton with the timestamp +source replaced by a date image, an entity image with implicit text dates, and an entity image with +explicit text dates respectively. + + Question Generation: TR (Order Ranking, Mode B) + Return ONLY valid JSON (no markdown/comments). Do NOT output an ’answer’ + field. + + TASK (ORDER_RANKING / Mode B clock): Generate exactly 8 facts (F1-F8), + each with ONE unique timestamp in time_points[0]. ONE needle contains + ’’ once with source=’image_clock’, granularity=’minute’. Needle + bound to {bound_event_id}, clock_label={clock_label} (fixed). + + TIME: Needle value=’YYYY/MM/DD HH:MM’; others=’YYYY/MM/DD’ or + ’YYYY/MM/DD HH:MM’. question_date strictly later than all timestamps. + + STYLE: First-person, conversational, distinct events. NO formatted + dates/times in text (use "late March", "that summer"). + + OUTPUT JSON: + { + "question_date":"YYYY/MM/DD", + "rationale":"Brief chronological reasoning.", + "facts":[ + {"fact_id":"F1","text":"...","is_needle":true/false, + "time_points":[ + {"role":"occurred_at|started_at|ended_at|arrived_at", + "value":"...", + "source":"text|image_clock", + "granularity":"date|minute"}], + "event_id":"...","event_type":"point"} + ], + "images":[ + {"image_id":"IMG1","file_path":"","bound_fact_id":"F?", + "grounding_type":"temporal","clock_label":"..."} + ] + } + + INPUTS + BACKGROUND: {paragraph} + EVENTS: {events_table} + CLOCK: bound_event_id={bound_event_id}, clock_label={clock_label} + + + +Knowledge Update. The chain-generation prompt produces a four-fact preference-evolution chain. +Each fact is at the category level; the specific entity at each step is later substituted by an image, so +that text alone can identify the time step but not the value at that step. + + Question Generation: KU (Atomic Evolution Chain) + You generate a 4-step atomic fact chain showing how MY preferences evolve + over time. + + Evolution theme: {evolution_theme} + Use these categories IN ORDER (one per fact, no repeats): {categories} + + TASK + Write exactly 4 short, natural-sounding facts (1-2 sentences each) that + describe realistic preference changes across time. + + HARD RULES + 1) CATEGORY-LEVEL ONLY: use only the provided category names (no specific + items/brands/subtypes). + 2) FIRST-PERSON: each fact must be written from my perspective using + "I / my / me". + 3) PREFERENCE CHANGE: include one simple preference-change marker per fact: + - Fact 1 marker: used to / before / at first + - Fact 2 marker: then / later / after that + + + + + 44 + - Fact 3 marker: lately / nowadays / recently + - Fact 4 marker: now / currently / these days + 4) DISTINCT CATEGORIES: Fact i must use category i from {categories}. + 5) COMPLETE SENTENCES: each fact must be a grammatical sentence (not + keywords), with a clear subject + verb. + 6) LENGTH: each fact must be <= 150 characters total. + + OUTPUT (STRICT JSON ONLY; no extra text) + { + "facts": [ + {"text": "...", "category": "", + "temporal_position": 1, "temporal_marker": ""}, + {"text": "...", "category": "", + "temporal_position": 2, "temporal_marker": ""}, + {"text": "...", "category": "", + "temporal_position": 3, "temporal_marker": ""}, + {"text": "...", "category": "", + "temporal_position": 4, "temporal_marker": ""} + ] + } + + Generate the chain for theme "{evolution_theme}" using categories + (in order): {categories}. + + + +Multi-Session Reasoning. The visual identity-match prompt is the canonical MSR generator (sub- +pattern D1). It produces a scenario where one named entity and one vague reference are scattered +across temporally distinct contexts, and an image attached to the vague reference is the only signal +that resolves identity. Five anti-leakage rules (temporal isolation, no ownership disambiguation, no +descriptive leakage, no text-based identity resolution, no cultural narrowing) are enforced inside the +prompt and re-checked by the text-only judge of subsection F.5. + + Question Generation: MSR (Entity Resolution / Visual Identity Match) + TASK: Generate a multi-session entity resolution question where the model + must determine whether an ambiguously referenced entity is the SAME AS a + specifically named entity. + + Text facts reference an entity using VAGUE terms (e.g., "the dog I + adopted", "that plant from the market"). One fact has an image that + reveals the entity’s visual identity. The model must match the image to a + named entity from another fact to answer. + + THEME: {theme} ({theme_description}) + EXAMPLE ITEMS: {example_entities} + NUM_FACTS: {num_facts} + + RULES: + - At least 1 TEXT NEEDLE names a specific entity (e.g., "I got a golden + retriever puppy named Max") + - At least 1 TEXT NEEDLE references an entity VAGUELY (e.g., "I adopted a + dog from the shelter last week") + - EXACTLY 1 IMAGE NEEDLE with token: shows the vaguely referenced + entity visually. Text uses only vague terms (e.g., "Here’s the dog I + brought home "). The image reveals whether it matches the named + entity. + - Question: "Is the [vague reference] the same as [named entity]?" + Answer: "Yes" or "No" + - {num_facts} facts as first-person chat messages ("I", "my"), + 2-4 sentences each + - CRITICAL: Every fact MUST be a first-person declarative statement. + NEVER phrase facts as questions. + - EVERY fact must be vital to resolving the entity identity. Only use + fact_type "text_needle" or "image_needle". + - Cross-modality: text alone cannot determine if the vague reference + matches the named entity (the name is never stated for the vague + reference). The image is the only way to confirm. + - Each fact should reference the entity with "entity_referenced" field + tracking which entity it refers to. + + ANTI-LEAKAGE RULES (violations cause rejection): + 1. TEMPORAL ISOLATION: The named entity and the vague reference MUST + appear in unrelated temporal contexts. NEVER place both in the same + event, occasion, trip, or timeframe. + 2. NO OWNERSHIP DISAMBIGUATION: Don’t assign entities to different named + owners if ownership alone reveals identity. Multiple entities should + + + + + 45 + plausibly belong to the same person. + 3. NO DESCRIPTIVE LEAKAGE: The vague reference text must NOT include + adjectives that confirm or contradict the named entity’s known + characteristics. + 4. NO TEXT-BASED IDENTITY RESOLUTION: No text fact may state what the + vague reference IS. Phrases like "matches", "is the same as", "turns + out to be", "which is actually" must NEVER appear in any fact. + 5. NO CULTURAL NARROWING: The context must allow multiple plausible + entity types. + + NEGATIVE EXAMPLES (DO NOT generate scenarios like these): + - BAD (temporal co-occurrence): F1: "I named my new puppy Max, a golden + retriever." F2: "I also got a dog from the shelter last week." + Both acquired recently -> reader infers same event. + - BAD (ownership leakage): F1: "My roommate’s cat is named Whiskers." + F2: "The cat in my bedroom..." Different owners disambiguate. + - BAD (descriptive contradiction): F1: "I got a huge Great Dane named + Duke." F2: "Here’s the tiny puppy I found " - "tiny" rules out + Great Dane. + - BAD (explicit resolution): F1: "The plant I bought turned out to be a + monstera." Text resolves identity directly. + + OUTPUT (JSON only, no markdown): + { + "theme": "{theme}", + "named_entity": "", + "vague_reference": "", + "same_entity": true/false, + "question": "Is [vague_reference] the same [category] as [named_entity]?", + "answer": "Yes" or "No", + "explanation": "Step-by-step: F1 names [entity]. F2 refers vaguely to + [reference]. F3’s image shows [what], which is/isn’t + [named_entity]. Therefore Yes/No.", + "facts": [ + {"fact_id":"F1","text":"...","fact_type":"text_needle", + "has_image":false,"entity_referenced":"..."}, + ..., + {"fact_id":"F3","text":"... ...", + "fact_type":"image_needle","has_image":true, + "entity_referenced":"...","image_description":"...", + "image_search_query":"...","image_search_object":"...", + "image_provides":"..."} + ] + } + + Generate a creative, realistic scenario now: + + + +Answer Refusal. The few-shot prompt below produces a question that is on-topic for the given +context but whose answer is not present in the context. The fixed gold answer is the literal string +NOT_MENTIONED, which the judge rubric in subsection F.1 treats as a successful abstention. + + Question Generation: AR (Abstention Few-Shot) + You are generating Abstention QA for a multimodal benchmark. + + Goal: craft a plausible, on-topic question that appears answerable from + the given info (paragraph + image caption if provided) but is actually + unanswerable. The gold answer must be the literal string "NOT_MENTIONED". + + Rules + 1) Use ONLY the provided context: + * CONTEXT: the paragraph + * VISUAL: the image or its caption (if provided) + 2) The question must be topically related to entities/events/attributes + present in the context. + - Good: asks for a missing detail about a mentioned item. + - Bad : asks about something totally unrelated. + 3) The correct answer must be truly unknowable from the given info. + 4) Output one compact JSON object with EXACTLY these keys: + { + "question": "one interrogative sentence", + "answer": "NOT_MENTIONED", + "evidence": "short extractive snippet proving the info is absent", + "explanation": "one-sentence rationale why we must abstain" + } + 5) Do NOT add extra keys or commentary outside the JSON. + + + + + 46 + ---------- Example ---------- + INPUT + CONTEXT: + User: I upgraded my old 10-gallon tank, which has my betta fish, Bubbles. + User: I added decorations to the 20-gallon tank for more hiding places. + + OUTPUT + { + "question": "How many fish are there in my 30-gallon tank?", + "answer": "NOT_MENTIONED", + "evidence": "The user never mentions owning a 30-gallon tank.", + "explanation": "No sentence discusses fish in a 30-gallon tank, so the + answer cannot be inferred." + } + ----------------------------- + + + + +F.4 Evidence Session Construction + +The evidence pipeline wraps each generated needle fact into a multi-turn session that is structurally +indistinguishable from a haystack session. The box below contains the prompt that drives the assistant +side of these sessions. Length is targeted at 250 to 350 words per turn, with a knowledge-oriented +follow-up rather than a personal-social one, so that the evidence session looks informative without +advertising the role as the resolution channel for any specific needle question. The user side is +generated by a paired template that injects the needle fact into the message body using a directive +that is itself constrained to first-person conversational style, after which a six-stage validator chain +(rule-based length, photo-directive, n-gram leakage, semantic leakage, ambiguity preservation, and +end-marker checks) decides whether the turn is accepted or regenerated. + + + Evidence Session: Assistant Turn Template + You are {assistant_name}, a helpful AI assistant having a casual + conversation about {topic}. + + Current conversation: + {chat_history} + + {user_name}’s last message: "{last_msg}" + + {image_context} + + Generate a helpful, conversational response (TARGET: 250-350 words - be + concise, not lecture-like): + + STRUCTURE YOUR RESPONSE: + 1. Acknowledge what they shared (1-2 sentences) + - If they shared a photo, briefly acknowledge it (1 sentence max), + then move on + 2. Share your perspective or relevant information (main body, 150-250 + words) + - Provide helpful context, explanations, or suggestions + - Keep it focused - cover 1-2 key points, not exhaustive lists + - If applicable, offer practical tips or recommendations + 3. End with a follow-up thought or question (1 sentence). Ask + knowledge-oriented questions ("Would you like tips on X?" or "What + part of this are you most stuck on?"), NOT personal/social questions + ("Do you prefer X or Y?" or "Have you ever tried X?"). + + IMPORTANT: Write like a helpful chat assistant, NOT a textbook. Keep + responses focused and avoid padding with extra context the user didn’t + ask for. + + STYLE GUIDELINES: + - Be warm, conversational, and genuinely helpful + - Use natural paragraph breaks for readability + - Keep photo acknowledgment BRIEF (don’t describe or analyze the image + in detail) + - Provide substantive value - don’t just agree, add to the conversation + - Do NOT repeat back personal details verbatim + - Do NOT use emojis + + Output only your response, nothing else. + + + + + 47 + F.5 Anti-Shortcut Quality Filter + +After question generation, every candidate is passed through a text-only judge that simulates a model +with no access to images. The box below shows the prompt used for this judge. A candidate is rejected +when the judge returns answerable_without_image = true, which guarantees that the released +benchmark cannot be solved by a strong text-only baseline through chain-of-thought guessing alone. +A second visual judge (omitted from this appendix, but structurally identical apart from the role swap) +checks the converse, that an image together with the question is sufficient to determine the answer; +the two filters together enforce the cross-modal grounding contract of the construction pipeline. + + Filter: Text-Only Leakage Judge + [SYSTEM] + You are a judge evaluating whether a question can be answered from text + alone, WITHOUT seeing any images. + + Your task is to determine if the answer to a question can be inferred + from the conversation text. + + Rules: + 1. Carefully read the conversation text + 2. Try to answer the question using ONLY the text (no images) + 3. If you can confidently answer the question from text alone -> + answerable_without_image = true + 4. If the answer requires visual inspection of an image -> + answerable_without_image = false + + Be strict: even if you can make an educated guess, if the text doesn’t + explicitly contain the answer, mark it as NOT answerable from text. + + [USER] + {conversation_text} + {question} + {answer} + + + Can this question be answered from the conversation text alone, without + seeing any images? + + Return JSON: + { + "answerable_without_image": true|false, + "text_based_answer": "your answer attempt from text only, or + ’CANNOT_DETERMINE’", + "confidence": "high|medium|low", + "explanation": "why the answer is/isn’t determinable from text", + "leakage_evidence": ["list of text snippets that reveal the answer, + if any"] + } + + + + +G Supplementary Experiments and Analysis +G.1 Extended Results Tables + +G.2 Canonical 195-question Subset for Agent Evaluation + +The seven memory-augmented agents in §4.3 (full analysis in Appendix G.6) are evaluated on a 195- +question canonical subset rather than the full 789-question benchmark, because agent pipelines are +substantially slower than direct-LVLM inference (M2A takes roughly 60× longer per question). This +appendix documents how the 195-question set is derived and confirms that its per-type composition +preserves the full-benchmark distribution. + +Derivation. The 195-question (a quarter of the original dataset) canonical subset is constructed by +stratified sampling (seed=42) from the full 789-question benchmark, preserving per-type proportions. +All seven memory-augmented agents are evaluated on this same subset. + +Per-type composition matches the full benchmark. Table 15 compares the 195-subset’s per-type +proportions against the full 789-question benchmark. Differences are below 0.2 percentage points for + + + 48 + Table 13: Comprehensive per-type accuracy (%) for all 27 LVLMs at 32K / 64K / 128K contexts +(LLM-as-Judge, n = 789). IE: Information Extraction, MSR: Multi-Session Reasoning, TR: Tempo- +ral Reasoning, KU: Knowledge Update, AR: Answer Refusal. Cells marked — indicate the model’s +context window does not support the input length. This table is the full roster corresponding to the +representative subset in Figure 2; agents are reported separately in Table 14. + 32K 64K 128K +Model IE MSR TR KU AR Ov. IE MSR TR KU AR Ov. IE MSR TR KU AR Ov. +Claude Sonnet 4.5 28.05 13.29 40.21 29.31 97.78 36.50 26.42 9.09 35.05 21.55 94.44 32.45 19.51 2.80 32.99 16.38 93.33 27.76 +Gemini-3.1-Pro 57.32 32.17 40.93 49.14 97.75 54.10 58.94 31.47 42.27 48.62 95.56 52.99 55.79 29.37 41.24 46.03 96.67 51.99 +GPT-5.4 69.51 28.18 39.18 47.41 97.78 52.72 63.01 27.27 42.27 43.10 96.67 52.34 60.16 21.68 39.69 43.10 94.44 49.56 +Kimi-K2.5 54.78 44.06 53.09 50.86 97.78 54.88 52.44 35.66 52.06 49.14 97.78 53.99 51.63 28.67 48.25 48.59 93.33 51.99 +Qwen3.5-122B-A10B 74.39 30.07 51.55 49.14 88.89 58.68 67.07 27.27 51.55 48.59 84.44 55.89 43.09 22.38 41.75 46.03 83.33 45.50 +Qwen3.5-27B 70.33 29.37 39.18 42.24 86.67 52.98 62.20 20.98 35.57 43.97 67.78 46.13 54.07 13.99 34.02 38.79 63.33 40.68 +Qwen3.5-9B 67.48 16.78 38.66 43.10 91.11 50.32 60.57 13.99 33.51 37.93 80.00 44.36 45.12 5.59 26.80 25.86 62.22 32.57 +Qwen3.5-4B 56.91 11.89 38.66 34.48 86.67 44.36 49.19 9.79 30.41 28.45 73.33 37.14 29.67 3.50 24.74 19.83 55.56 25.22 +Qwen3.5-2B 64.23 17.48 32.47 33.62 72.22 44.36 2.03 0.00 2.06 0.86 15.56 3.04 1.22 0.00 1.55 0.00 1.11 0.89 +Nemotron-Nano-12B 63.82 16.78 44.85 39.66 45.56 44.99 56.10 20.98 42.78 34.48 35.56 40.94 — — — — — — +Cosmos-Reason2-8B 55.69 23.78 48.45 31.90 68.89 46.13 46.34 20.28 45.88 24.14 54.44 39.16 36.99 20.28 43.81 12.93 35.56 31.94 +Phi4-Multimodal 31.71 20.28 43.81 19.83 43.33 32.19 28.05 20.98 39.18 24.14 21.11 28.14 22.76 22.38 30.93 14.66 15.56 22.69 +GLM-4.6V 59.35 20.28 53.09 43.97 93.33 52.34 61.79 23.78 53.09 34.48 64.44 49.05 52.85 23.08 43.81 28.45 30.00 39.04 +GLM-4.5V 61.79 15.38 39.69 24.14 51.11 41.19 45.93 15.38 33.51 14.66 32.22 31.18 — — — — — — +Gemma3-27B 34.96 23.78 41.24 30.17 68.89 37.64 26.42 22.38 44.33 27.59 61.11 34.22 19.11 18.18 37.11 13.79 48.89 25.98 +Gemma3-12B 36.18 22.38 43.81 29.31 53.33 36.50 30.89 22.38 44.33 20.69 38.89 32.07 23.58 18.18 28.35 16.38 28.89 23.32 +Gemma3-4B 25.20 21.68 41.75 23.28 14.44 27.12 17.89 18.88 28.87 14.66 8.89 19.26 14.63 12.59 24.23 10.34 5.56 14.96 +Qwen3-VL-235B (T) 51.63 25.17 49.48 36.21 88.89 48.29 38.62 18.18 42.78 35.34 75.56 39.67 29.27 15.38 43.30 30.17 67.78 34.73 +Qwen3-VL-235B (I) 60.98 18.88 55.67 40.52 97.78 53.23 56.10 20.98 54.12 36.21 95.56 50.82 51.22 18.18 54.64 30.17 92.22 47.66 +Qwen3-VL-30B (T) 41.46 11.89 35.57 30.17 71.11 36.38 39.02 11.89 37.11 29.31 65.56 35.23 33.74 10.49 36.60 20.69 60.00 31.31 +Qwen3-VL-30B (I) 65.45 18.88 60.82 37.93 93.33 55.01 56.91 20.28 55.15 39.66 78.89 49.81 56.50 16.08 52.58 24.14 63.33 44.23 +Qwen3-VL-8B (T) 51.22 17.48 52.58 32.76 80.00 46.01 42.28 18.88 50.52 30.17 54.44 39.67 33.33 8.39 39.18 24.14 45.56 30.29 +Qwen3-VL-8B (I) 52.85 17.48 58.25 33.62 90.00 49.18 44.72 19.58 47.42 29.31 82.22 42.84 32.52 20.28 44.85 15.52 66.67 34.73 +Qwen3-VL-4B (T) 46.75 20.98 43.30 33.62 82.22 43.35 28.05 11.19 26.80 18.10 70.00 28.01 17.07 9.09 13.92 10.34 52.22 17.87 +Qwen3-VL-4B (I) 52.03 17.48 42.78 25.86 92.22 44.23 42.28 19.58 35.57 19.83 75.56 37.01 28.46 16.08 35.05 13.79 50.00 28.14 +Qwen3-VL-2B (T) 26.83 9.09 19.07 11.21 87.78 26.36 19.11 6.99 11.86 16.38 54.44 18.76 16.67 12.59 5.67 8.62 14.44 11.79 +Qwen3-VL-2B (I) 32.52 16.78 50.52 20.69 72.22 36.88 32.52 21.68 40.72 16.38 65.56 33.97 25.20 18.88 32.99 8.62 47.78 26.11 + + + +Table 14: Comprehensive per-type accuracy (%) for all seven memory-augmented agents at 32K / +64K / 128K / 256K contexts, evaluated on 195-question canonical subsets. IE: Information Extraction, +MSR: Multi-Session Reasoning, TR: Temporal Reasoning, KU: Knowledge Update, AR: Answer +Refusal. Only memory agents are evaluated at 256K; direct LVLMs are not (see Table 13). + 32K 64K 128K 256K +Model IE MSR TR KU AR Ov. IE MSR TR KU AR Ov. IE MSR TR KU AR Ov. IE MSR TR KU AR Ov. +Mem0 13.11 25.71 50.00 17.24 77.27 31.79 11.48 22.86 45.83 24.14 77.27 31.28 8.20 20.00 50.00 24.14 77.27 30.26 11.48 22.86 47.92 20.69 72.73 30.77 +MemOS 18.03 22.86 39.58 24.14 68.18 30.26 18.03 22.86 39.58 24.14 68.18 30.77 16.39 22.86 43.75 17.24 81.82 31.28 16.39 22.86 37.50 17.24 72.73 29.74 +MemAgent-7B 18.03 25.71 62.50 41.38 13.64 32.82 13.11 28.57 45.83 24.14 4.55 25.64 14.75 28.57 60.42 20.69 9.09 28.21 11.48 22.86 50.00 20.69 4.55 24.62 +Memory-T1 18.03 25.71 62.50 13.79 9.09 28.72 19.67 22.86 56.25 17.24 18.18 28.72 21.31 22.86 56.25 27.59 13.64 30.26 18.03 25.71 58.33 24.14 9.09 29.23 +M3-Agent 18.03 22.86 29.17 6.90 13.64 19.49 18.03 17.14 37.50 3.45 4.55 18.97 22.95 20.00 22.92 6.90 18.18 19.49 22.95 28.57 27.08 13.79 22.73 23.59 +M3C 8.20 25.71 31.25 10.34 18.18 18.46 9.84 25.71 27.08 6.90 22.73 17.95 8.20 25.71 27.08 0.00 9.09 14.87 11.48 25.71 29.17 0.00 13.64 16.92 +M2A 14.75 8.57 2.08 0.00 22.73 15.38 18.03 14.29 25.00 0.00 13.64 15.90 21.31 11.43 27.08 0.00 13.64 16.92 21.31 11.43 25.00 0.00 13.64 16.41 + + + + +every type, so rankings computed on the subset transfer to the full benchmark without systematic +bias. + +Table 15: Per-type composition of the 195-question canonical subset compared against the full +789-question benchmark. Proportions are preserved to within 0.2 percentage points per type. + Type Full 789 (%) 195-subset 195 share (%) + Information Extraction (IE) 31.18 61 31.28 + Multi-Session Reasoning (MSR) 18.12 35 17.95 + Temporal Reasoning (TR) 24.59 48 24.62 + Knowledge Update (KU) 14.70 29 14.87 + Answer Refusal (AR) 11.41 22 11.28 + Total 100.00 195 100.00 + + +Direct-LVLM overlay on the 195-subset. To enable apples-to-apples comparison with the seven +memory agents, we re-score every direct-LVLM run used elsewhere in the paper on exactly the + + + 49 + 195 canonical qids. Table 16 reports overall and per-type accuracy for six representative LVLMs +at 32K/64K/128K and the four agents at 32K/64K/128K/256K. This is the matched-subset version +of Figure 2; rankings transfer between the subset and the full benchmark at 32K with Spearman +ρ = 0.94 (p < 0.01, n = 6 direct LVLMs), so conclusions drawn on the 195-subset are not an artifact +of subset choice. + +Table 16: Agent-vs-direct-LVLM comparison on the canonical 195-question subset. Overall and +per-type accuracy (%) at 32K/64K/128K (direct LVLMs) and 32K/64K/128K/256K (agents). The +agents listed are the original four systems; the six direct LVLMs span the best API system (GPT-5.4), +the best closed-commercial (Gemini-3.1-Pro), the best open (Qwen3.5-122B), and the three Qwen3- +VL sizes that serve as matched-backbone counterparts. The remaining agents (Mem0, MemOS, +MemAgent-7B) are reported in Table 14. Per-type n counts are shown once at the top (61 IE / 35 +MSR / 48 TR / 29 KU / 22 AR). + Overall IE MSR TR KU AR + System / Ctx (n=195) (61) (35) (48) (29) (22) + Direct LVLMs on the 195-subset + Gemini-3.1-Pro / 32K 55.38 60.66 45.71 39.58 48.28 100.00 + Gemini-3.1-Pro / 64K 58.46 59.02 48.57 41.67 65.52 100.00 + Gemini-3.1-Pro / 128K 58.46 68.85 42.86 43.75 48.28 100.00 + GPT-5.4 / 32K 51.28 63.93 17.14 35.42 55.17 100.00 + GPT-5.4 / 64K 56.92 65.57 37.14 47.92 44.83 100.00 + GPT-5.4 / 128K 51.79 67.21 28.57 37.50 41.38 90.91 + Qwen3.5-122B / 32K 63.59 75.41 45.71 58.33 44.83 95.45 + Qwen3.5-122B / 64K 59.49 67.21 37.14 52.08 58.62 90.91 + Qwen3.5-122B / 128K 49.23 45.90 31.43 43.75 58.62 86.36 + Qwen3-VL-30B (I) / 32K 58.46 65.57 25.71 62.50 48.28 95.45 + Qwen3-VL-30B (I) / 64K 52.31 57.38 25.71 56.25 44.83 81.82 + Qwen3-VL-30B (I) / 128K 50.77 57.38 25.71 66.67 27.59 68.18 + Qwen3-VL-8B (I) / 32K 50.77 52.46 22.86 58.33 44.83 81.82 + Qwen3-VL-8B (I) / 64K 47.69 52.46 25.71 50.00 41.38 72.73 + Qwen3-VL-8B (I) / 128K 34.36 27.87 22.86 54.17 13.79 54.55 + Qwen3-VL-2B (I) / 32K 38.46 27.87 20.00 62.50 24.14 63.64 + Qwen3-VL-2B (I) / 64K 35.38 29.51 37.14 47.92 10.34 54.55 + Qwen3-VL-2B (I) / 128K 27.69 18.03 28.57 41.67 10.34 45.45 + Memory-augmented agents on the 195-subset + M3-Agent / 32K 19.49 18.03 22.86 29.17 6.90 13.64 + M3-Agent / 64K 18.97 18.03 17.14 37.50 3.45 4.55 + M3-Agent / 128K 19.49 22.95 20.00 22.92 6.90 18.18 + M3-Agent / 256K 23.59 22.95 28.57 27.08 13.79 22.73 + M2A / 32K 15.38 14.75 8.57 2.08 0.00 22.73 + M2A / 64K 15.90 18.03 14.29 25.00 0.00 13.64 + M2A / 128K 16.92 21.31 11.43 27.08 0.00 13.64 + M2A / 256K 16.41 21.31 11.43 25.00 0.00 13.64 + M3C / 32K 18.46 8.20 25.71 31.25 10.34 18.18 + M3C / 64K 17.95 9.84 25.71 27.08 6.90 22.73 + M3C / 128K 14.87 8.20 25.71 27.08 0.00 9.09 + M3C / 256K 16.92 11.48 25.71 29.17 0.00 13.64 + Memory-T1 / 32K 28.72 18.03 25.71 62.50 13.79 9.09 + Memory-T1 / 64K 28.72 19.67 22.86 56.25 17.24 18.18 + Memory-T1 / 128K 30.26 21.31 22.86 56.25 27.59 13.64 + Memory-T1 / 256K 29.23 18.03 25.71 58.33 24.14 9.09 + + + + +Bootstrap confidence intervals on overall agent accuracy. We resample with replacement at +the question level (1000 iterations, percentile method) on the canonical 195 questions to bound +the subset-induced uncertainty in each agent’s overall accuracy. The 95% confidence intervals at +32K (mean ± half-width, n = 195) are: Mem0 31.79 ± 6.67, MemOS 30.26 ± 6.15, MemAgent- +7B 32.82 ± 6.42, Memory-T1 28.72 ± 6.15, M3-Agent 19.49 ± 5.38, M3C 18.46 ± 5.38, M2A +15.38 ± 5.38. At 128K: Mem0 30.26 ± 6.67, MemOS 31.28 ± 6.67, MemAgent-7B 28.21 ± 6.41, +Memory-T1 30.26 ± 6.41, M3-Agent 19.49 ± 5.38, M3C 14.87 ± 4.87, M2A 16.92 ± 5.13. All +half-widths fall within [4.87, 6.67]% (mean 5.93%); the four text-only / caption-based pipelines +(Mem0, MemOS, MemAgent-7B, Memory-T1) cluster within their overlapping intervals at both +lengths and are uniformly above the three multimodal pipelines (M3-Agent, M3C, M2A), whose +intervals also overlap with one another but not with the upper cluster. Subset resampling therefore +preserves the qualitative cluster structure used in §4.3 and Table 16, and the agent–LVLM gap remains +larger than any subset-induced confidence band. + + + 50 + Scope of agent claims. The accuracies in Table 16 and Table 14 reflect each memory pipeline as +released, with the input adaptations documented in Appendix B: text-only agents (Mem0, MemOS, +MemAgent-7B, Memory-T1) consume BLIP-2 captions in lieu of pixels; M3-Agent consumes per- +session composite images rendered to fit its video-LVLM input format; M2A and M3C consume +original images via their native LVLM backbones. We do not claim these accuracies are upper bounds +on memory-augmented architectures in general — a pipeline that retained pixel-level evidence at +retrieval time, or that swapped BLIP-2 for a stronger captioner (cf. Appendix H), might score higher. +The conclusion supported by Table 16 and §4.3 is the narrower one: under each agent’s released visual +interface, current memory pipelines lose faithfulness to original visual evidence relative to direct- +LVLM grounding on the same 195 questions, while remaining length-robust across the 32K–256K +range. + +G.3 Coverage and Per-Answer Accuracy + + +Table 17: Coverage and Per-Answer Accuracy (n = 789, 699 answerable). Coverage = % of +answerable questions attempted. PA = accuracy on attempted answers. Models are grouped by family +(APIs, then open-source LVLMs). The 64K and 128K coverage decompositions inherit the same +trend (frontier APIs over-refuse, open-weight LVLMs over-attempt). + Model M Judge Coverage PA Refused + Claude Sonnet 4.5 I 36.50 49.50 57.51 353 + Gemini-3.1-Pro I 54.10 66.76 69.10 232 + GPT-5.4 I 52.72 63.95 73.15 252 + Kimi-K2.5 I 54.88 72.39 67.98 193 + Qwen3.5-122B I 58.68 87.70 62.32 86 + Cosmos-Reason2-8B I 46.13 99.43 43.31 4 + GLM-4.6V I 52.34 95.85 48.96 29 + Qwen3-VL-235B I 53.23 79.26 59.75 145 + Qwen3-VL-30B I 55.01 96.99 51.47 21 + Qwen3-VL-8B I 49.18 92.13 47.52 55 + + +Coverage–accuracy trade-off. Overall accuracy conflates retrieval ability with calibration. De- +composing performance into coverage (fraction of answerable questions attempted) and per-answer +accuracy (PA) reveals two contrasting strategies. GPT-5.4 reaches the highest PA (73.15%) but +refuses 36.05% of answerable questions; Qwen3.5-122B attempts 87.70% but achieves only 62.32% +PA. Claude Sonnet 4.5, which ranks 23rd overall, achieves a respectable 57.51% PA behind a 50.50% +refusal rate. The full decomposition appears in Table 17. At 128K, most models answer more +questions than at 32K (coverage rises) despite lower accuracy, confirming that false confidence, not +increased caution, is the dominant dynamic at longer contexts. + +Claude Sonnet 4.5: a calibration-driven outlier. Per-type decomposition sharpens the diagnosis. +Claude’s coverage varies from 41.26% (MSR) to 64.95% (TR), uniformly below the frontier-API +median: GPT-5.4 and Gemini-3.1-Pro cover 45–83% and 52–80% respectively. On IE and KU, +where Claude’s overall accuracy appears weak (28.05% and 29.31%), its per-answer accuracy reaches +63.55% and 62.96%—competitive with models ranked considerably higher—but a ∼55% refusal rate +on these types suppresses the headline number. TR has the highest coverage (64.95%), consistent +with timestamps providing explicit retrieval anchors that raise the model’s confidence. MSR is the +one type where both coverage and per-answer accuracy are low (41.26% / 32.20%), suggesting that +cross-session aggregation genuinely challenges the model rather than merely triggering its refusal +heuristic. We note that Sonnet is a mid-tier offering in Anthropic’s model family; the refusal-driven +deficit reflects model-specific calibration rather than a benchmark fairness concern. + +G.4 Wrong-Answer Error Analysis + +This appendix supplements the wrong-answer error analysis in §4.3 with (a) the motivation for the +seven-label taxonomy, (b) the per-label detection procedure used to produce Figure 16 and Figure 17 + + + 51 + (Appendix G.4), and (c) the modality re-grouping (Table 19) that turns the seven labels into the five +categories shown in Figure 4b of the main text. + +Folding into two meta-categories. The four near-miss labels (grounding failure, computation slip, +closed-set selection, stale retrieval) share a property: each requires the model to have located the +relevant evidence before erring. The three total-miss labels (unsupported answer, answerability failure, +non-answer pathology) share the opposite property: the model produced content, or a non-answer, +without a correct evidence anchor. This dichotomy turns retrieval success into a binary signal that can +be read off the wrong-answer distribution, which is what permits the retrieval-degradation account in +§4.3. + +Two labels are not enough. The 32K → 128K aggregate shift is ±15.17% by partition identity, so +reporting only the meta-category totals hides which labels absorb the shift. Unsupported answer alone +carries +10.23% of the shift, while answerability failure and non-answer pathology each move by +less than 3%. The single dominant label inside total-miss cannot be identified without the seven-way +split. + +Grouping by question type is not enough either. Partitioning wrong answers by question type +(five categories) obscures the per-type failure signature: MSR Arithmetic errors are computation slips, +TR order-ranking errors are closed-set swaps, and KU errors are stale retrievals. These signatures +surface only when the error label is separated from the question type; collapsing either dimension +merges failure modes that have different responses to context length. + +Why exactly seven. Each of the seven labels either binds to a structural constraint of a question- +type subset (numeric answer ⇒ computation slip; closed answer set ⇒ closed-set selection; KU +chain ⇒ stale retrieval; AR item ⇒ answerability failure; non-terminating generation ⇒ non-answer +pathology), or is a catch-all for free-form answers (grounding failure when the prediction is evidence- +anchored but wrong; unsupported answer when it is not). Every attempted-but-incorrect answer +receives exactly one label, and no label admits a strictly finer partition without introducing sub-labels +that collapse to zero support on subsets of the benchmark. Table 18 summarises the seven labels with +their detection method and category assignment. + +Detection, Phase 1 (rule-based). Five labels are assigned by deterministic rules that depend only +on the question type and the prediction string. Stale retrieval (KU only): the prediction fuzzy-matches +the item’s old_answer field with SequenceMatcher ratio ≥ 0.7 on normalized (lower-cased, +alphanumeric) strings. Computation slip (MSR Counting and Arithmetic): the prediction and the +reference both parse as numbers, and the two differ. Closed-set selection (TR Duration A/B, MSR +Yes/No, TR order-ranking): the prediction lies in the closed answer set for the item but is not the +reference element. Answerability failure (AR items): the prediction is a substantive answer rather +than a refusal, detected by the absence of any refusal keyword from a fixed list of 14 phrases such +as “insufficient information” and “cannot be determined”. Non-answer pathology (any item): the +prediction is empty, or is verbose reasoning with no “answer:”, “therefore”, or “final answer” anchor. +Phase 1 labels take priority over Phase 2 when both would apply; Phase 1 covers roughly 40% of +attempted-but-incorrect items at each context. + +Detection, Phase 2 (LLM-judged). The remaining items have free-form answers with no Phase 1 +rule matching. A GPT-4o-mini call reads the question, the reference answer, and the prediction, and +assigns one of the two free-form labels. Grounding failure: the prediction is semantically related to +the reference but incomplete, over-specific, mislocalized, or at the wrong granularity. Unsupported +answer: the prediction is a specific entity or value with no meaningful semantic overlap with the +reference. LLM classifications are cached in phase2_cache_{ctx}.jsonl and reused across runs, +so the results are reproducible without new API calls. + +Coverage and caveats. Phase 1 and the Phase 2 cache jointly cover roughly 90% of wrong answers +per context; the residual are invalid answer which failed to give a confirmed final answer. This inflates +the unsupported-answer share by at most ∼ 10% of its own value; rerunning with the full LLM pass +is left as an extension because the direction and magnitude of the near-miss → total-miss shift are +stable under that default assignment. + + + 52 + Non-answer + pathology + Answerability 3.70% + failure + 3.87% + + + + + Grounding + failure + 29.66% + Unsupported + answer + 23.00% + + Near-miss + 69.44% + Total-miss + 30.57% + + Stale + retrieval + 4.53% + + Computation + Closed-set slip + selection 21.32% + 13.93% + + +Figure 16: Distribution of wrong-answer types at 32K context (n = 5,592 attempted-but-incorrect +LVLM answers, out of 27 × 789 = 21,303 evaluations). Near-miss errors (evidence located before +erring) account for 69.44% of wrong answers; total-miss errors (no correct evidence anchor) for +30.57%. + + +Table 18: Seven-label wrong-answer taxonomy. Labels partition every attempted-but-incorrect +answer into near-miss (evidence located) vs. total-miss (no evidence anchor). + Label Category Detection Definition + Grounding failure Near-miss LLM Right region, wrong detail. + Computation slip Near-miss Rule Right operands, wrong arithmetic. + Closed-set selection Near-miss Rule Right set, wrong element. + Stale retrieval Near-miss Rule Right fact, pre-update version. + Unsupported answer Total-miss LLM No anchor; fabricated content. + Answerability failure Total-miss Rule Answered an unanswerable item. + Non-answer pathology Total-miss Rule Never produced a final answer. + + + +Five-category modality view of wrong answers (Figure 4b). The wrong-answer pie (Figure 16) +reports the seven-label distribution at the attempted-but-incorrect level; the per-type bars in Figure 4b +(main text) re-group every wrong answer along a complementary modality axis by joining its seven- +label tag with the per-question image-dependency annotation (image-essential / image-supportive +/ text-sufficient, defined in §3.4). The result is five disjoint modality categories that exhaust every +attempted-but-incorrect answer; Table 19 gives the exact mapping. + • Visual. Grounding failure or unsupported answer on an image-essential question—the visual + evidence was the source of the missing information. + • Textual. Either (i) grounding failure or unsupported answer on a text-sufficient question, or (ii) + any stale-retrieval error (intrinsically a textual-update miss on KU)—text was the source of the + missing information. + • Mixed. Grounding failure or unsupported answer on an image-supportive question, where image + and text both contribute to the answer and either could be at fault. + + + 53 + • Reasoning. Computation slip, closed-set selection, or answerability failure—a label-specific + reasoning-shaped error (off-by-one arithmetic, wrong A/B choice, or substantive answer to an + unanswerable item). +• Output. Non-answer pathology (empty or non-extractable response). + + +Table 19: Mapping from the seven-label wrong-answer taxonomy (× per-question image dependency) +to the five modality categories of Figure 4b. A single column entry indicates that the dependency +does not affect the assignment. + Seven-label image-essential image-supportive text-sufficient + Grounding failure Visual Mixed Textual + Unsupported answer Visual Mixed Textual + Stale retrieval Textual + Computation slip Reasoning + Closed-set selection Reasoning + Answerability failure Reasoning + Non-answer pathology Output + + +The seven-label and five-modality views are two lenses on the same wrong-answer set. The seven- +label view records what kind of error the model made (e.g., right answer set but wrong element); the +five-modality view records which evidence channel the error relied on. Reading the two together +explains the per-type asymmetries in Figure 4b: a question type whose closed-set or arithmetic +structure makes computation-slips and selection-swaps the dominant Phase-1 labels (e.g., MSR +Counting/Arithmetic, MSR Yes/No, TR Duration A/B, TR order-ranking) will accordingly show a +large Reasoning share regardless of how reliably the model retrieves evidence—which is why the +Reasoning share in MSR/TR should be cross-checked against the oracle-retrieval diagnostic in §G.7 +before being interpreted as a reasoning bottleneck. + + + MSR: Arith -4.90 +1.10 +1.50 +2.40 + 20 + + MSR: Count -6.50 +4.10 +1.80 +0.60 + 15 + + MSR: Y/N -9.20 +3.20 +6.00 + + + + + Δ share, 128K − 32K (pp) + 10 + + + TR: Duration -9.40 +1.90 +8.40 -0.90 5 + + + TR: Date +1.40 -21.10 +18.40 +1.20 0 + + + TR: Order +6.00 -4.20 -1.70 −5 + + + −10 + IE -2.20 -10.00 +10.10 +2.10 + + −15 + KU -10.20 -8.00 +15.10 +3.10 + + −20 + AR + + tr. ip Se + t + ng te + d + lit + y + we + r + Re Sl d- di or bi + e p. se un p a ns + al m lo ro up er -A + St Co C G s sw on + Un An N + + +Figure 17: Wrong-answer error-type shift from 32K to 128K by question type (n = 789). Unsup- +ported answer (+10.23%) replaces grounding failure (−5.38%) and computation slip (−4.82%). + + +Case studies. To illustrate each error category concretely, we present one representative failure +case per label below. Each card shows the question, the reference answer, and the model’s incorrect +prediction at the 32K context length. + + + 54 + Case Study: Grounding Failure +The model located the relevant evidence region but extracted or misread a specific detail—answering at wrong granularity +or confusing a neighboring value. +Model: claude-sonnet-4.5 Task: Information Extraction Context: 32K +Question. Where on the sign is the photo of the mountains located? +Reference Answer. In the bottom right corner +Model Prediction. In the upper right corner of the kiosk sign. + + + +Case Study: Computation Slip +The model identified the correct operands and operation type but produced an incorrect arithmetic result (off-by-one, +summation error, or unit mismatch). +Model: claude-sonnet-4.5 Task: MSR — Arithmetic Context: 32K +Question. How much total have I spent on coffee makers? +Reference Answer. $260.00 +Model Prediction. $150.00 + + + +Case Study: Closed-Set Selection Error +The model recognised the closed answer set (A/B, Yes/No, or a fixed ordering) but selected the wrong element—typically +swapping two options. +Model: glm4.6v Task: TR — Duration Comparison Context: 32K +Question. Which of the following two durations is longer? Duration 1: the time spent working at JTB Corporation; +Duration 2: the time spent working at Rakuten Group. +Reference Answer. B +Model Prediction. a + + + +Case Study: Stale Retrieval +The model retrieved a factually correct but outdated answer—returning a previous state instead of the most recent update +in the knowledge-update chain. +Model: cosmos-reason2-8b Task: Knowledge Update Context: 32K +Question. What’s my favorite dip now? +Reference Answer. spinach artichoke dip +Model Prediction. Hummus +Previous (Outdated) Answer. hummus + + + +Case Study: Unsupported Answer +The model’s prediction has no meaningful semantic overlap with the reference; the response appears fabricated with no +correct evidence anchor. +Model: cosmos-reason2-8b Task: MSR — Arithmetic Context: 32K +Question. How much total have I spent on blenders? +Reference Answer. $433.99 +Model Prediction. You have spent a total of $40 on blenders. + + + +Case Study: Answerability Failure +The question is deliberately unanswerable (evidence image removed), yet the model produced a confident substantive +answer instead of refusing. +Model: glm4.5v Task: Answer Refusal Context: 32K +Question. What color are the words "rice is life" written in? +Reference Answer. NOT_MENTIONED +Model Prediction. The words "rice is life" are written in black. + + + + 55 + Case Study: Non-Answer Pathology + The model either produced an empty response or generated verbose reasoning without an extractable final answer. + Model: glm4.5v Task: MSR — Entity Resolution Context: 32K + Question. Based on the tweets and my observations, is my vehicle, Bluebird, subject to the recall mentioned by + @Woolworths_SA? + Reference Answer. No + Model Prediction. To determine if your vehicle, Bluebird, is subject to the recall mentioned by @Woolworths_SA, + follow these steps: 1. Identify the Recall Details: Check the Woolworths tweet for the specific vehicle mo. . . + + +G.5 Context-Length Analysis + +This appendix collects the context-length-centric analyses that complement the memory-ability and +multimodal perspectives in the main text. + +Size helps long-context retention within a family. Among the 22 open-source LVLMs with +complete 32K–128K data, the Spearman correlation between model size and the 32K-to-128K +retention ratio reaches ρ = 0.62 (p = 0.002, Figure 18): larger models generally hold up better +at 128K. Within the Qwen3-VL Instruct family, retention climbs from 0.71 at 8B to 0.90 at 235B; +within the Qwen3.5 dense series, 27B retains 0.77 versus 0.65 at 9B and 0.57 at 4B. + +Architectural family shifts both the ceiling and the degradation profile. At roughly matched +scale, Qwen3.5-122B-A10B loses more than twice as much accuracy at 128K as Qwen3-VL-235B- +A22B (−13.18% vs. −5.57%), despite similar active-parameter budgets. The type-wise profile also +diverges: the Qwen3-VL Instruct branch keeps TR high at 128K (52–55%) but lets KU collapse +(30–24%), whereas the Qwen3.5 branch preserves KU (46.03%) at the cost of TR (41.75%). Family +choice therefore shapes both overall retention and which question types give way first. + +No model family wins on every type. Every top-tier open-weight model at 128K is an MoE variant, +so the informative architectural axis is training recipe, not sparse versus dense. Kimi-K2.5 leads MSR +at 32K (44.06%) but drops to 28.67% at 128K; Gemini-3.1-Pro is the only model simultaneously +top-2 on IE, best on KU, and above 29% on MSR at 128K. Specialization follows training recipe +rather than dense-vs-sparse design. + +Hard questions stay hard across context lengths. The 280 questions that are hard at 32K (solved +by <20% of models) average 9.07% accuracy at 32K and 8.52% at 128K, so expanding context does +not unlock them. The difficulty floor reflects the skill requirements rather than an artifact of context +length. + +Statistical validation of degradation monotonicity. In a well-behaved needle-in-haystack bench- +mark, accuracy should decrease (or hold flat) as context grows, since the evidence is identical and +only the surrounding noise increases. We systematically scanned all 27 LVLMs and 7 agents across +five types and two context transitions (32K→64K, 64K→128K), identifying 72 transitions where +accuracy nominally increases—38 among LVLMs and 34 among agents. None survive Bonferroni +correction (α/72 = 0.0007). For the 38 LVLM transitions we applied McNemar’s exact test on +paired per-question binary outcomes (same 789 questions at both lengths), classifying each question +as stable-correct, stable-wrong, flip-to-correct, or flip-to-wrong. Only one LVLM transition reaches +p < 0.05 uncorrected: Qwen3-VL-2B-Thinking MSR at 64K→128K (+5.59%, p = 0.022, 9 flip-to- +correct vs. 1 flip-to-wrong). This improvement is a degenerate-output artifact: 8 of 9 flip-to-correct +questions hit the 2,048-token generation cap at 64K but produced normal outputs at 128K—stochastic +recovery in a highly unstable model (60.8% MSR degenerate rate). All 34 agent anomalies are +attributable to small per-type sample sizes (n = 22–61): a single question flip produces a 2–5% +swing. + +Bidirectional churn underlies apparent reversals. Per-question tracking across 21 LVLMs with +judge data at all three context lengths reveals that context transitions induce bidirectional churn: at +each step, models simultaneously lose 60–150 questions they previously answered correctly and gain +30–80 new correct answers. The observed accuracy decay is the net of these two opposing flows, +not a uniform loss of signal. Of questions that flip at 32K→64K, 38.7% flip back at 64K→128K, + + + 56 + confirming that a substantial fraction of single-step churn is stochastic. Context-robust models are +not exempt from this churn—they simply balance most of it: at 64K→128K, Gemini-3.1-Pro flips +∼71 questions in each direction (correct→wrong and wrong→correct) at an ≈18% churn rate, and +Kimi-K2.5 flips ∼83 in each direction; the slight asymmetry yields the small net Overall drops shown +in Table 13 (1pp for Gemini, 2pp for Kimi) rather than a true zero-sum reversal. This balanced +replacement, rather than static resistance, explains their flat accuracy trajectories. +We also tested whether MSR questions are systematically easier at 64K, since 10 individual LVLMs +show MSR improvement at that length. Cross-model question-level analysis reveals no systematic +positioning effect: more MSR questions become harder at 64K than easier across models, ruling out +a dataset-level artifact. +In summary, the expected monotonic degradation holds across the benchmark: no context-length +artifact is detected, and the rare per-model reversals are explained by bidirectional churn, model- +specific answer variability, or small-sample noise in agent subsets. + +AR accuracy degradation. AR accuracy degrades monotonically with context length across all +Qwen3-VL sizes, but the rate depends on both scale and decoding mode (Table 20). At the 235B +Instruct tier the drop is modest (−5.6% from 32K to 128K), whereas 2B-Thinking collapses from +87.8% to 14.4% (−73.3%): truncated reasoning traces produce substantive answers instead of +refusals. + +Table 20: Answer Refusal (AR) accuracy across input lengths for Qwen3-VL Instruct vs. Thinking +modes. All models lose AR accuracy at longer inputs; thinking mode degrades faster. + Size Mode 32K AR 64K AR 128K AR ∆32→128 + 235B Instruct 97.8 95.6 92.2 −5.6 + 235B Thinking 88.9 75.6 67.8 −21.1 + 30B Instruct 93.3 78.9 63.3 −30.0 + 30B Thinking 71.1 65.6 60.0 −11.1 + 8B Instruct 90.0 82.2 66.7 −23.3 + 8B Thinking 80.0 54.4 45.6 −34.4 + 4B Instruct 92.2 75.6 50.0 −42.2 + 4B Thinking 82.2 70.0 52.2 −30.0 + 2B Instruct 72.2 65.6 47.8 −24.4 + 2B Thinking 87.8 54.4 14.4 −73.3 + + + Qwen3-VL Instruct scaling at 32K + Active parameters — not total — drive Qwen3-VL performance. + 60 + MoE 30B with only 3B active + beats dense 8B (+6.2pp) + 55 + Overall accuracy at 32K (%) + + + + + 50 QVL-30B + (A3B / 30B total) + QVL-235B + (A22B / 235B total) + 45 QVL-8B + + + + 40 QVL-4B + + + + 35 + QVL-2B + Dense (Qwen3-VL) + MoE, plotted at active params + 30 + 2B 3B 4B 8B 22B + Active parameters (billion, log scale) + +Figure 18: Model-size scaling within the Qwen3-VL Instruct family (n = 789). Dense scaling from +2B to 8B is monotone; the 30B (3B-active) MoE outperforms the 8B dense variant, indicating that +active-parameter efficiency drives short-context performance more than total size. + + + + 57 + G.6 Agent Underperformance: Where in the Pipeline Is Information Lost? + +The main-text analysis (§4.3) establishes that agents trail LVLMs by 18–34% overall, exhibit inverted +type profiles, and suffer a 22% modality gap on image-essential questions. This appendix asks where +in the pipeline the information is lost. We evaluate all seven agents on the canonical 195-question +subset of Appendix G.2; the 32K-to-256K per-type roster is reproduced in Table 14. + +Agent failures split into retrieval-dominated and comprehension-dominated modes. For the +three agents with retrieval logs (Mem0, Memory-T1, and M3C), we decompose each wrong answer by +whether the evidence was retrieved before the error occurred. Evidence recall is defined as the fraction +of ground-truth evidence sessions surfaced by the retriever, and an error is classified as a retrieval +failure when recall falls below 0.5 (fewer than half of the required evidence sessions are retrieved); +the threshold is conservative, since most questions in M EM L ENS require all evidence sessions to +answer correctly. M3C is retrieval-bottlenecked, with 78.1% of its errors occurring because the LoRA +session retriever never surfaces the relevant evidence (mean recall 0.26). Mem0 and Memory-T1 sit +at the opposite end: they retrieve the evidence at high recall (0.82 to 0.89), yet 87 to 95% of their +errors occur after successful retrieval, indicating that the backbone model cannot reason over the +surfaced content (Figure 19). The two failure modes call for different interventions—better retrieval +for M3C and stronger reading comprehension for Mem0 and Memory-T1. + +Pipeline architecture dominates backbone quality. M2A builds on Qwen3-VL-8B-Instruct, a +current-generation 8B stock backbone, yet it scores only 14.21%. Memory-T1, whose backbone +is a 2.5× smaller text-only Qwen2.5-3B with RL fine-tuning, scores 29.50%, a 15.29% advantage +on a weaker substrate. The comparison conflates architecture with task-specific training, since +the RL objective of Memory-T1 targets temporal retrieval and partly explains its TR dominance +(63.46%), but the direction of the effect is consistent across agents. The cleanest quantitative measure +of pipeline cost is the backbone-matched contrast: M2A at 14.21% against direct evaluation of +Qwen3-VL-8B-Instruct at 49.18% yields a 34.97% deficit on the same backbone. Mem0 (32.50%, +Qwen3-8B backbone) and MemOS (34.00%, Qwen3-8B backbone) share the text-only Qwen3-8B +substrate but cannot be matched against a direct LVLM at the same scale, since text-only Qwen3-8B +is not in our LVLM roster; their backbone ablations (Table 21) span 14.65% for Mem0 and 2.50% +for MemOS across the alternative backbones, large in absolute terms yet still well below the 34.97% +deficit measured on M2A. Taken together, the overall agent system—combining architecture and +training recipe—matters more than backbone scale alone, a conclusion now supported by both the +cross-agent comparison above and the within-architecture ablation below. + +Controlled backbone ablation. The cross-agent comparison above is confounded by architecture, +training recipe, and backbone quality varying simultaneously. To isolate the backbone factor, we +re-evaluate Mem0 and MemOS with alternative backbones while keeping each pipeline unchanged +(Table 21). Within the Mem0 FAISS architecture, swapping the default Qwen3-8B for the larger +gpt-4.1-mini lifts overall accuracy by 10.65% to 43.15%, while substituting Qwen2.5-7B drops it +by 4.00% to 28.50%, for a total spread of 14.65% across the three backbones. Within MemOS, +replacing the default Qwen3-8B with Qwen2.5-7B counter-intuitively raises overall accuracy by +2.50%, indicating that newer-generation backbones do not always translate to better in-pipeline +behavior at this scale. However, both spreads remain well below the 34.97% deficit measured on +M2A, reinforcing the conclusion that architecture is the dominant factor. +Per-type profiles shift markedly across backbones. In Mem0, the default Qwen3-8B reaches 77.27% +AR while the Qwen2.5-7B variant achieves perfect refusal (100%), indicating that the backbone’s +intrinsic calibration against hallucination propagates through the memory pipeline. Conversely, +Qwen3-8B leads on TR (50.00% vs. 32.69%) within the same FAISS architecture, suggesting +complementary strengths that the pipeline cannot arbitrate. Crucially, all backbone variants preserve +the context-length invariance observed for the default configurations: the 32K-to-256K range stays +below 5% for every variant, confirming that flatness is an architectural property independent of +backbone quality. + +Context invariance is real but insufficient. The context stability noted in §4.3, with six of seven +agents staying within ±7% across 32K to 256K, is genuine robustness rather than a floor effect. +On correctly answered questions, the Jaccard overlap between adjacent context lengths exceeds the + + + 58 + Table 21: Backbone ablation within fixed agent architectures at 32K. Each row uses the same pipeline +with a different backbone model, evaluated on the 195-question canonical subset. ∆: spread between +best and worst backbone within each framework. + Framework Backbone IE MSR TR KU AR Ov. + Mem0 Qwen3-8B (default) 13.11 25.00 50.00 17.24 77.27 32.50 + Mem0 gpt-4.1-mini 31.15 33.33 42.86 44.83 90.91 43.15 + Mem0 Qwen2.5-7B 3.28 19.44 32.69 31.03 100.00 28.50 + ∆ 14.65 + MemOS Qwen3-8B (default) 18.03 22.22 40.38 24.14 68.18 34.00 + MemOS Qwen2.5-7B 29.51 22.22 40.38 20.69 90.91 36.50 + ∆ 2.50 + + +random baseline of independent draws at the observed accuracy by a factor of 3.3 to 6.9 across agents. +At 128K this advantage narrows the gap to the Qwen3-VL-8B tier from 35% to 17%, yet the absolute +deficit remains large because the information lost during memorization and retrieval continues to +outweigh the context-robustness advantage. + +A common question subset defeats multiple agents regardless of architecture. On the shared +answerable subset at 32K, 52% of the questions are answered incorrectly by all four agents with +complete per-question logs (M3-Agent, M3C, M2A, and Memory-T1, with overall accuracy in the 14 +to 30% range), despite architecturally disjoint retrieval pipelines that include ColPali (M3-Agent), +LoRA session retrieval (M3C), dual-layer SQLite (M2A), and BM25 (Memory-T1). Mem0, MemOS, +and MemAgent-7B are excluded from this overlap analysis because their evaluation outputs lack +per-question identifiers needed for cross-agent alignment, and the four included agents already span +the four retrieval paradigms represented in our pool. The all-wrong set is dominated by KU (69%) +and MSR (69%), with TR the least affected type (21%). Whether this ceiling persists for stronger +agents or instead reflects a shared bottleneck of current sub-10B retrieval-based memory systems is +an open question that would require matched-scale implementations we leave to future work. + + Correct Comprehension failure Retrieval failure + + 100% + + 80% + Proportion (%) + + + + + 60% + + 40% + + 20% + + 0% + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + + k + 12 k + 25 k + 6k + 32 + 64 + 8 + + 32 + 64 + 8 + + 32 + 64 + 8 + + + + 32 + 64 + 8 + + 32 + 64 + 8 + + 32 + 64 + 8 + + + + 32 + 64 + 8 + + 32 + 64 + 8 + + 32 + 64 + 8 + + + + 32 + 64 + 8 + + 32 + 64 + 8 + + 32 + 64 + 8 + + + + + Mem0 IE + M3C Memory-T1 Mem0 MSR + M3C Memory-T1 Mem0 TR + M3C Memory-T1 Mem0 KU + M3C Memory-T1 + + +Figure 19: Retrieval attribution for three agents with retrieval logs at 32K, decomposed by question +type. Each bar partitions outcomes into correct (green), comprehension failure (yellow, evidence +retrieved but answer wrong), and retrieval failure (red, evidence not retrieved). M3C is retrieval- +dominated across all types; Mem0 and Memory-T1 are comprehension-dominated despite high +evidence recall. + + +G.7 MSR Ceiling Diagnostic: Retrieval-Bounded Difficulty + +MSR is the hardest ability in M EM L ENS (§4.2), with cross-session aggregation over three to eight +sessions defeating every evaluated system. A natural concern is whether this ceiling reflects intrin- +sic question difficulty—i.e., the aggregation tasks are too hard for current models—or a retrieval +bottleneck imposed by the long-context evaluation format. +The cross-modality ablation (Table 3) provides a direct diagnostic. When frontier models receive +ground-truth evidence sessions with their associated images—bypassing the haystack retrieval chal- +lenge entirely—MSR accuracy reaches 100.00% for GPT-5.4 and 90.21% for Gemini-3.1-Pro. Both + + + 59 + models achieve near-perfect cross-session aggregation when the required three to eight evidence +sessions are delivered directly, confirming that the counting and arithmetic operations underlying +MSR are well within frontier reasoning capacity. +The gap between oracle-retrieval MSR (90–100%) and full-benchmark MSR therefore confirms that +the ceiling is retrieval-bounded rather than reasoning-bounded: models can solve the aggregation task +once evidence is located, but fail to collect the required evidence sessions from a long multi-session +conversation. This interpretation refines the per-type error analysis in §4.3: surface aggregation +failures in the error breakdown are downstream consequences of incomplete evidence collection +rather than deficits in reasoning capacity. When a model locates only four of six required sessions, its +count is necessarily wrong, registering as an aggregation error in the taxonomy despite originating +from a retrieval miss. The remaining ∼10% gap for Gemini-3.1-Pro (90.21% vs. 100%) suggests that +a small fraction of MSR items—likely the more complex arithmetic patterns (MSR-Arithmetic)—do +challenge reasoning capacity even with perfect evidence delivery, but the dominant bottleneck is +evidence location, not evidence comprehension. + +G.8 Session Indistinguishability Validation + +Evidence sessions and haystack sessions are produced by the same generation pipeline. Both use +GPT-5.1 as the user and Gemini-3-Pro as the assistant, follow the same prompt templates with +identical style constraints (250–350 words per turn, knowledge-oriented follow-ups rather than +personal or social ones), and share the same image integration pathway (§F.4). The sole structural +difference is that an evidence session embeds one or more needle facts into the user turns; a six-stage +validator chain (rule-based length and format checks, n-gram overlap detection, semantic leakage +filtering, chain-ambiguity preservation, and end-marker verification) then screens each generated turn, +rejecting and regenerating any output that departs from the haystack register or reveals the embedded +fact through surface cues. Session indistinguishability is therefore a construction-level property: +because both session types emerge from the same generator under the same constraints, systematic +stylistic divergence would require the validator chain to fail consistently across thousands of sessions. +The conversation history assembly stage imposes type-specific ordering constraints beyond the +random placement and non-contamination invariants described in the main text. For MSR and +KU questions, whose answers depend on the temporal sequence of facts across sessions (a four- +step knowledge update chain or a cross-session counting task, for instance), the evidence sessions +are inserted into the haystack sequence in their original fact order, preserving the chronological +progression that the question probes. For TR questions, each evidence session is assigned a timestamp +that strictly precedes the question’s reference date, so that temporal reasoning over session dates +remains internally consistent. These ordering constraints, together with the non-contamination +invariant and the visual-density normalization by text-only filler sessions, ensure that evidence +positions carry no structural signal beyond what random interleaving would produce. +To validate the indistinguishability claim quantitatively, we trained two binary classifiers to distinguish +evidence from haystack sessions. From the evidence and haystack sessions in the benchmark, we +drew a balanced 698/698 sample and applied five-fold cross-validation with question-grouped splits, +ensuring that no question appears in both the training and test sets of any fold. Speaker labels +were normalized to a uniform User:/Assistant: format and the placeholder token was +removed, eliminating trivial classification shortcuts. A TF-IDF logistic regression model (unigrams +and bigrams, 10,000 features, sublinear term frequency) achieved 56.71% ± 1.20% accuracy with +an ROC-AUC of 0.59, and a fine-tuned DeBERTa-v3-base classifier (512-token truncation, three +epochs) reached 57.92% ± 0.85% with an ROC-AUC of 0.55. Both results sit only marginally above +the 50% chance floor for balanced binary classification (Cohen’s h < 0.16, a small effect). A pooled +binomial test against H0 : p = 0.5 on the TF-IDF predictions (805 of 1,396 correct) yields a two-sided +p < 10−7 ; the rejection reflects the large sample size rather than a practically meaningful signal, as +the absolute accuracy gap of ∼7 percentage points above chance provides negligible leverage for +locating evidence within a 30+-session history. +A follow-up vocabulary ablation progressively removed the K most discriminative n-grams from the +TF-IDF feature set (K = 20, 50, 100, 200, 500, 1,000; balanced removal from each class direction, +ranked on training-fold coefficients only to prevent test-fold leakage). Across the entire sweep, accu- +racy remained in a narrow band between 54.77% and 56.71%, never exceeding 7 percentage points +above chance. Removing up to 1,000 top-ranked n-grams leaves accuracy essentially unchanged, + + + 60 + indicating that the weak signal is diffuse rather than concentrated in identifiable stylistic markers. +Together, the construction-level design and the post-hoc classifier results establish that evidence +sessions carry no practically exploitable stylistic fingerprint relative to the surrounding haystack; the +retrieval difficulty reported in §4.2 is not inflated by surface-level cues that might allow a model to +shortcut evidence location. + + +H Detailed Limitations and Future Work + +Synthetic conversations and human-in-the-loop naturalness. The evidence and haystack sessions +in M EM L ENS are LLM-generated (GPT-5.1 user, Gemini-3-Pro assistant; §3.2), with ShareGPT and +UltraChat [65] dialogues used as filler in the surrounding context. Naturalness is enforced through an +intensive human-in-the-loop review pipeline rather than automatic metrics alone (§E.1): Round 1 +audits each candidate question; Round 2 audits each session against a conversational-naturalness +criterion covering colloquial phrasing, turn-taking coherence, and indirect embedding of factual +content, returning stilted or overly formal sessions to the generation pipeline until annotators accept +them as plausible user–assistant exchanges; Round 3 audits the assembled haystack. This pipeline +reduces the candidate pool from ∼20k to 789 final questions, following the human-review practice +established by LongMemEval [14]. As a complementary surface-shortcut control, the session- +indistinguishability validation (§G.8) trains binary classifiers to separate evidence from haystack on +text features and reports only marginally above-chance accuracy (DeBERTa F1 57.92%, TF-IDF +56.71%), confirming that evidence carries negligible stylistic signal for localization. M EM L ENS +therefore stands as a controlled diagnostic stress test for cross-modal evidence retrieval, multi-session +aggregation, and length-controlled comparability under uniform construction; characterizing the gap +to the distribution of real long-term human–assistant interactions remains an open research question. + + +Question generator and test-taker overlap. The final question-generation model in our pipeline +is Gemini-3-Pro (§3.2, §C.4), and the top-evaluated model in our leaderboard is Gemini-3.1-Pro +(§4.2, Table 13); these are different versions in the same Gemini-3 family, so the residual concern +is intra-family generator familiarity rather than strict model-identity circularity. The construction +pipeline constrains question content rather than rewarding any particular output style: each question +is conditioned on a fixed (entity, image, abstracted paragraph) triple under a deterministic prompt +template, and is then filtered by automated text-leakage and quality checks followed by human review +(§C.4, §3.3). Two oracle-retrieval diagnostics already in the paper provide indirect evidence that the +128K leaderboard position of Gemini-3.1-Pro is not explained by intra-family generator familiarity. +On the answerability test with full evidence supplied (Table 3), GPT-5.4 reaches 93.13% overall +versus 89.42% for Gemini-3.1-Pro; on the MSR cross-modality oracle (§G.7), GPT-5.4 reaches +100.00% versus 90.21% for Gemini-3.1-Pro. Under conditions where retrieval is bypassed—the +conditions in which generator-style familiarity would most plausibly manifest—Gemini-3.1-Pro does +not lead, so the 128K full-benchmark inversion is more consistent with retrieval-robustness than +with intra-family familiarity bias. We did not, however, run a direct test in which an independent +generator (e.g., Claude-Sonnet-4.5, GPT-5.4) regenerates a stratified subset of questions on the same +input triples and the top LVLMs are re-ranked on those items; this targeted ablation would isolate any +residual generator-familiarity confound and is left to future work. + + +Judge limitations. LLM-as-judge validation in this paper covers 800 of 73,784 judge calls +(≈1.08%) for cross-family agreement against GPT-5.4-mini and 484 items for human verifica- +tion by three annotators with consensus adjudication. Three limitations follow. First, κ = 0.86 +measures judge-vs-consensus-label agreement; we do not separately report inter-annotator κ among +the three raters, since disagreements were resolved to a single label before release, so inter-annotator +reliability on this benchmark remains an open question. Second, the 29 false positives versus 2 false +negatives indicate the judge is systematically lenient on borderline partial-match and verbose-correct +outputs; the format-dependent bias correction (Appendix E.2) addresses the short-output subset +but not the full leniency channel. Third, per-subtype judge agreement (Appendix E.2) inherits the +reporting taxonomy of §C.3 (9 subtypes, all n ≥ 46); finer per-generator splits inside MSR Entity +and TR Temporal Grounding remain statistically noisy. A separate inter-annotator reliability study on +the 120 double-annotation-tagged items and a per-rare-type audit are left to future work. + + + 61 + Static-length vs. streaming evaluation. M EM L ENS queries each frozen multi-session history +offline at four context lengths (32K–256K tokens); a complementary online streaming protocol +that preserves temporal causality between memory writes and queries [98] would further probe +irreversible-update and forgetting dynamics in the multimodal setting, and is left to future work. + +Broader impacts. M EM L ENS provides a length-controlled diagnostic for multimodal conversa- +tional memory, intended to make memory-faithfulness regressions in long, multi-session multimodal +assistants visible to model developers before deployment. Two negative impacts are foreseeable. +First, leaderboard targeting can encourage over-fitting to the construction pipeline (§3.2); the released +anti-shortcut filter (§F.5) and the canonical 195-question subset (§G.2) are designed so that such +over-fitting becomes detectable rather than rewarded. Second, the synthetic dialogue construction +does not capture the full distribution of real user–assistant interactions, so accuracy on M EM L ENS is +a necessary but not sufficient indicator of deployed memory quality. As mitigation, M EM L ENS is +released with versioned frozen tags so that any specific evaluation run remains reproducible; images +are sourced from the public web and each retains its original source-site license, as documented in +§C.7. + +Ethics statement. M EM L ENS is released for the evaluation of multimodal long-term conversational +memory, and we do not endorse using its dialogues for training or fine-tuning, since exposure of +evaluation items would compromise diagnostic value; the versioned frozen tags (§C.7) and the per- +image perceptual hashes support detection of leakage in derived models. Two dual-use considerations +follow. First, memory-augmented assistants whose deployed memory faithfulness regresses below +the levels reported here may amplify mis-recall in user-facing settings; the per-ability decomposition +(§4.3) is designed so that such regressions are visible before deployment. Second, the synthetic- +dialogue construction (§3.2) does not capture the distribution of real long-term human–assistant +interactions, so M EM L ENS scores are a necessary but not sufficient indicator of deployed memory +quality. Image sourcing, licensing, privacy, and takedown protocol are documented in Appendix C.7. + + +I Design Rationale for the Memory Ability Taxonomy + +This appendix motivates the five memory abilities evaluated in M EM L ENS and argues that they jointly +cover the space of capabilities required for long-term multimodal conversational agents. +Recent memory-agent benchmarks converge on three functional dimensions of conversational mem- +ory [14, 17]: retrieval (can the agent recall stored information?), reasoning (can it synthesize in- +ferences across distributed evidence?), and knowledge update (can it correctly update its internal +state as the conversation evolves?). Our taxonomy is designed so that the five abilities collectively +span all three dimensions. IE addresses retrieval at single-session granularity; MSR and TR exercise +reasoning from complementary angles; and KU and AR probe opposing facets of knowledge update. +We justify each ability below. + +Information Extraction (IE): single-session retrieval. IE operationalizes the most fundamental +function of memory: accurate recall of previously encountered information. Each IE question requires +the agent to locate a specific evidence session within a long interaction history and extract the relevant +fact. The multimodal extension adds a layer of image understanding: Entity-subtype questions present +an abstracted entity visible only in an image, requiring visual recognition before textual retrieval, +while PrevInfo-subtype questions ask the agent to recall a visual detail from an image shared in an +earlier session. This two-hop structure ensures that IE measures genuine cross-modal memory rather +than unimodal text matching. + +Multi-Session Reasoning (MSR): cross-session aggregation. MSR extends single-session re- +trieval to aggregative reasoning over information distributed across three to eight sessions. The tested +operations (counting, arithmetic, and entity resolution) all require the agent to identify multiple +relevant sessions, extract the pertinent facts from each, and combine them into a coherent answer. We +retain a subset of text-only MSR questions alongside multimodal ones. These text-only items serve as +a controlled modality ablation: they isolate cross-session reasoning from the added difficulty of visual +retrieval, yielding a cleaner signal of the reasoning capability itself. Including them also preserves +ecological validity, since not every piece of cross-session information in a realistic conversation + + + 62 + history involves images. Mem-Gallery [17] adopts a compatible design in its Multi-entity Reasoning +subtask, where the entities involved can be textual or visual. + +Temporal Reasoning (TR): temporal awareness. Temporal reasoning has emerged as a distinct +research focus in LLM evaluation [14, 99–101]. In long-running interaction histories, temporal +information takes heterogeneous forms: natural-language date expressions in utterances, session-level +timestamps in metadata, and implicit ordering from the session sequence itself. Our TR questions +test whether the agent can jointly process these signals together with visual content (e.g., clock faces +and calendar images that replace textual temporal expressions) to answer questions about durati +fon comparison, event ordering, and temporal grounding. This ability is especially relevant for +personal assistants, where users frequently ask questions that require temporal contextualization of +past interactions. + +Knowledge Update (KU): current-state tracking. A defining requirement for personalized con- +versational agents is the ability to track how user attributes evolve over time. Users routinely update +preferences, correct earlier statements, and revise plans across sessions [57]. Each KU question +presents a chain of four successive updates to a single attribute, and the agent must identify the +current state rather than earlier, superseded values. This tests selective forgetting as much as recall: +the agent must not only retrieve the relevant update chain but also recognize which value is most +recent. Mem-Gallery [17] identifies the same concern under Knowledge Resolution, framing it as +maintaining consistency when “new, contradictory information appears in the dialogue.” + +Answer Refusal (AR): epistemic calibration. AR tests whether the agent can recognize the limits +of its available evidence. Each AR question is constructed by removing all evidence sessions from +an otherwise answerable instance, so that no supporting information remains in the conversation +history. A correct agent must decline to answer rather than hallucinate a plausible response. This +ability addresses a well-documented failure mode of large language models [58] and is essential for +trustworthy deployment: a personal assistant that fabricates answers from absent evidence is more +harmful than one that acknowledges uncertainty. + +Coverage and empirical distinctiveness. The five abilities span the retrieval–reasoning–update +space with minimal redundancy. IE and MSR cover retrieval at single-session and cross-session +granularity, respectively. TR and MSR address reasoning from complementary angles: temporal +inference over timestamps and session metadata versus aggregative computation over distributed facts. +KU and AR target opposite facets of knowledge update: KU penalizes failure to revise stored values, +while AR penalizes failure to abstain. The cross-type Spearman correlation analysis in §4.3 validates +this design empirically: no pair of abilities exhibits consistently strong positive correlation across +context lengths, confirming that each probes a distinct aspect of long-term multimodal memory. + +Code: https://github.com/xrenaf/MEMLENS (the repository contains pointers to the dataset +and per-image metadata). + + + + + 63 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/metadata.xml b/wizard-work/sphinx/archive-memlens-2026-10-22/metadata.xml new file mode 100644 index 0000000..15e0547 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/metadata.xml @@ -0,0 +1,64 @@ + + + https://arxiv.org/api/9bXDYvkY10G8yc5ebyvdMPgP1Lw + arXiv Query: search_query=&id_list=2605.14906v1&start=0&max_results=10 + 2026-10-22T22:18:57Z + + 10 + 1 + 0 + + http://arxiv.org/abs/2605.14906v1 + MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models + 2026-05-14T14:41:17Z + + + Memory is essential for large vision-language models (LVLMs) to handle long, multimodal interactions, with two method directions providing this capability: long-context LVLMs and memory-augmented agents. However, no existing benchmark conducts a systematic comparison of the two on questions that genuinely require multimodal evidence. To close this gap, we introduce MEMLENS, a comprehensive benchmark for memory in multimodal multi-session conversations, comprising 789 questions across five memory abilities (information extraction, multi-session reasoning, temporal reasoning, knowledge update, and answer refusal) at four standard context lengths (32K-256K tokens) under a cross-modal token-counting scheme. An image-ablation study confirms that solving MEMLENS requires visual evidence: removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images. Evaluating 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs achieve high short-context accuracy through direct visual grounding but degrade as conversations grow, whereas memory agents are length-stable but lose visual fidelity under storage-time compression. Multi-session reasoning caps most systems below 30%, and neither approach alone solves the task. These results motivate hybrid architectures that combine long-context attention with structured multimodal retrieval. Our code is available at https://github.com/xrenaf/MEMLENS. + + 2026-05-14T14:41:17Z + Work in progress + + + Xiyu Ren + + + Zhaowei Wang + + + Yiming Du + + + Zhongwei Xie + + + Chi Liu + + + Xinlin Yang + + + Haoyue Feng + + + Wenjun Pan + + + Tianshi Zheng + + + Baixuan Xu + + + Zhengnan Li + + + Yangqiu Song + + + Ginny Wong + + + Simon See + + + diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_01.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_01.txt new file mode 100644 index 0000000..2043119 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_01.txt @@ -0,0 +1,61 @@ + M EM L ENS: Benchmarking Multimodal Long-Term + Memory in Large Vision-Language Models + + + Xiyu Ren1 Zhaowei Wang1 Yiming Du2 Zhongwei Xie1 + Chi Liu1 Xinlin Yang1 Haoyue Feng1 Wenjun Pan1 Tianshi Zheng1 + Baixuan Xu1 Zhengnan Li3 Yangqiu Song1 Ginny Wong4 Simon See4 +arXiv:2605.14906v1 [cs.CV] 14 May 2026 + + + + + 1 + CSE Deparment, HKUST 2 CUHK + 3 + OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. + 4 + NVIDIA AI Technology Center (NVAITC), NVIDIA, Santa Clara, USA + {xrenaf, zwanggy, yqsong}@cse.ust.hk + ydu@se.cuhk.edu.hk lzhengnan389@gmail.com + + + + Abstract + + Memory is essential for large vision-language models (LVLMs) to handle long, + multimodal interactions, with two method directions providing this capability: long- + context LVLMs and memory-augmented agents. However, no existing benchmark + conducts a systematic comparison of the two on questions that genuinely require + multimodal evidence. To close this gap, we introduce M EM L ENS, a comprehensive + benchmark for memory in multimodal multi-session conversations, comprising + 789 questions across five memory abilities (information extraction, multi-session + reasoning, temporal reasoning, knowledge update, and answer refusal) at four + standard context lengths (32K–256K tokens) under a cross-modal token-counting + scheme. An image-ablation study confirms that solving M EM L ENS requires vi- + sual evidence: removing evidence images drops two frontier LVLMs below 2% + accuracy on the 80.4% of questions whose evidence includes images. Evaluating + 27 LVLMs and 7 memory-augmented agents, we find that long-context LVLMs + achieve high short-context accuracy through direct visual grounding but degrade + as conversations grow, whereas memory agents are length-stable but lose visual fi- + delity under storage-time compression. Multi-session reasoning caps most systems + below 30%, and neither approach alone solves the task. These results motivate hy- + brid architectures that combine long-context attention with structured multimodal + retrieval. Our code is available at https://github.com/xrenaf/MEMLENS. + + + + 1 Introduction + + Memory is essential for enabling large vision-language models (LVLMs) [1, 2] to maintain con- + sistency across extended multimodal interactions and to continuously incorporate new information + over time [3–5]. When LVLMs are deployed as agents, the inputs they handle accumulate incre- + mentally rather than arriving all at once. As interaction histories expand, the agent must recall past + content and reason over the growing context to remain consistent with prior turns. At the same time, + it must integrate new facts as they arrive and revise outdated information to keep its knowledge + current. Two directions have emerged to provide this capability, namely long-context LVLMs and + memory-augmented agents. Long-context LVLMs enlarge the native context window so that the + complete dialogue history, including interleaved images, can be processed directly by the model [5–7]. + Memory-augmented agents, building on retrieval-augmented generation, instead compress, index, + and selectively retrieve past content from an external store [8–10]. + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_02.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_02.txt new file mode 100644 index 0000000..10b5cb4 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_02.txt @@ -0,0 +1,73 @@ +Table 1: Left: Comparison of M EM L ENS with existing long-context and conversational memory +benchmarks. Type — Dial (Dialogue) or Doc (Document); Max L — maximum context length +reported (“—” if not applicable); L Control — standardized token length control; Comp. Eval. — +paper benchmarks both LVLMs and memory-augmented agents on the same data; Memory Tasks — +dot pattern showing coverage of the five evaluation tasks (• = supported, ◦ = not supported) in fixed +order: IE (Information Extraction), MSR (Multi-Session Reasoning), TR (Temporal Reasoning), +KU (Knowledge Update), AR (Answer Refusal). Right: Distribution of the 789 evaluation questions +in M EM L ENS across the five major task types (inner ring) and nine reporting subtypes (outer ring). + + Max Multi- Multi- L Comp. Memory +Benchmark Type Refuse Entity + L modal Sess. Control Eval. Tasks 11.4% 15.2% +MemoryBank [18] Dial — ✗ ✓ ✗ ✗ •◦◦◦◦ AR +LoCoMo [16] Dial 10K ✓ ✓ ✗ ✗ •••◦◦ Update 11.4% + 14.7% IE +MM-NIAH [12] Doc 128K ✓ ✗ ✓ ✗ •◦◦◦◦ KU 31.2% + 14.7% PrevInfo + 16.0% +MMLongBench-Doc [13] Doc 128K ✓ ✗ ✗ ✗ •◦◦◦◦ +MRAG-Bench [19] Doc — ✓ ✗ ✗ ✗ •◦◦◦◦ + Count. +LongMemEval [14] Dial 1.5M ✗ ✓ ✓ ✗ ••••• 5.8% MSR +MMLongBench [11] Doc 128K ✓ ✗ ✓ ✗ •◦◦◦◦ 18.1% TR + Ent. Res. 24.6% Temp. Gnd. +Multimodal NIAH [20] Doc 128K ✓ ✗ ✓ ✗ •◦◦◦◦ 6.0% + Arith. 13.1% +MemAgentBench [15] Dial 115K ✗ ✓ ✗ ✓ •◦◦◦◦ 6.3% Dur. Cmp. + 11.5% +M EM L ENS (Ours) Dial 256K ✓ ✓ ✓ ✓ ••••• + + + + +Despite progress along both directions, no current benchmark conducts a length-controlled compari- +son of long-context LVLMs and memory-augmented agents on questions that genuinely require visual +evidence. Long-context multimodal benchmarks measure context-length scaling on long documents +and retrieval tasks [11–13], but cover mainly LVLMs and do not place them alongside memory- +augmented agents for direct comparison. Text-only conversational memory benchmarks such as +LongMemEval [14] and MemoryAgentBench [15] overlook the visual modality entirely, treating +memory as a single-modality problem. Multimodal conversational benchmarks such as LoCoMo [16] +and Mem-Gallery [17] retain both visual and text modalities, yet most of their questions admit a +text-only shortcut, rendering the visual modality redundant. As Table 1 summarizes, no existing +benchmark requires visual evidence to answer its questions while supporting a length-controlled +comparison of long-context LVLMs and memory-augmented agents. +To bridge the gap, we introduce M EM L ENS, a comprehensive benchmark for assessing memory in +multimodal multi-session conversations. M EM L ENS consists of 789 questions covering five core +memory abilities: information extraction, multi-session reasoning, temporal reasoning, knowledge +update, and answer refusal. We design questions that demand cross-modal reasoning over both +visual and textual evidence, requiring the retrieval of multimodal information hidden within one +or more conversations between a user and an assistant. To verify that solving M EM L ENS requires +visual evidence, we conduct an image-ablation study on the 80.4% of questions whose evidence +includes images: when these images are removed, the accuracy of two frontier LVLMs collapses +below 2% (§3.4). Following the needle-in-a-haystack construction paradigm [21], we design a +four-stage pipeline that builds a coherent multi-session chat history for each question, distributing +the evidence across one or more user-assistant sessions alongside topically related distractor turns. +Because distractor turns can be added independently of the evidence, the chat length is freely +extensible. We release four standard context lengths (32K, 64K, 128K, and 256K tokens) under a +cross-modal token-counting scheme [11] that aligns text and vision tokens. M EM L ENS thus enables a +length-controlled comparison of long-context LVLMs and memory-augmented agents on multimodal +questions spanning five comprehensive memory abilities. +Using M EM L ENS, we evaluate 27 LVLMs and 7 memory-augmented agents across all four context +lengths (32K, 64K, 128K, and 256K). Our evaluation yields three key findings. First, the five +memory abilities are largely independent: strong information extraction does not predict multi- +session reasoning, and multi-session reasoning caps most evaluated systems below 30%. Second, +the two approaches exhibit complementary failure modes: long-context LVLMs deliver high short- +context accuracy through direct visual grounding, but this advantage shrinks as conversations grow; +memory agents, in contrast, are length-stable but lose visual fidelity under storage-time compression, +and memory-oriented post-training of agent backbones can additionally weaken their abstention +behavior. Third, neither approach comes close to solving the task of long-term memory. These + + + 2 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_03.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_03.txt new file mode 100644 index 0000000..91790b8 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_03.txt @@ -0,0 +1,59 @@ +results point to a clear next step: architectures that combine long-context attention with structured +multimodal retrieval, rather than scaling either component in isolation. + +2 Related Work +Memory-Augmented LLM Agents. Recent surveys systematize memory representations and oper- +ations across LLM-based agents [22]. Text-only memory agents span structured-fact stores [18, 23], +OS-inspired paging [8], tree-summarized retrieval [24], neurobiological graphs [25], relational em- +beddings [9], agentic self-organizing notes [26], intent-driven memory selection [27], RL-selected +time-aware memory [28], layered memory tiers [29], and sliding-window RL agents [30]. Mul- +timodal extensions add ColPali-style document retrieval [31, 32], multimodal embeddings [33], +dual-layer semantic memory [34], entity-centric audio-visual memory [35], sparse video mem- +ory [36], LoRA-tuned session retrieval [37], and intent-guided multimodal response retrieval over +multi-session conversations [38]. Recent work probes the intersection of long-context LLMs and +retrieval-augmented generation [10, 39, 40], suggesting retrieval and long attention are complemen- +tary rather than competing. + +Long-Context and Conversational Memory Benchmarks. Most current long-context benchmarks +established core protocols for retrieval and length scaling [41–47], but neglect the visual modality. +Multimodal extensions, such as MMLongBench [11], scale context length over documents [11, 13], +needle-style retrieval [12, 20], long-form video [48, 49], super-long documents [50, 51], and multi- +image inputs [52–55]. Yet their inputs are documents or videos rather than multi-session conver- +sations. Memory under conversational interaction therefore remains unexercised. Conversational +memory benchmarks restore the multi-session structure but rarely require visual perception: Long- +MemEval [14] (500 questions over five memory abilities, up to 1.5M tokens), PerLTQA [56], +and MemoryAgentBench [15] (retrieval, test-time learning, long-range understanding, selective +forgetting) discard images entirely, while LoCoMo [16] and Mem-Gallery [17] embed images in +persona-grounded dialogue but allow most questions to be answered from text alone, so neither +stress-tests multimodal memory. + +3 The M EM L ENS Benchmark +We propose M EM L ENS, a multimodal long-term conversational memory benchmark that comprises +789 questions instantiated at four standardized input lengths (32K/64K/128K/256K tokens). In +contrast to prior benchmarks restricted to text-only conversations and questions, M EM L ENS provides +multimodal conversation sessions in which text and images are interleaved, together with questions +that require cross-modal reasoning over evidence images and the surrounding textual context. §3.1 +defines the five memory abilities, §3.2 describes the construction pipeline (Figure 1), §3.3 presents +the quality control, and §3.4 describes the cross-modality validation. + +3.1 Memory Abilities + +To comprehensively reflect the retrieval, reasoning, and update that a conversational assistant must +perform over a long multimodal history, M EM L ENS formulates five core memory abilities. We break +each ability into subtypes that target specific reasoning operations. + + • Information Extraction (IE) tests the recall of a specific fact from a single evidence session, + with two subtypes. Entity questions form a two-hop chain: the model first identifies an + abstracted entity in the evidence image, then retrieves the associated information from the + surrounding text. PrevInfo (previous information) questions instead abstract the session + reference, asking the model to recall a visual detail from an image shared in an earlier + session. The visual identification hop draws on image-understanding skills such as fine- + grained recognition, object counting, spatial reasoning, and numerical reasoning. + • Multi-Session Reasoning (MSR) evaluates aggregation across three to eight sessions: count- + ing tallies unique items identified only by their evidence images, arithmetic sums values + stated in text or embedded in visual artifacts, and entity resolution determines co-reference + by comparing images across sessions. + + + 3 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_04.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_04.txt new file mode 100644 index 0000000..c218681 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_04.txt @@ -0,0 +1,97 @@ + Multimodal Session Simulation Conversation History Assembly + 1 Topic Ontology 2 Web Image 3 Two-agent 4 Multimodal Session + M M: Multimodal Session + Retrieval + Filtering Dialogue + E E: Evidence Session + VLM VLM F F: Filler Session (Text-only) + User Assistant ··· + ··· Assembled Multimodal Conversation History + ··· + 11/02 M ··· + ··· + Topic Pool + 12/02 F ··· + + ✓ image selected before dialogue → structural image-text alignment 。 E ··· + 。 F ··· + Question-Answer Pair Construction Evidence Session Construction 。 + 。 M ··· + 1 Background Generation 2 Visual Anchor Retrieval 3 Entity Ambiguity VLM / Agent + 。 E ··· + Sampled Topic Ontology replace name with generic placeholder 。 + Anchor Entity Golden BEFORE (named entity) AFTER (generic placeholder) + Gate Bridge : What time did I meet Emma F ··· + Emma and I Emma and I at the Golden Gate Bridge last + finally met finally met Saturday? 14/04 E ··· + crossed the crossed the + Golden Gate bridge : 6:42 AM + Bridge together together last Fillers (text-only sessions) + Emma and I finally met last Saturday ··· Saturday ··· Atomic Facts (evidence units) + crossed the Golden Gate + Bridge together last F1 F2 + My sister Emma My sister and I + Saturday, three years after + and I crossed the started crossing + From ShareGPT / UltraChat + we first planned to. We met bridge in + Golden Gate Bridge the bridge the bridge at 6:42 + on the the south side of the last Saturday AM that morning. + bridge at at 6:42 AM to morning. + catch the sunrise, and I + wore Dad’s red windbreaker + Cross with Saturday Cross with Saturday Fixed text/image ratio + from his 1998 photo. + Emma 6:42 AM Emma 6:42 AM indirectly mention the fact in conversation Context Budgets + Halfway across, near the + S1 S2 + second tower, Marcus + proposed to Emma. We text-only path (unsolvable) + celebrated afterward at Pier 32K 64K 128K 256K + Anchor Image + 39. + ✗ Requires image evidence + + + Figure 1: M EM L ENS construction pipeline. + + + • Temporal Reasoning (TR) assesses joint reasoning over temporal references, including both + natural-language expressions and session timestamps, together with visual content: duration + comparison compares two intervals derived from textual or visual cues, and temporal + grounding either orders events chronologically or extracts the specific date of an event. + Beyond entity abstraction, TR also replaces temporal expressions with visual artifacts such + as clock faces and calendar pages. + • Knowledge Update (KU) tests the ability to track an evolving user attribute across a chain of + four successive updates [57] (e.g., “I used to like apples” → “now I prefer kiwi”), requiring + the model to reason from the final state of the chain rather than earlier superseded ones. + • Answer Refusal (AR) removes all supporting evidence from an otherwise answerable ques- + tion, so it can no longer be answered from the remaining context; a correct model must + decline rather than hallucinate [58]. AR serves as a calibration check for hallucination + detection, not a core memory retrieval task. + +A single cross-modal principle unifies the four answerable types: the evidence image carries infor- +mation that the text deliberately withholds, so correct answers require joint visual–textual reasoning +(§3.2); §3.4 validates this dependency empirically. The formal problem specification appears in +Appendix C.1. Appendix I grounds this taxonomy in established memory research, and Appendix C.3 +provides the complete subtype breakdown with representative examples. + +3.2 Data Curation + +The construction proceeds in four components (Figure 1): (i) first, multimodal session simulation +generates topic-grounded multimodal dialogue sessions that form the conversation background; +(ii) next, question construction produces evaluation questions whose answers require visual content +through entity abstraction; (iii) evidence session construction wraps each evidence fact into a complete +session that matches its topical and stylistic profile; and (iv) conversation history assembly interleaves +evidence, haystack, and text-only filler sessions in timestamp order and compiles each question into +four standardized input lengths. + +Multimodal session construction. Each session begins with topic sampling from a hierarchical +ontology. For each topic, we generate an image query to retrieve a batch of candidate images from the +web1 . We then filter these candidates for visual relevance. From this filtered set, we select the final + 1 https://github.com/hellock/icrawler; sourcing, licensing, privacy, and release detailed in Appendix C.7. + + + + + 4 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_05.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_05.txt new file mode 100644 index 0000000..c2af2da --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_05.txt @@ -0,0 +1,62 @@ +image. Given this image, we generate sessions through a dual-model simulation with GPT-5.1 [2] +as the user and Gemini-3-Pro [59] as the assistant, producing multi-turn dialogues that incorporate +the selected images into interleaved image-text sequences [60]. Because image selection precedes +dialogue generation, image–text alignment is a structural property of the data rather than a post-hoc +filter. The image retrieval pipeline (§C.5), full topic ontology (§C.2), and image diversity statistics +(§C.6) are detailed in the Appendix. + +Question-Answer Pair. Question generation follows a four-step pipeline. (i) A topic is sampled +from the hierarchical ontology, and a LVLM generates a background paragraph containing salient +named-entities, for instance, a paragraph about San Francisco landmarks that mentions the Golden +Gate Bridge. (ii) One entity is selected as the visual anchor for the question. A text query derived +from the entity is issued to the same web-crawling and multi-model scoring pipeline used for haystack +images (§C.5), yielding a high-relevance image [61]; the anchor may range from a famous landmark +to a specific product model. (iii) Cross-modal dependency is enforced through entity abstraction [62– +64]: the entity in the background paragraph is replaced with a higher level concept drawn from +a dictionary spanning 55 entity categories (e.g., “Golden Gate Bridge” → “the bridge shown in +”). After replacement, the paragraph no longer names the entity, and only the evidence +image can resolve the reference. (iv) The abstracted paragraph, evidence image, and original entity +name are provided to Gemini-3-Pro [59], which generates a (question, answer) pair together with +atomic evidence facts. The generation prompt constrains the question to require both the image and +the surrounding text, closing the text-only shortcut documented in prior work [16]. The full pipeline +and per-subtype generation routes appear in Appendix C.4. + +Evidence session. Directly inserting evidence statements into the conversation history creates +abrupt semantic shifts that make the evidence trivially locatable by similarity-based retrieval. Prior +work has shown that increasing contextual similarity between evidence and distractors raises retrieval +difficulty, a design principle also adopted by LongMemEval [14]. Accordingly, each evidence fact +is wrapped in a complete evidence session generated using the same pipeline as haystack sessions +but grounded on the evidence fact rather than a sampled topic. Consequently, the evidence facts +match the topical and stylistic profile of the surrounding haystack, so that evidence cannot be located +by surface-level shortcuts. To further increase the difficulty, we also prompt the user model to +mention the facts indirectly without stressing them: To embed the fact “I started a new job last +month,” the user turn might open by asking about updating the tax withholding and mention the job +change incidentally later. When an evidence fact carries an evidence image, the image is placed +adjacent to the corresponding textual mention within the session, preserving unambiguous image-text +co-reference after entity abstraction. + +Conversation history assembly. For each question, evidence Table 2: Dataset statistics. +sessions are inserted into a timestamp-ordered history of haystack +sessions, with positions chosen uniformly at random (for KU, we Statistic Value +preserve the evidence-session order; Appendix C.8). Haystack Questions 789 +sessions are curated to be contextually related but uninformative Types / Subtypes 5/9 +for the question, and never include answer-relevant details. We Evidence sessions 2,145 +vary the number of haystack sessions to produce four standardized Avg. turns/session ∼10 +context lengths (32K/64K/128K/256K tokens) using the cross- Avg. images/session ∼1.5 +modal counting scheme of MMLongBench [11] (Table 2). To Tokens/image ∼2,000 +avoid revealing evidence positions via image clustering, we keep 32K → 256K +a fixed text-per-image ratio across the history, padding with text- Sessions/instance 14 → 93 +only filler sessions from ShareGPT and UltraChat [65]. A post- Images/instance 20 → 138 +hoc classifier achieves only marginally above-chance accuracy +when separating evidence from haystack text (Appendix G.8), and the generator scales beyond 256K. + +3.3 Quality Control + +Automated filtering. Each candidate question is screened by two automated checks. A rule-based +pre-filter removes images from the question and evidence and drops cases where the remaining text +already determines the answer. An LLM judge (GPT-5.1 [2]) then sees only the question text (without +evidence or conversation history) and removes items solvable from parametric knowledge alone [66]. + + + 5 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_06.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_06.txt new file mode 100644 index 0000000..e0f2378 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_06.txt @@ -0,0 +1,64 @@ +Table 3: Cross-modality ablation on the image-essential and image-supportive questions. With +evidence: question + evidence facts + evidence images, no haystack (n = 634). W/o evidence: +question + evidence facts, no evidence images. + Model Input Overall IE MSR TR KU ∆ + With evidence image 93.13 94.31 100.00 96.91 75.86 — + GPT-5.4 + W/o evidence image 1.74 0.41 0.00 5.15 0.00 −91.39 + With evidence image 89.42 89.02 90.21 96.19 82.24 — + Gemini-3.1-Pro + W/o evidence image 1.89 0.00 0.00 6.19 0.00 −87.53 + + + +Remaining questions therefore require using the evidence image, which is a prerequisite for the +multimodal analyses below. + +Human review. Three rounds of human review operate as a cumulative quality gate on the filtered +questions. Round 1 verifies that the evidence image carries answer-critical information; for AR +questions, which are constructed by removing the evidence facts, this round instead confirms that the +removed facts was answer-critical. Round 2 checks that evidence facts are naturally embedded and +recoverable from their sessions, and that each session reads as a plausible user–assistant exchange +with natural dialogue flow; sessions that fail the naturalness criterion are subsequently refined by +humans. Round 3 reviews the haystack sessions to assess both the quality of the images and the +naturalness of the dialogues. Together, the automated filters and three-round review reduce the initial +pool of 20k candidates to the final 789 questions. The filler sessions drawn from ShareGPT and +UltraChat (§3.2) further ground the surrounding context in authentic user–AI conversational patterns. +Annotation guidelines and inter-annotator agreement appear in Appendix E.1. + +3.4 Cross-modality Validation + +The construction pipeline targets image-necessary questions: across M EM L ENS, 65.7% of questions +are image-essential (the answer is unrecoverable without the evidence image), 14.7% are image- +supportive (the image confirms or disambiguates a textual fact), and 19.6% are text-sufficient (all AR +questions plus a subset of MSR items that test cross-session reasoning over purely textual evidence). +We verify this cross-modal dependency with two empirical checks (Table 3). An answerability test +supplies each image-essential and image-supportive question (n = 634) with its full evidence (textual +facts and evidence images) and confirms that the questions are answerable: GPT-5.4 reaches 93.13% +overall and Gemini-3.1-Pro 89.42%. A multimodal ablation then removes all evidence images: +overall accuracy collapses below 2% for both models. Two frontier proprietary LVLMs converge on +near-identical collapses, showing our questions are highly multimodal. + + +4 Evaluation and Analysis + +4.1 Experimental Setup + +We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS. The LVLMs span +closed-source systems including GPT-5.4 [2], Claude Sonnet 4.5 [7], and Gemini-3.1-Pro [67], +alongside major open-source families such as Kimi-K2.5 [68], Qwen3.5 [69], GLM-4.6V [70], +and Gemma3 [71]; the complete list is provided in Appendix B. The memory-augmented agents +comprise three multimodal pipelines (M3-Agent [35], M2A [34], and M3C [37]) and four text-only +pipelines (Mem0 [9], MemOS [29], MemAgent-7B [30], and Memory-T1 [28]), with backbone +and adapter details reported in Appendix G. Since the four text-only memory agents do not accept +visual inputs, we follow standard practice and replace the image inputs with captions generated by +BLIP-2 [72]; the remaining input adaptations are documented in Appendix B. For comparability, +the LVLMs are evaluated at three context lengths (32K, 64K, and 128K), as many of them do not +natively support 256K, whereas the agents are evaluated across all four context lengths (32K, 64K, +128K, and 256K). We report LLM-as-Judge accuracy [73] using Qwen3-VL-235B-A22B-Instruct +as the judge, cross-validated by GPT-5.4-mini re-judging (κ = 0.93) and a three-annotator human +consensus (Appendix E.2). + + + 6 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_07.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_07.txt new file mode 100644 index 0000000..d56f188 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_07.txt @@ -0,0 +1,74 @@ + IE MSR TR KU AR Overall + Claude Sonnet 4.5 28.0 26.4 19.5 13.3 9.1 2.8 40.2 35.1 33.0 29.3 21.6 16.4 97.8 94.4 93.3 36.5 32.4 27.8 + + Gemini-3.1-Pro 57.3 58.9 55.8 32.2 31.5 29.4 40.9 42.3 41.2 49.1 48.6 46.0 97.8 95.6 96.7 54.1 53.0 52.0 + + GPT-5.4 69.5 63.0 60.2 28.2 27.3 21.7 39.2 42.3 39.7 47.4 43.1 43.1 97.8 96.7 94.4 52.7 52.3 49.6 + + Kimi-K2.5 54.8 52.4 51.6 44.1 35.7 28.7 53.1 52.1 48.2 50.9 49.1 48.6 97.8 97.8 93.3 54.9 54.0 52.0 + +Qwen3.5-122B-A10B 74.4 67.1 43.1 30.1 27.3 22.4 51.5 51.5 41.8 49.1 48.6 46.0 88.9 84.4 83.3 58.7 55.9 45.5 + + Qwen3.5-27B 70.3 62.2 54.1 29.4 21.0 14.0 39.2 35.6 34.0 42.2 44.0 38.8 86.7 67.8 63.3 53.0 46.1 40.7 + +Nemotron-Nano-12B 63.8 56.1 — 16.8 21.0 — 44.8 42.8 — 39.7 34.5 — 45.6 35.6 — 45.0 40.9 — + +Cosmos-Reason2-8B 55.7 46.3 37.0 23.8 20.3 20.3 48.5 45.9 43.8 31.9 24.1 12.9 68.9 54.4 35.6 46.1 39.2 31.9 + + Phi4-Multimodal 31.7 28.0 22.8 20.3 21.0 22.4 43.8 39.2 30.9 19.8 24.1 14.7 43.3 21.1 15.6 32.2 28.1 22.7 + + GLM-4.6V 59.3 61.8 52.8 20.3 23.8 23.1 53.1 53.1 43.8 44.0 34.5 28.4 93.3 64.4 30.0 52.3 49.0 39.0 + + Gemma3-27B 35.0 26.4 19.1 23.8 22.4 18.2 41.2 44.3 37.1 30.2 27.6 13.8 68.9 61.1 48.9 37.6 34.2 26.0 + + Qwen3-VL-235B 51.6 38.6 29.3 25.2 18.2 15.4 49.5 42.8 43.3 36.2 35.3 30.2 88.9 75.6 67.8 48.3 39.7 34.7 + + Qwen3-VL-30B 41.5 39.0 33.7 11.9 11.9 10.5 35.6 37.1 36.6 30.2 29.3 20.7 71.1 65.6 60.0 36.4 35.2 31.3 + + Mem0 13.1 11.5 8.2 25.7 22.9 20.0 50.0 45.8 50.0 17.2 24.1 24.1 77.3 77.3 77.3 31.8 31.3 30.3 + + MemOS 18.0 18.0 16.4 22.9 22.9 22.9 39.6 39.6 43.8 24.1 24.1 17.2 68.2 68.2 81.8 30.3 30.8 31.3 + + MemAgent-7B 18.0 13.1 14.8 25.7 28.6 28.6 62.5 45.8 60.4 41.4 24.1 20.7 13.6 4.5 9.1 32.8 25.6 28.2 + + Memory-T1 18.0 19.7 21.3 25.7 22.9 22.9 62.5 56.2 56.2 13.8 17.2 27.6 9.1 18.2 13.6 28.7 28.7 30.3 + + M3-Agent 18.0 18.0 22.9 22.9 17.1 20.0 29.2 37.5 22.9 6.9 3.5 6.9 13.6 4.5 18.2 19.5 19.0 19.5 + + M2A 14.8 18.0 21.3 8.6 14.3 11.4 2.1 25.0 27.1 0.0 0.0 0.0 22.7 13.6 13.6 15.4 15.9 16.9 + + 32K 64K128K 32K 64K128K 32K 64K128K 32K 64K128K 32K 64K128K 32K 64K128K + +Figure 2: Per-type accuracy (%) by context length for representative 13 LVLMs and 6 memory- +augmented agents. Each panel shows one question type; cells use a green colormap. Missing +cells indicate models that exceed their usable context budget at 128K. LVLMs are evaluated on the +full 789-question benchmark, agents on the 195-question canonical subset. Full rosters appear in +Tables 13 and 14. + + +4.2 Main Results + +Overall performance. Figure 2 reports per-type accuracy across context lengths for a represen- +tative 13-LVLM cohort and six memory agents; the full 27-LVLM and seven-agent rosters are in +Appendix G.1. At 32K, the top eight LVLMs fall within a 6.34% band, so short-context accuracy +no longer separates frontier systems. The picture inverts at 128K: several open-weight leaders lose +more than 13%, while Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% +drop). AR shows the steepest context-driven decline in the open-weight LVLM family, suggesting +that hallucination control is the ability most exposed to evidence dilution at long contexts. Memory +agents occupy a narrower range, with the top four text-only systems clustering within 5% of each +other, while M3-Agent, M3C, and M2A fall substantially lower. Memory agents trail LVLMs across +nearly all types, with the largest gaps on visually grounded retrieval (IE, KU) and on answer refusal +(AR). + +Type-specific difficulty. Per-type accuracy ceilings range from 97.78% on AR down to 44.06% +on MSR (Kimi-K2.5) at 32K. AR, which serves as a calibration check for hallucination detection, +is the easiest type as expected, although its apparent ease erodes substantially at long contexts +for open-weight LVLMs (e.g., GLM-4.6V AR drops from 93.33% at 32K to 30.00% at 128K). +TR follows (60.82% ceiling), because timestamps and dates in session metadata provide explicit +retrieval anchors. IE (74.39% ceiling) and KU (50.86%) form an intermediate band: IE requires +visual grounding, with Entity questions demanding two-hop reasoning through the abstracted image +reference, while KU requires tracking a four-fact update chain in which missing a single image anchor + + + 7 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_08.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_08.txt new file mode 100644 index 0000000..ec602ea --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_08.txt @@ -0,0 +1,85 @@ + Visual Textual Output + Mixed Reasoning + IE TR AR 100 + 8% + 80 MSR KU 14% + + + + + % of wrong answers + 70 80 11% + 44% + 60 +Accuracy (%) + + + 60 + 50 73% + + 40 40 + 89% + 75% + 30 42% + + + 20 20 + + 10 10% 11% + 0 + 0 IE MSR TR KU + 32K 64K 128K Memory type + Context length + (b) Wrong-answer error decomposition at 128K con- +(a) Per-type accuracy vs. input length. Solid: LVLM text across the four answerable memory types, split +average (27 models, n=789); dashed: agent average by failure modality (visual, textual, mixed, reasoning, +(7 systems, n=195). Bands: 95% CI. output). Category definitions appear in Table 19. + + +flips the predicted state. MSR is the hardest, as cross-session aggregation over three to eight sessions +defeats every evaluated system: only Kimi-K2.5 (44.06%) and Gemini-3.1-Pro (32.17%) clear 30% +by margin, exposing this as the shared capability ceiling of M EM L ENS. Within types, IE-Entity is +consistently harder than single-hop IE-PrevInfo, and MSR-Arithmetic is the most difficult subtype +overall (Appendix C.3). + +No model dominates all memory abilities. No single model fam- IE MSR TR KU AR +ily dominates across all types (Figure 3). GLM-4.6V [70] leads GPT-5.4 69.5 28.2 39.2 47.4 97.8 +TR but collapses on KU, while Qwen3.5 [69] inverts the pattern. Gemini-3.1-Pro 57.3 32.2 40.9 49.1 97.8 +Kimi-K2.5 [68] is relatively strongest on MSR at 32K, though this + Kimi-K2.5 54.8 44.1 53.1 50.9 97.8 +advantage fades at longer contexts. Gemini-3.1-Pro is the only + Qwen3.5 (2) 72.4 29.7 45.4 45.7 87.8 +model simultaneously competitive on IE, KU, and MSR at 128K. + GLM-4.6V 59.3 20.3 53.1 44.0 93.3 +Memory agents exhibit an inverted profile: Memory-T1 reaches + Mem0 13.1 25.0 50.0 17.2 77.3 +high TR accuracy through BM25 date matching but falls well below +direct LVLMs on IE, substituting keyword retrieval for the visual MemAgent-7B 17.2 25.0 61.5 40.6 13.6 + +grounding that IE demands. Memory-T1 18.0 25.0 63.5 13.8 9.1 + + M3-Agent 19.0 22.2 30.6 6.2 13.6 + +4.3 Analysis + Figure 3: Memory-ability spe- +Current memory pipelines lose faithfulness to original visual cialization across representative + LVLMs and memory agents. +evidence. The gap between agents and LVLMs is largest on the +visually grounded types (IE and KU). Despite their different input formats (§4.1), both text-only +and multimodal pipelines compress evidence visual information into a fixed memory representation +at storage time, leaving the original image pixels inaccessible at query time. Retrieval over these +compressed encodings is also less reliable than retrieval over the raw conversation text. The loss +is sharpest on IE and KU because captions and embeddings retain only the gist of an image and +discard fine-grained visual cues, such as counts, attributes, and spatial relations, that these two +types specifically probe. The same gap appears when blip-captions replace raw images for text-only +agents, indicating that the bottleneck lies in lossy cross-modality storage rather than in the answering +model. Closing this gap requires memory architectures that preserve image-level evidence rather than +caption-based compression. + +Post-training on memory agent backbones weakens abstention. Memory agents fall far below +their direct-inference counterparts on AR. We evaluate two agents that keep the backbone frozen, +Mem0 (77.27%) and MemOS (68.18%), and five that further finetune the backbone via RL or LoRA + + + 8 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_09.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_09.txt new file mode 100644 index 0000000..afa573d --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_09.txt @@ -0,0 +1,79 @@ +for memory management (M2A, M3-Agent, M3C, MemAgent-7B, and Memory-T1). While the +frozen-backbone agents preserve much of the abstention behavior of the underlying LLM, the others +collapse to 9–22% AR. M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score +81.82% under direct inference, and the backbone ablation in Appendix G.6 confirms that stronger +backbones alone do not close this gap. This suggests that the reward design of current RL/SFT +fine-tuning on memory agent backbones optimizes mainly answer correctness and retrieval success, +providing no signal that refusing an unanswerable question is correct, so abstention degrades after +training. Future agent designs should therefore optimize memory access, answer accuracy, and +evidence-sensitive abstention jointly [74], rather than treating memory management as independent +of hallucination control. + +LVLMs and memory agents show complementary scaling trade-offs. Figure 4a plots accuracy +across five abilities from 32K to 128K for both LVLMs and memory agents. The two approaches +respond to context scaling in structurally different ways. LVLMs degrade substantially on retrieval- +heavy types, with IE and KU losing ∼20% and ∼12% respectively as evidence images become harder +to locate among growing filler content. MSR shows apparent flatness that reflects a floor effect near +30% rather than genuine robustness. AR shows the steepest LVLM decline (∼75% at 32K to ∼45% +at 128K), suggesting that growing filler content erodes abstention and pushes LVLMs to hallucinate +on unanswerable questions rather than refuse them. Six of seven memory agents, by contrast, stay +within ±7% from 32K to 256K because their retrieve-then-reason pipeline is length-invariant by +construction. The two failure modes are orthogonal: LVLMs lose to context length, while memory +agents lose to lossy multimodal compression at storage time. Because each architecture covers only +one axis, scaling along that axis leaves the other failure mode untreated, motivating hybrid designs +that span both axes. + +Memory ability correlations reveal distinct sources of difficulty. +We analyze pairwise Spearman correlations among the five question IE 1.00 -0.19 0.12 0.65 0.36 0.23 + 0.31 + + + +types at 32K (Figure 5). Correlations vary substantially across type MSR -0.19 1.00 0.33 0.21 -0.11 0.060.21 + +pairs, indicating that the types do not measure a single capability. IE +and KU form the most consistent retrieval-oriented pair: they have TR 0.12 0.33 1.00 0.29 -0.00 0.18 + 0.13 + + + +the strongest correlation at 32K and remain significantly correlated KU 0.65 0.21 0.29 1.00 0.59 0.43 + 0.19 + + +across context lengths, reflecting their shared need to first locate +the relevant evidence image. KU also correlates with AR at 32K, AR 0.36 -0.11-0.00 0.59 1.00 0.21 + 0.28 + + + +suggesting that some KU questions still depend on accurate evidence IE MSR TR KU AR Avgstd + +selection. In contrast, MSR shows weak correlations with IE and Figure 5: Spearman rank correla- +AR at all lengths, implying that its main challenge is aggregating tion (ρ) at 32K across all 34 evalu- +information across multiple evidence pieces rather than retrieving ated LVLMs and memory agents. +a single image. These findings reveal two complementary difficulty +axes: an evidence-retrieval axis (IE, KU) and an aggregation axis (MSR). They also illustrate how +a single aggregate score can obscure differences among long-context abilities, motivating per-type +evaluation alongside overall performance [43, 44]. Full correlation matrices at 64K and 128K appear +in Appendix G.5. + +Error analysis. Figure 4b decomposes wrong answers at 128K across the four answerable types, +with the category definitions provided in Appendix G.4. On IE and KU, nearly 90% of errors fall in +the Visual category, showing that the model fails to locate or read the evidence image; once the image +is reached, the answer is usually extracted correctly. TR errors split between Mixed and Reasoning, +reflecting both image-supportive grounding and small closed-set selection. MSR is the only type +whose errors are dominated by the Reasoning category (73%). We further conduct an oracle-retrieval +diagnostic in Appendix G.7, which restores accuracy on MSR to 90–100% when only the required +sessions are supplied. This suggests a reasoning error at long context is largely induced by upstream +retrieval failure, since the model computes correctly over the wrong subset of evidence. LVLMs +and memory agents fail differently. LVLMs lose visual evidence as filler content accumulates in the +context window, whereas memory agents lose it through lossy cross-modal compression at storage +time. As memory grows, near-misses at 32K shift toward total misses at 128K (Appendix G.4) [75], +suggesting that growth weakens evidence retrieval rather than reasoning. Future memory designs +should therefore focus more on cross-modal evidence-retrieval fidelity instead of solely reasoning +improvements. + + + 9 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_10.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_10.txt new file mode 100644 index 0000000..05387d9 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_10.txt @@ -0,0 +1,56 @@ +5 Conclusion +We introduced M EM L ENS, the first benchmark for multimodal conversational memory to evaluate +LVLMs and memory-augmented agents under a unified, length-controlled protocol (32K–256K +tokens; five memory abilities). The task remains challenging: even the strongest LVLM reaches +only 58.68% at 32K. LVLMs degrade sharply once input contexts grow, while memory agents stay +length-stable but discard fine-grained visual evidence at storage time, and post-training on memory +agent backbones can erode the agents’ abstention behavior. The five abilities show low cross-type +correlation, confirming that per-type evaluation is necessary, and nearly 90% of information-extraction +errors stem from visual perception rather than comprehension, indicating that scaling first harms +evidence retrieval, not reasoning. Visual-evidence retention and retrieval, rather than raw scaling of +either context or memory, therefore emerges as the principal bottleneck to address in the future. + +Acknowledgments and Disclosure of Funding +The authors of this paper were supported by the National Key Research and Development Program of +China (2025YFE0200500), the ITSP Platform Research Project (ITS/189/23FP) from ITC of Hong +Kong, SAR, China, and the AoE (AoE/E-601/24-N), the RIF (R6021-20) and the GRF (16205322) +from RGC of Hong Kong, SAR, China. We also thank the NVIDIA AI Technology Center (NVAITC) +and OmniMemory (Shenzhen) Intelligent Technology Co., Ltd. for their support and additional +funding. + +Ethics Statement +M EM L ENS is designed exclusively for evaluating multimodal long-term conversational memory +in large vision-language models (LVLMs) and memory-augmented agents under a unified, length- +controlled protocol. The benchmark is not intended as a training dataset for developing memory- +equipped systems but rather as a controlled diagnostic for fair comparison across architectures. We +release every artifact under frozen version tags so that any reported leaderboard cell remains traceable +to the exact item set on which it was scored. +The 4,695 source images underlying M EM L ENS are retrieved from public web image search via +iCrawler against a non-person-centric topic ontology; candidates carrying watermarks, stock-photo +logos, or copyright overlays are excluded at retrieval time, and the construction prompts never request +the model to identify, name, or describe any depicted person. Author-produced artifacts (questions, +evidence facts, prompts, and human-annotation records) are released under CC-BY-4.0 and the +evaluation harness under MIT, while third-party images retain their original source-site licenses; +per-image provenance metadata (source URL, retrieval timestamp, perceptual hash) accompanies the +release, and a takedown contact in the project repository allows seven-day removal of any flagged +image. +Human review is conducted by project members rather than crowd-workers, so no third-party data +collection or human-subjects protocol is involved. Three sequential rounds (Appendix E.1) audit +cross-modal necessity at the question level, naturalness and recoverability at the session level, and a +stratified sample of haystack sessions before any item reaches the released set. +We caution that M EM L ENS is not intended as a training dataset: exposure of the 789 evaluation items +as supervised data would compromise their diagnostic value, and the datasheet explicitly discourages +such use. While the benchmark is grounded in multi-session multimodal conversation, the five +memory abilities it isolates (information extraction, multi-session reasoning, temporal reasoning, +knowledge update, and answer refusal) describe general functional requirements of any long-horizon +conversational assistant. We therefore expect M EM L ENS to serve as a useful indicator of memory +robustness in adjacent settings, such as document-grounded or voice-grounded assistants, where +retrieval, integration, and calibrated abstention play comparable roles. + +Reproducibility Statement +All closed-source LVLM evaluations and synthetic data construction were performed via the public +APIs of OpenAI, Anthropic, and Google, with a total experimental cost of approximately 4,500 USD + + + 10 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_11.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_11.txt new file mode 100644 index 0000000..68fddea --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_11.txt @@ -0,0 +1,58 @@ +covering both pipeline construction and benchmarking. Open-weight LVLMs are served locally +with vLLM v0.17–0.18 on 8×A100-80GB nodes with tensor parallelism for context windows at +and above 128K. Per-model decoding budgets, judge protocol, retrieval depth for memory agents, +and adapter details for the four text-only pipelines (Mem0, MemOS, MemAgent-7B, Memory- +T1) are documented in Appendix B. The four-stage construction pipeline (multimodal session +simulation, question construction, evidence session construction, conversation history assembly) +is described in Section 3, with per-subtype generation routes in Appendix C.4 and full prompt +templates in Appendix F. Quality control combines automated filtering with a three-round human +review (Section 3.3, Appendix E.1) and an LLM-as-Judge protocol cross-validated by an independent +judge family and a three-annotator human consensus; agreement statistics and per-question-type +breakdowns appear in Appendix E.2. The 789-question benchmark, 4,695 source images with +provenance metadata, evaluation harness, and prompt templates are publicly released under frozen +version tags at https://huggingface.co/datasets/xiyuRenBill/MEMLENS (dataset) and +https://github.com/xrenaf/MEMLENS (code) so that every reported leaderboard cell can be +reproduced end-to-end. + +References + [1] Bytedance Seed. Seed2. 0 model card: Towards intelligence frontier for real-world complex- + ity. Technical report, Technical report, Bytedance, 2025. URL https://lf3-static. bytednsdoc. + com . . . . + [2] OpenAI. Openai gpt-5 system card, 2025. URL https://arxiv.org/abs/2601.03267. + [3] OpenAI. Gpt-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. + [4] Anthropic. Claude 3 model card. https://assets.anthropic.com/m/61e7d27f8c8f5 + 919/original/Claude-3-Model-Card.pdf, 2024. Accessed: 2026-04-30. + [5] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett + Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal + understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024. + [6] Bytedance Seed. Seed1. 8 model card: Towards generalized real-world agency. arXiv preprint + arXiv:2603.20633, 2026. + [7] Anthropic. Claude Sonnet 4.5 System Card. https://assets.anthropic.com/m/12f21 + 4efcc2f457a/original/Claude-Sonnet-4-5-System-Card.pdf, 2025. + [8] Charles Packer, Sarah Wooders, Kevin Lin, Vivian Fang, Shishir G. Patil, Ion Stoica, and + Joseph E. Gonzalez. Memgpt: Towards llms as operating systems, 2024. URL https: + //arxiv.org/abs/2310.08560. + [9] Deshraj Yadav, Taranjeet Singh, and Prashant Srivastava. Mem0: Building production-ready + AI agents with scalable long-term memory, 2025. URL https://arxiv.org/abs/2504.1 + 9413. + [10] Bowen Jin, Jinsung Yoon, Jiawei Han, and Sercan O Arik. Long-context llms meet rag: + Overcoming challenges for long inputs in rag. arXiv preprint arXiv:2410.05983, 2024. + [11] Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, + Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, and Mark Steedman. MMLong- + Bench: Benchmarking long-context vision-language models effectively and thoroughly, 2025. + URL https://arxiv.org/abs/2505.10610. + [12] Weiyun Wang, Shuibo Zhang, Yiming Ren, Yuchen Duan, Tiantong Li, Shuo Liu, Mengkang + Hu, Zhe Chen, Kaipeng Zhang, Lewei Lu, Xizhou Zhu, Ping Luo, Yu Qiao, Jifeng Dai, + Wenqi Shao, and Wenhai Wang. Needle in a multimodal haystack, 2024. URL https: + //arxiv.org/abs/2406.07230. + [13] Yubo Ma, Yuhang Zang, Liangyu Chen, Meiqi Chen, Yizhu Jiao, Xinze Li, Xinyuan Lu, Ziyu + Liu, Yan Ma, Xiaoyi Dong, Pan Zhang, Liangming Pan, Yu-Gang Jiang, Jiaqi Wang, Yixin + Cao, and Aixin Sun. Mmlongbench-doc: Benchmarking long-context document understanding + with visualizations, 2024. URL https://arxiv.org/abs/2407.01523. + [14] Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. Long- + memeval: Benchmarking chat assistants on long-term interactive memory, 2025. URL + https://arxiv.org/abs/2410.10813. + + + 11 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_12.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_12.txt new file mode 100644 index 0000000..e08671f --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_12.txt @@ -0,0 +1,57 @@ +[15] Yuanzhe Hu, Yu Wang, and Julian McAuley. Evaluating memory in LLM agents via incremen- + tal multi-turn interactions, 2026. URL https://arxiv.org/abs/2507.05257. +[16] Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and + Yuwei Fang. Evaluating very long-term conversational memory of llm agents, 2024. URL + https://arxiv.org/abs/2402.17753. +[17] Yuanchen Bei, Tianxin Wei, Xuying Ning, Yanjun Zhao, Zhining Liu, Xiao Lin, Yada Zhu, + Hendrik Hamann, Jingrui He, and Hanghang Tong. Mem-Gallery: Benchmarking multimodal + long-term conversational memory for MLLM agents, 2026. URL https://arxiv.org/ab + s/2601.03515. +[18] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing + large language models with long-term memory, 2023. URL https://arxiv.org/abs/23 + 05.10250. +[19] Wenbo Hu, Jia-Chen Gu, Zi-Yi Dou, Mohsen Fayyaz, Pan Lu, Kai-Wei Chang, and Nanyun + Peng. MRAG-Bench: Vision-centric evaluation for retrieval-augmented multimodal models, + 2025. URL https://arxiv.org/abs/2410.08182. +[20] Hengyi Wang, Haizhou Shi, Shiwei Tan, Weiyi Qin, Wenyuan Wang, Tunyu Zhang, Akshay + Nambi, Tanuja Ganu, and Hao Wang. Multimodal needle in a haystack: Benchmarking + long-context capability of multimodal large language models, 2025. URL https://arxiv. + org/abs/2406.11230. +[21] Greg Kamradt. Needle in a haystack — pressure testing LLMs. https://github.com/gka + mradt/LLMTest_NeedleInAHaystack, 2023. GitHub repository. +[22] Yiming Du, Wenyu Huang, Danna Zheng, Zhaowei Wang, Sebastien Montella, Mirella Lapata, + Kam-Fai Wong, and Jeff Z. Pan. Rethinking memory in llm based agents: Representations, + operations, and emerging topics, 2025. URL https://arxiv.org/abs/2505.00675. +[23] Bing Wang, Xinnian Liang, Jian Yang, Hui Huang, Shuangzhi Wu, Peihao Wu, Lu Lu, Zejun + Ma, and Zhoujun Li. SCM: Enhancing large language model with self-controlled memory + framework, 2025. URL https://arxiv.org/abs/2304.13343. +[24] Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. + Manning. RAPTOR: Recursive abstractive processing for tree-organized retrieval, 2024. +[25] Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: + Neurobiologically inspired long-term memory for large language models, 2025. URL https: + //arxiv.org/abs/2405.14831. +[26] Wujiang Xu, Zujie Liang, Kai Mei, Hang Gao, Juntao Tan, and Yongfeng Zhang. A-MEM: + Agentic memory for LLM agents, 2025. URL https://arxiv.org/abs/2502.12110. +[27] Yiming Du, Bingbing Wang, Yang He, Bin Liang, Baojun Wang, Zhongyang Li, Lin Gui, + Jeff Z. Pan, Ruifeng Xu, and Kam-Fai Wong. Memguide: Intent-driven memory selection for + goal-oriented multi-session llm agents, 2025. URL https://arxiv.org/abs/2505.20231. +[28] Yiming Du, Baojun Wang, Yifan Xiang, Zhaowei Wang, Wenyu Huang, Boyang Xue, Bin + Liang, Xingshan Zeng, Fei Mi, Haoli Bai, Lifeng Shang, Jeff Z. Pan, Yuxin Jiang, and Kam-Fai + Wong. Memory-t1: Reinforcement learning for temporal reasoning in multi-session agents, + 2025. URL https://arxiv.org/abs/2512.20092. +[29] Zhiyu Li, Shichao Song, Hanyu Wang, Simin Niu, Ding Chen, Jiawei Yang, Chenyang Xi, + Huayi Lai, Jihao Zhao, Yezhaohui Wang, Junpeng Ren, Zehao Lin, Jiahao Huo, Tianyi Chen, + Kai Chen, Kehang Li, Zhiqiang Yin, Qingchen Yu, Bo Tang, Hongkang Yang, Zhi-Qin John + Xu, and Feiyu Xiong. Memos: An operating system for memory-augmented generation (mag) + in large language models, 2025. URL https://arxiv.org/abs/2505.22101. +[30] Hongli Yu, Tinghong Chen, Jiangtao Feng, Jiangjie Chen, Weinan Dai, Qiying Yu, Ya-Qin + Zhang, Wei-Ying Ma, Jingjing Liu, Mingxuan Wang, and Hao Zhou. Memagent: Reshaping + long-context llm with multi-conv rl-based memory agent, 2025. URL https://arxiv.org/ + abs/2507.02259. +[31] Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. M3docrag: Multi- + modal retrieval is what you need for multi-page multi-document understanding, 2024. URL + https://arxiv.org/abs/2411.04952. + + + 12 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_13.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_13.txt new file mode 100644 index 0000000..57a47fc --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_13.txt @@ -0,0 +1,58 @@ +[32] Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and + Pierre Colombo. Colpali: Efficient document retrieval with vision language models, 2025. + URL https://arxiv.org/abs/2407.01449. +[33] Rui Meng, Ziyan Jiang, Ye Liu, Mingyi Su, Xinyi Yang, Yuepeng Fu, Can Qin, Zeyuan Chen, + Ran Xu, Caiming Xiong, Yingbo Zhou, Wenhu Chen, and Semih Yavuz. VLM2Vec-V2: + Advancing multimodal embedding for videos, images, and visual documents, 2025. URL + https://arxiv.org/abs/2507.04590. +[34] Junyu Feng, Binxiao Xu, Jiayi Chen, Mengyu Dai, Cenyang Wu, Haodong Li, Bohan Zeng, + Yunliu Xie, Hao Liang, Ming Lu, and Wentao Zhang. M2A: Multimodal memory agent + with dual-layer hybrid memory for long-term personalized interactions, 2026. URL https: + //arxiv.org/abs/2602.07624. +[35] Lin Long, Yichen He, Wentao Ye, Yiyuan Pan, Yuan Lin, Hang Li, Junbo Zhao, and Wei Li. + Seeing, listening, remembering, and reasoning: A multimodal agent with long-term memory, + 2025. URL https://arxiv.org/abs/2508.09736. +[36] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, + Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. Moviechat: From dense token to + sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on + Computer Vision and Pattern Recognition, pages 18221–18232, 2024. +[37] Jihyoung Jang, Minwook Bae, Minji Kim, Dilek Hakkani-Tur, and Hyounghun Kim. Enabling + chatbots with eyes and ears: An immersive multimodal conversation system for dynamic + interactions, 2025. URL https://arxiv.org/abs/2506.00421. +[38] Bingbing Wang, Yiming Du, Bin Liang, Zhixin Bai, Min Yang, Baojun Wang, Kam-Fai + Wong, and Ruifeng Xu. A new formula for sticker retrieval: Reply with stickers in multi- + modal and multi-session conversation. Proceedings of the AAAI Conference on Artificial + Intelligence, 39(24):25327–25335, Apr. 2025. doi: 10.1609/aaai.v39i24.34720. URL + https://ojs.aaai.org/index.php/AAAI/article/view/34720. +[39] Ziyan Jiang, Xueguang Ma, and Wenhu Chen. Longrag: Enhancing retrieval-augmented + generation with long-context llms. arXiv preprint arXiv:2406.15319, 2024. +[40] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: + Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International + Conference on Learning Representations, 2023. +[41] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao + Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for + long context understanding. In Proceedings of the 62nd Annual Meeting of the Association for + Computational Linguistics (Volume 1: Long Papers), pages 3119–3137, 2024. +[42] Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng + Xu, Lei Hou, Yuxiao Dong, et al. Longbench v2: Towards deeper understanding and reasoning + on realistic long-context multitasks. arXiv preprint arXiv:2412.15204, 2024. +[43] Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, + and Boris Ginsburg. Ruler: What’s the real context size of your long-context language models? + In First Conference on Language Modeling, 2024. +[44] Howard Yen, Tianyu Gao, Minmin Hou, Ke Ding, Daniel Fleischer, Peter Izsak, Moshe + Wasserblat, and Danqi Chen. Helmet: How to evaluate long-context language models effec- + tively and thoroughly. arXiv preprint arXiv:2410.02694, 2024. +[45] Chenxin An, Shansan Gong, Ming Zhong, Xingjian Zhao, Mukai Li, Jun Zhang, Lingpeng + Kong, and Xipeng Qiu. L-eval: Instituting standardized evaluation for long context language + models. In Proceedings of the 62nd Annual Meeting of the Association for Computational + Linguistics (Volume 1: Long Papers), pages 14388–14411, 2024. +[46] Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, + Zhen Thai, Shuo Wang, Zhiyuan Liu, et al. ∞Bench: Extending long context evaluation + beyond 100k tokens. In Proceedings of the 62nd Annual Meeting of the Association for + Computational Linguistics (Volume 1: Long Papers), pages 15262–15277, 2024. +[47] Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. Needlebench: Can llms do retrieval and + reasoning in 1 million context window? arXiv preprint arXiv:2407.11963, 2024. + + + 13 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/page_14.txt b/wizard-work/sphinx/archive-memlens-2026-10-22/page_14.txt new file mode 100644 index 0000000..801784e --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/page_14.txt @@ -0,0 +1,58 @@ +[48] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Zhengyang Liang, Shitao Xiao, Minghao Qin, + Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: Benchmarking + multi-task long video understanding, 2025. URL https://arxiv.org/abs/2406.04264. +[49] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for + long-context interleaved video-language understanding, 2024. URL https://arxiv.org/ + abs/2407.15754. +[50] Chao Deng, Jiale Yuan, Pi Bu, Peijie Wang, Zhong-Zhi Li, Jian Xu, Xiao-Hui Li, Yuan Gao, Jun + Song, Bo Zheng, et al. Longdocurl: a comprehensive multimodal long document benchmark + integrating understanding, reasoning, and locating. arXiv preprint arXiv:2412.18424, 2024. +[51] Yew Ken Chia, Liying Cheng, Hou Pong Chan, CHAOQUN LIU, Maojia Song, Mahani + Aljunied, Soujanya Poria, and Lidong Bing. M-longdoc: A benchmark for multimodal + super-long document understanding and a retrieval-aware tuning framework. 2024. +[52] Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman + Pan, Yangqiu Song, and Dong Yu. Divscene: Towards open-vocabulary object navigation with + large vision language models in diverse scenes. arXiv preprint arXiv:2410.02730, 2024. +[53] Dingjie Song, Shunian Chen, Guiming Hardy Chen, Fei Yu, Xiang Wan, and Benyou Wang. + Milebench: Benchmarking mllms in long context. arXiv preprint arXiv:2404.18532, 2024. +[54] Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: + Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture. arXiv preprint + arXiv:2409.02889, 2024. +[55] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and + Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large + language models. CoRR, 2024. +[56] Yiming Du, Hongru Wang, Zhengyi Zhao, Bin Liang, Baojun Wang, Wanjun Zhong, Zezhong + Wang, and Kam-Fai Wong. Perltqa: A personal long-term memory dataset for memory + classification, retrieval, and synthesis in question answering, 2024. URL https://arxiv.or + g/abs/2402.16288. +[57] Rongwu Xu, Zehan Qi, Zhijiang Guo, Cunxiang Wang, Hongru Wang, Yue Zhang, and Wei + Xu. Knowledge conflicts for LLMs: A survey. In Proceedings of the 2024 Conference on + Empirical Methods in Natural Language Processing, pages 8541–8565, 2024. +[58] Hanning Zhang, Shizhe Diao, Yong Lin, Yi R. Fung, Qing Lian, Xingyao Wang, Yangyi Chen, + Heng Ji, and Tong Zhang. R-tuning: Instructing large language models to say “I don’t know”. + In Proceedings of the 2024 Conference of the North American Chapter of the Association for + Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages + 7113–7139, 2024. +[59] Google DeepMind. Gemini 3 Pro Model Card. https://deepmind.google/models/mod + el-cards/gemini-3-pro/, 2026. +[60] Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton + Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: + An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural + Information Processing Systems, 36:71683–71702, 2023. +[61] Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina + Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing + millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on + Computer Vision, pages 12065–12075, 2023. +[62] Zhaowei Wang, Haochen Shi, Weiqi Wang, Tianqing Fang, Hongming Zhang, Sehyun Choi, + Xin Liu, and Yangqiu Song. Abspyramid: Benchmarking the abstraction ability of language + models with a unified entailment graph. In Findings of the Association for Computational + Linguistics: NAACL 2024, pages 3991–4010, 2024. +[63] Zhaowei Wang, Wei Fan, Qing Zong, Hongming Zhang, Sehyun Choi, Tianqing Fang, Xin + Liu, Yangqiu Song, Ginny Wong, and Simon See. Absinstruct: Eliciting abstraction ability + from llms through explanation tuning with plausibility estimation. In Proceedings of the 62nd + Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), + pages 973–994, 2024. + + + 14 + \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/quotes.json b/wizard-work/sphinx/archive-memlens-2026-10-22/quotes.json new file mode 100644 index 0000000..af7427e --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/quotes.json @@ -0,0 +1,114 @@ +[ + { + "q": "removing evidence images drops two frontier LVLMs below 2% accuracy on the 80.4% of questions whose evidence includes images", + "ok": true, + "pages": [ + 1 + ] + }, + { + "q": "the five memory abilities are largely independent: strong information extraction does not predict multi-session reasoning, and multi-session reasoning caps most evaluated systems below 30%", + "ok": true, + "pages": [ + 2 + ] + }, + { + "q": "long-context LVLMs deliver high short-context accuracy through direct visual grounding, but this advantage shrinks as conversations grow", + "ok": true, + "pages": [ + 2 + ] + }, + { + "q": "neither approach comes close to solving the task of long-term memory", + "ok": true, + "pages": [ + 2 + ] + }, + { + "q": "789 questions instantiated at four standardized input lengths", + "ok": true, + "pages": [ + 3 + ] + }, + { + "q": "Evidence sessions", + "ok": true, + "pages": [ + 5 + ] + }, + { + "q": "We evaluate 27 LVLMs and seven memory-augmented agents on M EM L ENS", + "ok": true, + "pages": [ + 6 + ] + }, + { + "q": "65.7% of questions are image-essential", + "ok": true, + "pages": [ + 6 + ] + }, + { + "q": "GPT-5.4 reaches 93.13% overall and Gemini-3.1-Pro 89.42%", + "ok": true, + "pages": [ + 6 + ] + }, + { + "q": "At 32K, the top eight LVLMs fall within a 6.34% band", + "ok": true, + "pages": [ + 7 + ] + }, + { + "q": "Gemini-3.1-Pro retains 51.99% accuracy and degrades least overall (a 2.11% drop)", + "ok": true, + "pages": [ + 7 + ] + }, + { + "q": "Per-type accuracy ceilings range from 97.78% on AR down to 44.06% on MSR (Kimi-K2.5) at 32K", + "ok": true, + "pages": [ + 7 + ] + }, + { + "q": "M2A reaches only 22.73% on the same Qwen3-VL-8B backbone that score 81.82% under direct inference", + "ok": true, + "pages": [ + 9 + ] + }, + { + "q": "nearly 90% of information-extraction errors stem from visual perception rather than comprehension", + "ok": true, + "pages": [ + 10 + ] + }, + { + "q": "even the strongest LVLM reaches only 58.68% at 32K", + "ok": true, + "pages": [ + 10 + ] + }, + { + "q": "The benchmark is not intended as a training dataset for developing memory-equipped systems", + "ok": true, + "pages": [ + 10 + ] + } +] \ No newline at end of file diff --git a/wizard-work/sphinx/archive-memlens-2026-10-22/verify_memlens_archive.py b/wizard-work/sphinx/archive-memlens-2026-10-22/verify_memlens_archive.py new file mode 100644 index 0000000..39a5552 --- /dev/null +++ b/wizard-work/sphinx/archive-memlens-2026-10-22/verify_memlens_archive.py @@ -0,0 +1,157 @@ +#!/usr/bin/env python3 +"""Verify the MemLens archive: every quotation must be verbatim in the archived paper. + +Exit code 0 = archive is sound (all quotations real, hashes intact, metadata matches). +Exit code nonzero = something in this archive is wrong or fabricated. +""" +import hashlib +import json +import os +import re +import sys +import xml.etree.ElementTree as ET + +HERE = os.path.dirname(os.path.abspath(__file__)) +EXPECTED_ARXIV_ID = "2605.14906v1" +EXPECTED_TITLE = "MemLens: Benchmarking Multimodal Long-Term Memory in Large Vision-Language Models" +REQUIRED_SECTIONS = [ + "## 1. Identity", + "## 2. Provenance", + "## 3. One-line verdict", + "## 4. What the paper contributes", + "## 5. Cross-modality validation", + "## 6. Key findings", + "## 7. Verified quotations", + "## 8. Caveats and limits", + "## 9. Verification", +] +N_PAGES = 14 + +fails = [] +checks = 0 + + +def ok(msg): + global checks + checks += 1 + print(" PASS %s" % msg) + + +def bad(msg): + global checks + checks += 1 + fails.append(msg) + print(" FAIL %s" % msg) + + +def norm(s): + s = s.replace("\u00ad", "") + s = re.sub(r"-\s*\n\s*", "-", s) # undo hyphenated line breaks + s = re.sub(r"\s+", " ", s) + return s.strip() + + +def read(name): + with open(os.path.join(HERE, name), encoding="utf-8") as fh: + return fh.read() + + +print("== 1. Quotations are verbatim in the archived paper ==") +quotes = json.loads(read("quotes.json")) +if len(quotes) != 16: + bad("expected 16 quotations in quotes.json, found %d" % len(quotes)) +else: + ok("quotes.json holds 16 quotations") + +pages = {n: norm(read("page_%02d.txt" % n)) for n in range(1, N_PAGES + 1)} +full_text = norm(read("memlens.txt")) + +for entry in quotes: + q = entry["q"] + nq = norm(q) + label = q[:58].replace("\n", " ") + if nq not in full_text: + bad("QUOTE NOT IN PAPER TEXT: %s" % label) + continue + on_pages = [n for n, t in sorted(pages.items()) if nq in t] + if not on_pages: + bad("QUOTE NOT FOUND ON ANY PAGE: %s" % label) + continue + claimed = entry.get("pages") or [] + if claimed and claimed != on_pages: + bad("PAGE MISMATCH for %s: claimed %s, actual %s" % (label, claimed, on_pages)) + continue + ok("p%s %s" % (on_pages, label)) + +print("== 2. SHA256SUMS matches archived files ==") +for line in read("SHA256SUMS").splitlines(): + line = line.strip() + if not line: + continue + digest, name = line.split(None, 1) + name = name.strip() + path = os.path.join(HERE, name) + if not os.path.exists(path): + bad("missing archived file %s" % name) + continue + actual = hashlib.sha256(open(path, "rb").read()).hexdigest() + if actual == digest: + ok("%s %s" % (digest[:16], name)) + else: + bad("HASH MISMATCH %s: recorded %s, actual %s" % (name, digest[:16], actual[:16])) + +print("== 3. arXiv metadata ==") +root = ET.parse(os.path.join(HERE, "metadata.xml")).getroot() +ns = {"a": "http://www.w3.org/2005/Atom"} +entry = root.find("a:entry", ns) +meta_id = entry.find("a:id", ns).text.strip().split("/abs/")[-1] +meta_title = " ".join(entry.find("a:title", ns).text.split()) +if meta_id == EXPECTED_ARXIV_ID: + ok("arXiv id %s" % meta_id) +else: + bad("arXiv id mismatch: %s" % meta_id) +if meta_title == EXPECTED_TITLE: + ok("title matches") +else: + bad("title mismatch: %r" % meta_title) +n_authors = len(entry.findall("a:author", ns)) +if n_authors == 14: + ok("14 authors") +else: + bad("author count %d, expected 14" % n_authors) + +print("== 4. Research note structure ==") +import glob +notes = sorted(glob.glob(os.path.join(HERE, "REPORT-*.md"))) +if not notes: + notes = [os.path.join(HERE, "README.md")] +note_path = notes[0] +readme = read(os.path.relpath(note_path, HERE)) +print(" note file: %s" % os.path.basename(note_path)) +for sec in REQUIRED_SECTIONS: + if sec in readme: + ok("section present: %s" % sec) + else: + bad("section MISSING: %s" % sec) +if "16/16" in readme: + ok("note records the 16/16 quotation count") +else: + bad("note does not record the 16/16 count") + +print("== 5. Source PDF is intact ==") +pdf = os.path.join(HERE, "memlens.pdf") +head = open(pdf, "rb").read(8) +if head.startswith(b"%PDF-"): + ok("memlens.pdf has a valid PDF header (%d bytes)" % os.path.getsize(pdf)) +else: + bad("memlens.pdf is not a PDF (header %r)" % head) + +print() +print("checks run: %d failures: %d" % (checks, len(fails))) +if fails: + print("VERDICT: FAIL") + for f in fails: + print(" - %s" % f) + sys.exit(1) +print("VERDICT: PASS — all quotations verbatim, hashes and metadata verified") +sys.exit(0)