11 KiB
Long-Context Multimodal Understanding — 134-Paper Archive and Reading
Source: live arXiv Atom API, 5 queries, fetched 2026-10-26. Every number below is
recomputed by verify.py from papers.jsonl; every quotation is checked as a
verbatim substring of the cited abstract. Claim ids in brackets map to
claims.jsonl.
0. Provenance
Five API requests (raw/q1..q5.xml, sha256 recorded in raw_manifest.json)
returned 150 entries; de-duplication on the arXiv base id leaves 134 unique
papers [C01] spanning 2025-11-28 [C03] to 2026-10-23 [C04]. Their abstracts,
concatenated in CORPUS.txt, total 228,277 characters [C02] — the pack is long
enough that answering questions over it requires retrieval or chunking, not a
single skim. 15 papers were returned by more than one query [C21], so the five
queries are not disjoint silos; they overlap on a shared core of
"long-context multimodal" work.
The slice is recent and steeply skewed: 33 papers from September 2026 [C19] and 38 from October 2026 [C20]. This is a snapshot of an active frontier, not a balanced historical sample, and should be read that way.
1. What the corpus is made of
The field is still filed under computer vision: 96 of 134 papers (71.6%) list
cs.CV as primary category [C05]. Video is the dominant modality, named in 71
abstracts [C06]; documents are second, named in 30 [C11]. Hyphenated
"long-context" appears in 42 abstracts [C10], which is the tell that the
vocabulary has consolidated: authors no longer explain the problem, they name it.
Two design axes dominate the vocabulary: retrieval (45 abstracts) [C08] and memory (44) [C07]. Agents appear in 21 [C12], RAG explicitly in 14 [C14]. The striking gap is serving-level engineering: only 7 abstracts discuss the KV cache [C13]. Most of the field attacks the long-context problem by deciding which evidence to keep, not by making attention cheaper — and the one paper that does attack the cache directly is a frontier model report, not an academic method paper.
Benchmarks are a minority artifact: 32 abstracts mention a benchmark [C09], and 27 of those never mention a dataset, i.e. they evaluate on existing benchmarks rather than shipping new ones. Measurement infrastructure is being consumed faster than it is produced.
2. Finding 1 — the bottleneck is stated as an asymptotic cost problem
The clearest statement of the field's shared premise comes from FramePack-U, which describes the default recipe and why it fails:
"the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length" — 2610.16118 [C22]
Its critique of the previous generation of fixes is precise:
"Existing approaches, such as token reduction and keyframe selection, shorten the token sequence but leave the O(n) scaling intact" — 2610.16118 [C23]
That distinction — constant factor versus asymptote — is the useful lens for this whole archive. Work that only shortens the token sequence buys budget; work that fixes the budget converts an unbounded problem into a bounded one. 11 abstracts pair "token" with "compression" [C18], the largest coherent cluster inside the budget-cutting branch, and most of them sit on the constant-factor side of that line. Only 4 abstracts use the exact phrase "frame selection" [C15], yet keyframe selection is the default baseline being argued against, which is why the phrase under-counts the thread.
The serving-side answer is equally blunt about magnitude: DeepSeek-V4.1-Flash reports it can
"reduce its global KV cache footprint (always in HBM) to 890 bytes per token" — 2609.19969 [C34]
with a million-token context window. Read together, [C22]/[C23] and [C34] mark the two ends of the same argument: the algorithmic literature is trying to bound O(n) by choosing less input, while the model literature is trying to make O(n) affordable per unit. Neither has won; both are being published in the same month.
3. Finding 2 — memory has split into write-time and read-time camps
Memory work is the densest theme: 28 abstracts mention both video and memory [C17], and 15 mention memory together with retrieval [C16] — the write-then-read pattern. Within that, the archive shows a genuine architectural fork.
Write-time camp: compress the stream into a store, then answer cheaply. KeyRec is the cleanest statement of the split:
"During query-agnostic writing, KeyRec preserves fine-grained recent observations in a visual cache and organizes historical evidence into a structured event bank." — 2609.32182 [C25]
MIT^3 pushes the same idea to its limit, moving the store into the weights:
"writes each long video into compact, video-specific adaptive parameters for video-free, text-only question answering" — 2610.26910 [C27]
That is a different contract with the user: the video is consumed once, and questions are answered with no video in context at all.
Read-time camp: keep a compressed context and retrieve per query. ReMem states the failure mode it is built to fix:
"most existing works tend to progressively lose long context information as length of input stream increases" — 2610.05940 [C24]
The empirical pressure on both camps is that proactivity — deciding what to remember before being asked — is not solved:
"even the strongest models fail to capture memory both timely and accurately" — 2610.30089 [C26]
ProactEgoMem's framing matters more than its leaderboard: it evaluates whether an assistant initiates memory maintenance, which neither camp's architecture addresses. Memory that is written query-agnostically still needs a policy for what to write.
4. Finding 3 — evidence localisation, not context length, is the failure mode
If the corpus has one consensus result, it is that giving a model more document does not give it the right evidence. VELD-QA measures this directly on 15,000 human-authored QA pairs over 5,859 PDFs:
"Retrieval-based methods substantially underperform the oracle setting with ground-truth supporting pages" — 2610.20964 [C28]
and, critically, the naive fix does not work either:
"full-document access does not close the gap" — 2610.20964 [C29]
That second quote is the load-bearing one. It kills the assumption that a large enough context window substitutes for retrieval. DocScope reaches the same conclusion from the evaluation side, arguing that
"answer accuracy cannot substitute for trajectory-level evaluation" — 2605.08888 [C30]
and locating the weakness precisely:
"region grounding remains the weakest trajectory stage" — 2605.08888 [C31]
The survey in the archive frames the same shift historically: priorities are moving
"from multimodal content understanding to sufficient, high-fidelity evidence acquisition and sustained evidence management" — 2610.31385 [C36]
Three independent artifacts — a QA benchmark, a reasoning-trajectory benchmark, and a survey — converge on evidence handling rather than perception as the bottleneck. That convergence is the strongest signal in this archive.
5. Finding 4 — the evaluators are not yet trustworthy at day scale
The archive's most uncomfortable results are about measurement itself. At 100-hour scale, video-language judges are not reliable enough to grade the systems they are used to train:
"frontier judges reach only 75.4% pairwise accuracy" — 2609.34314 [C32]
and the failure is not a constant offset that could be calibrated away:
"judge accuracy degrades as the playlist set grows" — 2609.34314 [C33]
The task itself is also harder than headline numbers suggest once audio is admitted as evidence rather than decoration:
"even strong proprietary models achieve below 60% accuracy" — 2609.17248 [C35]
FrameScout's framing of the input distribution explains why these numbers look bad: the task is queries over
"videos spanning tens of minutes to hours" — 2610.23975 [C37]
Put [C32], [C33], [C35] next to [C09]: most papers in this slice are graded on benchmarks whose day-scale validity is itself an open research question. Any leaderboard claim over hour-long video should be read as provisional.
6. Limits of this archive
Stated plainly, because the artifact is only worth what its caveats are:
- Abstract-level evidence. Claims here describe what papers say, verified verbatim, not what their experiments show. No full texts were read, so effect sizes and baselines are out of scope.
- Five queries, one source. arXiv only; no OpenReview, no journal-only venues, no industry blogs. A paper that does not use the words "long context", "long-context", "long video", "long document", or "video-language" plus "long" is invisible here.
- Recency skew. 71 of 134 papers are from the last two months of the window ([C19] + [C20]); this is a frontier snapshot, and citation counts are meaningless at this age.
- Term counts are lexical. "Memory" in an abstract may mean KV cache, a parameter store, or a retrieval index — [C07]/[C16] measure vocabulary, not mechanism. The qualitative sections exist to disambiguate what the counts cannot.
- No PDFs, no tables, no figures. The multimodal content of the papers is not archived — only their text abstracts. This is an archive about multimodal long-context work, not a multimodal archive.
7. Anchor papers
| id | date | what it contributes |
|---|---|---|
| 2610.31385 | 2026-10-22 | task-centric survey of long-document VQA; evidence-centric taxonomy |
| 2610.30089 | 2026-10-21 | ProactEgoMem: benchmark for proactive memory in streaming egocentric video |
| 2610.26910 | 2026-10-17 | MIT^3: per-video test-time training, video-free QA |
| 2610.23975 | 2026-10-13 | FrameScout: streaming keyframe selector with contrastive objective |
| 2610.20964 | 2026-10-14 | VELD-QA: 15k QA pairs, 5,859 PDFs, evidence-grounding annotations |
| 2610.16118 | 2026-10-12 | FramePack-U: constant-memory frame packing |
| 2610.05940 | 2026-10-05 | ReMem: streaming context memory + retrieved vision memory |
| 2609.34314 | 2026-09-28 | PlaylistEval: day-scale judge benchmark, 630 pairs |
| 2609.32182 | 2026-09-26 | KeyRec: bounded visual memory, 10% token budget |
| 2609.19969 | 2026-09-17 | DeepSeek-V4.1-Flash: 890 B/token KV cache, 1M context |
| 2609.17248 | 2026-09-15 | Video-HolmesV2: audio-visual evidence reasoning benchmark |
| 2605.08888 | 2026-05-09 | DocScope: trajectory-level verifiable reasoning benchmark |
8. Reproducing this
python3 verify.py # full audit, exit 0 == archive is consistent
python3 verify.py --quiet # failures only
verify.py re-hashes the five raw Atom responses, re-parses them and compares
every title/abstract/date against papers.jsonl, recomputes all 37 claims from
the archive, and rejects any claim id or arXiv id in this report that does not
resolve. A failure anywhere returns nonzero.