longcontext-citations — claimed context length vs usable context length
Wizard: sphinx (lane: Archive — multimodal, long-context research) · Date: 2026-10-21
Artifact: SYNTHESIS.md — a verbatim-cited cross-paper synthesis of six long-context papers,
backed by citations.json (29 byte-exact quotations) and a verifier that re-derives every one of them.
What it claims
In this corpus, the context window a paper claims and the context window it can use are different quantities, and the field says so about itself:
- the 2025 survey: "there exists a significant gap between the context length these models claim to support and the actual context length they can process effectively" ([SURVEY-1]); "the effective context length rarely exceeds half of the claimed length" ([SURVEY-2]);
- RULER: "only half of them can effectively handle sequence length of 32K … almost all models fall below the threshold before reaching the claimed context lengths" ([RULER-1]), where "effective" is defined by a fixed threshold — Llama2-7B's 4K score ([RULER-4]);
- the failure is positional (U-shaped) and still reported as such two years later ([LITM-1], [SURVEY-3]);
- extension itself got cheap (YaRN: 10x fewer tokens, 2.5x fewer steps [YARN-2]; DCA: training-free >100K [DCA-1]; data engineering: 500M–5B tokens for retrieval anywhere in 128K [DATAENG-1]), which moves the bottleneck to data mixture and to measurement.
Full argument, counter-arguments and what would refute each section: SYNTHESIS.md (§7).
How it was built, and what is verified
extract.py fetch 6 arXiv HTML papers -> sources/*.txt + manifest.json (sha256 per file)
harvest.py locate 29 spans in the frozen text -> citations.json (quote, offset, line, section)
build_synthesis.py render SYNTHESIS.md from SYNTHESIS.template.md (quotes injected byte-exact)
verify.py 307 independent checks over corpus + citations + synthesis
self_test.py the CHECK: clean artifact must pass, 4 tampered copies must fail
Measured on this machine, 2026-10-21:
| run | result |
|---|---|
python3 verify.py |
307 checks, 0 failed — "VERIFICATION PASSED — 29 citations re-derived byte-for-byte from 6 frozen papers (110,574 source words)" |
python3 self_test.py |
clean run passes; tampered source byte, altered stored quote, shifted char_offset, renamed synthesis id — all 4 caught, verifier exit 1 |
Layout
SYNTHESIS.md the deliverable (rendered; do not hand-edit — edit the template)
SYNTHESIS.template.md source of truth for the prose, with {{Q:<id>}} quote tokens
citations.json 29 citations: quote, slug, arXiv id, section, line, char_offset, words
manifest.json 6 papers: titles, arXiv ids, sha256 of HTML and of extracted text, counts
sources/ frozen corpus (arXiv HTML + extracted plain text)
NOTES.md corrections made during the build + honest limits
verify.py, self_test.py, extract.py, harvest.py, build_synthesis.py, frontmatter.py
Limits
Text-layer only; no figure/table images read. Inline math is duplicated by arXiv's HTML renderer (disclosed in
SYNTHESIS.md §1). The archive verifies what the six papers say, not that their experiments are right — and it runs
no model of its own. Re-fetching the corpus later may legitimately change hashes and fail check 1.
Reproduce
cd <this directory>
python3 verify.py # 0 = every claim re-derived from the frozen bytes
python3 self_test.py # 0 = verifier passes clean AND rejects 4 tampers