archive-transformer-pdf-modality-2026-10-23
A PDF-container modality audit of a multimodal research artifact: arXiv:1706.03762v7
(Attention Is All You Need, 15 pages, 2,215,244 bytes, sha256
bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697).
The archive lane's question here is not "what does the paper say" but "what does the PDF container actually carry, and how much of the paper's meaning is lost when you extract only one modality". Every number below is measured from the byte stream, not inferred from the paper's prose.
The three findings
1. Raster extraction is not figure extraction. pdfimages -list finds exactly 6
raster objects — 3 image + 3 smask (soft-mask alpha channels) — and all 6 sit on
pages 3 and 4. But the paper carries 5 figure captions, on pages 3, 4, 13, 14 and
15. Pages 13, 14 and 15 hold captions for figures that are drawn as vector paths, so
they have no raster object at all. A raster-only figure extractor silently returns
3 of the 5 figures and loses the attention-visualisation figures entirely.
2. One sentence renders three different ways. The abstract's headline sentence, "Our model achieves 28.4 BLEU on the WMT 2014 English-to-German translation task", survives three different byte strings:
| rendering | what the bytes contain |
|---|---|
pdftotext (default) |
EnglishtoGerman — the line-break hyphen is dropped |
pdftotext -layout |
English- at end of one line, to-German at the start of the next |
| the HTML rendering (archived source) | English-to-German — the hyphen is kept |
A needle written against any one of these fails silently against the other two. This is the concrete mechanism behind every "quote not found" failure in a multimodal archive.
3. The paper contradicts itself on its own headline number. Section 6.1 prose says
"our big model achieves a BLEU score of 41.0," for EN-FR; the abstract says
"state-of-the-art BLEU score of 41.8 after"; and Table 2 on the same page (page 8)
prints 41.8 in the Transformer (big) row, which precedes the 41.8 cell. So the
abstract and the table agree at 41.8 while the prose says 41.0. Both values are
recorded verbatim in index.json, with the page each was found on.
What is in the box
source/1706.03762v7.pdf 2,215,244 B sha256 bdfaa68d…df697
text/page-01..15.txt 15 × pdftotext (default) per page
text-layout/page-01..15.txt 15 × pdftotext -layout per page
figures/obj-00..05-pageNN.png 6 × pdfimages -png extraction (sha256 each)
rasters/p01-01,p04-04,p15-15.png 3 × pdftoppm -r 72 page rasters (sha256 each)
cross-rendering/
html-rendering-source.md the third rendering, archived, sha256 8e0403fc…3161
html-rendering-slice.txt the recorded line + its source digest
index.json the record: digests, counts, measured offsets
build_archive.py regenerates the record from the bytes
verify_archive.py recomputes every fact from the committed bytes
mutation_test.py 13 planted corruptions, each must be caught
run_verdict.py live re-fetch + the whole offline battery
CHECK.py the verdict command (offline, stable)
index.json records 19 verbatim claims, each located at a measured character
offset, line number, per-page occurrence count and cross-page occurrence list —
plus the container metadata (pdfinfo), the raw pdfimages -list and pdffonts
output (byte-for-byte), a page-raster determinism probe (page 1 rendered twice,
digests identical), and the modality map above.
Reproduce
python3 CHECK.py # offline verdict — exit 0 = the archive holds
python3 verify_archive.py # 54 checks, 0 failures
python3 mutation_test.py # 13/13 corruptions caught
python3 run_verdict.py # the above, plus a live arXiv re-fetch
What this archive is for
It is a negative-capability artifact: it demonstrates, with bytes rather than assertions, that (a) raster extraction under-counts figures when figures are vector, (b) a single extraction setting changes the text layer, and (c) a long-context paper can state two different values for one metric inside a few pages of each other. Any multimodal archive that pins one rendering and one figure count is recording its own losses without noticing.