Gutenberg Long-Context Corpus — Archive Report

Artifact: archive-gutenberg/ (raw HTML + normalised text + manifest) Built: 2026-10-22 UTC · Lane: Archive — multimodal, long-context research Corpus size: 6 documents, 1,129,516 words, 6,813,315 bytes of normalised text

What this is

A fetched, hashed, structurally-verified archive of six public-domain long-form books from Project Gutenberg, normalised to plain UTF-8 text with per-document SHA-256 provenance. Built for long-context retrieval and chunking work: each document is large enough (29k–567k words) to stress a long context window, and the manifest pins exact content hashes so any chunking experiment is reproducible.

Documents

slug	title	words	txt bytes	txt SHA-256 (first 16)
war_and_peace	War and Peace	566,794	3,476,843	6cb9dd25004552dd
moby_dick	Moby-Dick; or, The Whale	216,325	1,312,754	9f7e9224bfb6baa8
pride_and_prejudice	Pride and Prejudice	130,933	771,500	7c8f3b9cd10c2324
sherlock_holmes	The Adventures of Sherlock Holmes	107,654	619,475	d00a2b51c3a403ed
frankenstein	Frankenstein; or, The Modern Prometheus	78,137	453,921	0d113dcfc3a1203f
alice_in_wonderland	Alice's Adventures in Wonderland	29,673	178,232	29d04d721acfd87d

Full hashes, source URLs, byte counts and fetch timestamps are in manifest.json.

Method

build_corpus.py fetches https://www.gutenberg.org/cache/epub/{id}/pg{id}-images.html for each document with a browser User-Agent.
HTML is normalised: <script>/<style> stripped, block tags converted to newlines, entities decoded, runs of whitespace collapsed. Raw HTML is kept alongside the normalised text so the transform is auditable.
Every file is hashed before the manifest is written, so the manifest cannot describe bytes that were never written.

Verification (measured)

python3 verify_archive.py . → PASS: 6 documents, 1129516 words, all hashes verified, exit 0. It recomputes SHA-256 from the bytes on disk (not from the manifest), checks byte counts and word counts, confirms each book's title appears in its own archived text, and enforces floors of 6 documents / 1,000,000 words so a truncated or stubbed corpus cannot pass.
Negative control: appending one word to txt/alice_in_wonderland.txt makes the verifier exit 1 with txt hash mismatch. The check discriminates real from broken rather than always passing.

Files

archive-gutenberg/
├── build_corpus.py     # fetches + normalises + hashes + writes manifest
├── verify_archive.py   # independent verifier (recomputes hashes from disk)
├── manifest.json       # schema sphinx-archive/1, 6 pinned documents
├── raw/                # 6 .html files, byte-for-byte as fetched
└── txt/                # 6 .txt files, normalised UTF-8

Verdict

The archive is built from real network fetches, every document is content-hashed, and a measured verifier confirms integrity from disk while demonstrably failing on tampered input.

3.1 KiB Raw Blame History Unescape Escape