Q8 pilot: summary fidelity, 10-entry sample

recoveries/summary-fidelity-pilot.md

@@ -0,0 +1,73 @@
+# Q8 pilot — summary fidelity, 10-entry sample
+
+Compiled 2026-09-25 by zephyrus-sparkborn. Q8 asks whether each entry's machine-written summary
+overstates its article. This pilot defines the sample as **every 10th magazine entry** (`001, 011,
+021, 031, 041, 051, 061, 071, 081, 091`) — non-cherry-picked — and grades every factual or
+interpretive clause of the archive's **"What happened"** and **"In plain English"** prose against the
+live article. The "From the sources" blockquotes were excluded: quote-fidelity was already verified at
+269/269 in `recoveries/quote-fidelity-live.md`.
+
+**Method.** For each entry: `browser.open` the entry page
+(`https://alexanderwhitestone.com/stackchain/magazine-0NN/`), copy the two summary sections verbatim,
+`browser.open` the live article URL from `recoveries/magazine-article-map.md`, split the summary into
+clauses, and grade each: **holds** (article says it) / **overstated** (article supports less) /
+**unsupported** (article doesn't say it) / **contradicted** (public sources say otherwise) /
+**unverified** (could not fetch). Every grade carries the summary clause and the settling article
+sentence verbatim. All 20 pages fetched cleanly on 2026-09-25; raw article texts kept locally, full
+per-clause tables with verbatim quotes in the two halves' working files.
+
+## Totals — 10 entries, 98 clauses
+
+| grade | count |
+|---|---|
+| holds | 91 |
+| overstated | 4 |
+| unsupported | 2 |
+| contradicted | 1 |
+| unverified | 0 |
+
+Per entry (holds / overstated / unsupported / contradicted / unverified):
+`001`: 10/0/0/0/0 · `011`: 11/1/1/0/0 · `021`: 10/0/0/1/0 · `031`: 8/0/0/0/0 · `041`: 9/1/0/0/0 ·
+`051`: 11/0/0/0/0 · `061`: 10/1/1/0/0 · `071`: 8/0/0/0/0 · `081`: 9/0/0/0/0 · `091`: 6/1/0/0/0.
+
+## Entries that need an archive-side wording fix (7 clauses)
+
+1. **magazine-011** — summary says "merch for Mass Adoption and the Boston Bitcoin Festival".
+   **Overstated.** Article: "I'm also working with Mass Adoption, the Boston Bitcoin group. I make all
+   their merch and I'll be printing live June 24th at The Boston Bitcoin Festival…" — merch for Mass
+   Adoption, printing live *at* the festival, not merch *for* the festival.
+2. **magazine-011** — IPE calls John "founder of the Bitcoin-only printing company Proof of Ink".
+   **Unsupported.** The article never uses the word "founder"; byline is "Proof of Ink by -John". John's
+   "it's 100% all me" establishes sole operation, but "founder" is the archive's inference.
+3. **magazine-021** — "(only Loko and Happy responded)". **Contradicted.** Article: "Then after a few
+   minutes I checked the chat. A few positive responses. Some funny shit talking, oh look, Happy even
+   added some lyrics!" — more than two people responded; "only" is wrong.
+4. **magazine-041** — IPE: "Stackchain's rowdiest fork war". **Overstated.** Article: "This may go down
+   as the most fun battle of Stackchain just because of the amount of people involved." — "most fun"
+   is supported; "rowdiest" is the archive's superlative, not stated.
+5. **magazine-061** — "why he keeps a weekly Thursday intelligence-style cadence of work".
+   **Unsupported.** No sentence in the article text mentions a Thursday, a weekly cadence, or any
+   "intelligence-style" routine. Looks like a summarizer hallucination (unless it comes from an image,
+   which was not checked).
+6. **magazine-061** — "which the magazine note ties to a centerfold in issue 5 (Round 5)".
+   **Overstated (minor).** The "(see centerfold)" reference is in the interview answer itself; the
+   magazine note covers "his epic contribution to the mag in issue 5 (Round 5)". Attribution is wrong
+   though the fact is supported.
+7. **magazine-091** — "Comparing Strategy, Twenty One, Nakamoto, Metaplanet, Coinsilium and Smarter Web
+   returns and drawdowns against Bitcoin". **Overstated.** Article: "We set aside pioneers like
+   Strategy, which started in 2020 and is up over 400% since, as it's not a fair comp for these
+   newcomers." Strategy was deliberately excluded from the comparison; the summary lists it as if it
+   were one of the analyzed firms.
+
+None of the 7 breaks the entry's core thesis; the archive's summaries are high-fidelity with small
+summarizer-generated embellishments (added superlatives, title inferences, one possible hallucination).
+
+## Not archive-side decisions
+
+Per standing convention, wording changes to live archive pages are not made from the pipe; the 7
+clauses above are flagged for the archive editor with the exact proposed corrections implied by the
+article sentences quoted. The full per-clause evidence tables are in this file's two working halves.
+
+Material: `recoveries/magazine-article-map.md` (entry → article URLs, all 200). Sample entries
+`/magazine-001,011,021,031,041,051,061,071,081,091/`. Full-fidelity sweep of the remaining 81 entries
+proposed as next work (Q7 in today's check-in).