diff --git a/recoveries/summary-fidelity-pilot.md b/recoveries/summary-fidelity-pilot.md new file mode 100644 index 0000000..3b82857 --- /dev/null +++ b/recoveries/summary-fidelity-pilot.md @@ -0,0 +1,73 @@ +# Q8 pilot — summary fidelity, 10-entry sample + +Compiled 2026-09-25 by zephyrus-sparkborn. Q8 asks whether each entry's machine-written summary +overstates its article. This pilot defines the sample as **every 10th magazine entry** (`001, 011, +021, 031, 041, 051, 061, 071, 081, 091`) — non-cherry-picked — and grades every factual or +interpretive clause of the archive's **"What happened"** and **"In plain English"** prose against the +live article. The "From the sources" blockquotes were excluded: quote-fidelity was already verified at +269/269 in `recoveries/quote-fidelity-live.md`. + +**Method.** For each entry: `browser.open` the entry page +(`https://alexanderwhitestone.com/stackchain/magazine-0NN/`), copy the two summary sections verbatim, +`browser.open` the live article URL from `recoveries/magazine-article-map.md`, split the summary into +clauses, and grade each: **holds** (article says it) / **overstated** (article supports less) / +**unsupported** (article doesn't say it) / **contradicted** (public sources say otherwise) / +**unverified** (could not fetch). Every grade carries the summary clause and the settling article +sentence verbatim. All 20 pages fetched cleanly on 2026-09-25; raw article texts kept locally, full +per-clause tables with verbatim quotes in the two halves' working files. + +## Totals — 10 entries, 98 clauses + +| grade | count | +|---|---| +| holds | 91 | +| overstated | 4 | +| unsupported | 2 | +| contradicted | 1 | +| unverified | 0 | + +Per entry (holds / overstated / unsupported / contradicted / unverified): +`001`: 10/0/0/0/0 · `011`: 11/1/1/0/0 · `021`: 10/0/0/1/0 · `031`: 8/0/0/0/0 · `041`: 9/1/0/0/0 · +`051`: 11/0/0/0/0 · `061`: 10/1/1/0/0 · `071`: 8/0/0/0/0 · `081`: 9/0/0/0/0 · `091`: 6/1/0/0/0. + +## Entries that need an archive-side wording fix (7 clauses) + +1. **magazine-011** — summary says "merch for Mass Adoption and the Boston Bitcoin Festival". + **Overstated.** Article: "I'm also working with Mass Adoption, the Boston Bitcoin group. I make all + their merch and I'll be printing live June 24th at The Boston Bitcoin Festival…" — merch for Mass + Adoption, printing live *at* the festival, not merch *for* the festival. +2. **magazine-011** — IPE calls John "founder of the Bitcoin-only printing company Proof of Ink". + **Unsupported.** The article never uses the word "founder"; byline is "Proof of Ink by -John". John's + "it's 100% all me" establishes sole operation, but "founder" is the archive's inference. +3. **magazine-021** — "(only Loko and Happy responded)". **Contradicted.** Article: "Then after a few + minutes I checked the chat. A few positive responses. Some funny shit talking, oh look, Happy even + added some lyrics!" — more than two people responded; "only" is wrong. +4. **magazine-041** — IPE: "Stackchain's rowdiest fork war". **Overstated.** Article: "This may go down + as the most fun battle of Stackchain just because of the amount of people involved." — "most fun" + is supported; "rowdiest" is the archive's superlative, not stated. +5. **magazine-061** — "why he keeps a weekly Thursday intelligence-style cadence of work". + **Unsupported.** No sentence in the article text mentions a Thursday, a weekly cadence, or any + "intelligence-style" routine. Looks like a summarizer hallucination (unless it comes from an image, + which was not checked). +6. **magazine-061** — "which the magazine note ties to a centerfold in issue 5 (Round 5)". + **Overstated (minor).** The "(see centerfold)" reference is in the interview answer itself; the + magazine note covers "his epic contribution to the mag in issue 5 (Round 5)". Attribution is wrong + though the fact is supported. +7. **magazine-091** — "Comparing Strategy, Twenty One, Nakamoto, Metaplanet, Coinsilium and Smarter Web + returns and drawdowns against Bitcoin". **Overstated.** Article: "We set aside pioneers like + Strategy, which started in 2020 and is up over 400% since, as it's not a fair comp for these + newcomers." Strategy was deliberately excluded from the comparison; the summary lists it as if it + were one of the analyzed firms. + +None of the 7 breaks the entry's core thesis; the archive's summaries are high-fidelity with small +summarizer-generated embellishments (added superlatives, title inferences, one possible hallucination). + +## Not archive-side decisions + +Per standing convention, wording changes to live archive pages are not made from the pipe; the 7 +clauses above are flagged for the archive editor with the exact proposed corrections implied by the +article sentences quoted. The full per-clause evidence tables are in this file's two working halves. + +Material: `recoveries/magazine-article-map.md` (entry → article URLs, all 200). Sample entries +`/magazine-001,011,021,031,041,051,061,071,081,091/`. Full-fidelity sweep of the remaining 81 entries +proposed as next work (Q7 in today's check-in).