Q8 pilot: summary fidelity, 10-entry sample
This commit is contained in:
zephyrus-sparkborn committed
1 parent
2a0cb01247
commit
dcf91a1a15
1 file changed
+73
@@ -0,0 +1,73 @@
|
|||||||
|
# Q8 pilot — summary fidelity, 10-entry sample
|
||||||
|
|
||||||
|
Compiled 2026-09-25 by zephyrus-sparkborn. Q8 asks whether each entry's machine-written summary
|
||||||
|
overstates its article. This pilot defines the sample as **every 10th magazine entry** (`001, 011,
|
||||||
|
021, 031, 041, 051, 061, 071, 081, 091`) — non-cherry-picked — and grades every factual or
|
||||||
|
interpretive clause of the archive's **"What happened"** and **"In plain English"** prose against the
|
||||||
|
live article. The "From the sources" blockquotes were excluded: quote-fidelity was already verified at
|
||||||
|
269/269 in `recoveries/quote-fidelity-live.md`.
|
||||||
|
|
||||||
|
**Method.** For each entry: `browser.open` the entry page
|
||||||
|
(`https://alexanderwhitestone.com/stackchain/magazine-0NN/`), copy the two summary sections verbatim,
|
||||||
|
`browser.open` the live article URL from `recoveries/magazine-article-map.md`, split the summary into
|
||||||
|
clauses, and grade each: **holds** (article says it) / **overstated** (article supports less) /
|
||||||
|
**unsupported** (article doesn't say it) / **contradicted** (public sources say otherwise) /
|
||||||
|
**unverified** (could not fetch). Every grade carries the summary clause and the settling article
|
||||||
|
sentence verbatim. All 20 pages fetched cleanly on 2026-09-25; raw article texts kept locally, full
|
||||||
|
per-clause tables with verbatim quotes in the two halves' working files.
|
||||||
|
|
||||||
|
## Totals — 10 entries, 98 clauses
|
||||||
|
|
||||||
|
| grade | count |
|
||||||
|
|---|---|
|
||||||
|
| holds | 91 |
|
||||||
|
| overstated | 4 |
|
||||||
|
| unsupported | 2 |
|
||||||
|
| contradicted | 1 |
|
||||||
|
| unverified | 0 |
|
||||||
|
|
||||||
|
Per entry (holds / overstated / unsupported / contradicted / unverified):
|
||||||
|
`001`: 10/0/0/0/0 · `011`: 11/1/1/0/0 · `021`: 10/0/0/1/0 · `031`: 8/0/0/0/0 · `041`: 9/1/0/0/0 ·
|
||||||
|
`051`: 11/0/0/0/0 · `061`: 10/1/1/0/0 · `071`: 8/0/0/0/0 · `081`: 9/0/0/0/0 · `091`: 6/1/0/0/0.
|
||||||
|
|
||||||
|
## Entries that need an archive-side wording fix (7 clauses)
|
||||||
|
|
||||||
|
1. **magazine-011** — summary says "merch for Mass Adoption and the Boston Bitcoin Festival".
|
||||||
|
**Overstated.** Article: "I'm also working with Mass Adoption, the Boston Bitcoin group. I make all
|
||||||
|
their merch and I'll be printing live June 24th at The Boston Bitcoin Festival…" — merch for Mass
|
||||||
|
Adoption, printing live *at* the festival, not merch *for* the festival.
|
||||||
|
2. **magazine-011** — IPE calls John "founder of the Bitcoin-only printing company Proof of Ink".
|
||||||
|
**Unsupported.** The article never uses the word "founder"; byline is "Proof of Ink by -John". John's
|
||||||
|
"it's 100% all me" establishes sole operation, but "founder" is the archive's inference.
|
||||||
|
3. **magazine-021** — "(only Loko and Happy responded)". **Contradicted.** Article: "Then after a few
|
||||||
|
minutes I checked the chat. A few positive responses. Some funny shit talking, oh look, Happy even
|
||||||
|
added some lyrics!" — more than two people responded; "only" is wrong.
|
||||||
|
4. **magazine-041** — IPE: "Stackchain's rowdiest fork war". **Overstated.** Article: "This may go down
|
||||||
|
as the most fun battle of Stackchain just because of the amount of people involved." — "most fun"
|
||||||
|
is supported; "rowdiest" is the archive's superlative, not stated.
|
||||||
|
5. **magazine-061** — "why he keeps a weekly Thursday intelligence-style cadence of work".
|
||||||
|
**Unsupported.** No sentence in the article text mentions a Thursday, a weekly cadence, or any
|
||||||
|
"intelligence-style" routine. Looks like a summarizer hallucination (unless it comes from an image,
|
||||||
|
which was not checked).
|
||||||
|
6. **magazine-061** — "which the magazine note ties to a centerfold in issue 5 (Round 5)".
|
||||||
|
**Overstated (minor).** The "(see centerfold)" reference is in the interview answer itself; the
|
||||||
|
magazine note covers "his epic contribution to the mag in issue 5 (Round 5)". Attribution is wrong
|
||||||
|
though the fact is supported.
|
||||||
|
7. **magazine-091** — "Comparing Strategy, Twenty One, Nakamoto, Metaplanet, Coinsilium and Smarter Web
|
||||||
|
returns and drawdowns against Bitcoin". **Overstated.** Article: "We set aside pioneers like
|
||||||
|
Strategy, which started in 2020 and is up over 400% since, as it's not a fair comp for these
|
||||||
|
newcomers." Strategy was deliberately excluded from the comparison; the summary lists it as if it
|
||||||
|
were one of the analyzed firms.
|
||||||
|
|
||||||
|
None of the 7 breaks the entry's core thesis; the archive's summaries are high-fidelity with small
|
||||||
|
summarizer-generated embellishments (added superlatives, title inferences, one possible hallucination).
|
||||||
|
|
||||||
|
## Not archive-side decisions
|
||||||
|
|
||||||
|
Per standing convention, wording changes to live archive pages are not made from the pipe; the 7
|
||||||
|
clauses above are flagged for the archive editor with the exact proposed corrections implied by the
|
||||||
|
article sentences quoted. The full per-clause evidence tables are in this file's two working halves.
|
||||||
|
|
||||||
|
Material: `recoveries/magazine-article-map.md` (entry → article URLs, all 200). Sample entries
|
||||||
|
`/magazine-001,011,021,031,041,051,061,071,081,091/`. Full-fidelity sweep of the remaining 81 entries
|
||||||
|
proposed as next work (Q7 in today's check-in).
|
||||||
Reference in new issue
Block a user