Publish the dataset card and labeling guide from the manifest (#27) #89

2026-10-28T11:35:58Z

rockachopa commented

2026-10-28 11:35:58 +00:00

Closes #27

What this delivers

The corpus already had an append-only manifest (#25), a blind reviewer tool (#26) and a held-out evaluation harness (#29). It had no page a stranger could read: what is in the corpus, what a label may assert, and how contributor leakage is prevented lived in code and in reviewers' heads.

scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back from real records; every fact the pipeline does not collect (devices, contexts, licenses) prints as pending, never an estimate.
The generator refuses to render a card when one contributor straddles two splits, or when a recorded split is not the one the subject key derives. It never publishes a subject key or a withdrawn sample's labels — the withdrawal appears as a contribution id and a reason only.
--check fails when the committed page is not the generator's own output, so the published card cannot drift into a hand-written document.
docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, photo quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to a human decision. Exclusions named; the contributor-level split policy stated where it is enforced.

Tests

tests/dataset-card.test.py — 14 tests (counts, pending facts, withdrawal receipt, both refusals, determinism, empty corpus, committed-page identity, README discovery, both documents' prohibitions). All synthetic.
Full JS suite: npm test → 229 pass / 0 fail. Python suites: dataset-card 14, corpus-manifest 8, corpus-review 21, calibration-harness 19 → all OK.
npm run check:syntax and npm run check:diff clean.
Note: tests/staging-deploy.test.py fails 5 tests on this host (/mnt/data/... vs /root/.hermes/... scratch path mismatch). Verified pre-existing on origin/main in a clean worktree — not caused by this branch.

Demonstration path

scripts/record_release_demo.mjs gained a DATASET CARD scene that runs the real ingest command on three committed synthetic fixtures, withdraws one contribution, publishes the card, shows the refusal a planted contributor earns, and re-checks docs/DATASET_CARD.md. Verified by a full real-browser recording (exit 0, 19.8 MB webm); the caption receipt confirms all four new captions were painted as on-screen bars. Captions are gated in build_release.py, which also runs the new suite before a release can exist.

Boundary

No change to the medical/non-diagnostic boundary, no infrastructure spend, no model weights, no real medical images, no production service change. The card states the corpus is empty today (0 active samples) rather than implying data exists.

Ownership note

The maintainer comment on #27 approved a docs-only slice for Zephyrus on 2026-09-19; it has been idle since. This PR delivers the approved scope as a generated artifact instead of hand-written prose, so the numbers cannot drift. The docs remain editable by anyone; the generator is the source of the numbers.

Closes #27 ## What this delivers The corpus already had an append-only manifest (#25), a blind reviewer tool (#26) and a held-out evaluation harness (#29). It had no page a stranger could read: what is in the corpus, what a label may assert, and how contributor leakage is prevented lived in code and in reviewers' heads. - `scripts/dataset_card.py` generates `docs/DATASET_CARD.md` from `manifest.jsonl`. Every count is read back from real records; every fact the pipeline does not collect (devices, contexts, licenses) prints as `pending`, never an estimate. - The generator refuses to render a card when one contributor straddles two splits, or when a recorded split is not the one the subject key derives. It never publishes a subject key or a withdrawn sample's labels — the withdrawal appears as a contribution id and a reason only. - `--check` fails when the committed page is not the generator's own output, so the published card cannot drift into a hand-written document. - `docs/LABELING_GUIDE.md` is the reviewer's rulebook: visible form, broad visible color, photo quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to a human decision. Exclusions named; the contributor-level split policy stated where it is enforced. ## Tests - `tests/dataset-card.test.py` — 14 tests (counts, pending facts, withdrawal receipt, both refusals, determinism, empty corpus, committed-page identity, README discovery, both documents' prohibitions). All synthetic. - Full JS suite: `npm test` → 229 pass / 0 fail. Python suites: dataset-card 14, corpus-manifest 8, corpus-review 21, calibration-harness 19 → all OK. - `npm run check:syntax` and `npm run check:diff` clean. - Note: `tests/staging-deploy.test.py` fails 5 tests on this host (`/mnt/data/...` vs `/root/.hermes/...` scratch path mismatch). Verified pre-existing on `origin/main` in a clean worktree — not caused by this branch. ## Demonstration path `scripts/record_release_demo.mjs` gained a DATASET CARD scene that runs the real ingest command on three committed synthetic fixtures, withdraws one contribution, publishes the card, shows the refusal a planted contributor earns, and re-checks `docs/DATASET_CARD.md`. Verified by a full real-browser recording (exit 0, 19.8 MB webm); the caption receipt confirms all four new captions were painted as on-screen bars. Captions are gated in `build_release.py`, which also runs the new suite before a release can exist. ## Boundary No change to the medical/non-diagnostic boundary, no infrastructure spend, no model weights, no real medical images, no production service change. The card states the corpus is empty today (0 active samples) rather than implying data exists. ## Ownership note The maintainer comment on #27 approved a docs-only slice for Zephyrus on 2026-09-19; it has been idle since. This PR delivers the approved scope as a generated artifact instead of hand-written prose, so the numbers cannot drift. The docs remain editable by anyone; the generator is the source of the numbers.

rockachopa added this to the M2 — Consented Dataset & Specialist Model milestone 2026-10-28 11:35:58 +00:00

rockachopa added 1 commit 2026-10-28 11:35:58 +00:00

feat(corpus): publish the dataset card and labeling guide from the manifest (#27 )

Quality gates / quality (pull_request) Canceled after 0s

Details

9dabad495b

The corpus had an append-only manifest, a reviewer tool, and an evaluation harness, but nothing a
stranger could read: what is in the corpus, what a label means, and how leakage is prevented lived in
code and in reviewers' heads.

scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back
from real records — active samples, contributors, per-split contributor and sample counts, class
balance over active samples only — and every fact the pipeline does not collect (devices, contexts,
licenses) is printed as pending rather than estimated. The generator refuses to render a card for a
corpus where one contributor straddles two splits or where a recorded split is not the one the
subject key derives, and it never publishes a subject key or a withdrawn sample's labels. --check
fails when the committed page is not the generator's own output, so the card cannot drift into a
hand-written document.

docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, and photo
quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to
a human decision. Exclusions are named, and the split policy is stated where it is enforced.

The demo's new DATASET CARD scene builds a throwaway corpus with the real ingest command from
committed synthetic fixtures, publishes the card, and shows the refusal a planted contributor earns,
then re-checks the committed page. Its captions are release gates in build_release.py, which also runs
tests/dataset-card.test.py before a release exists.

tests/dataset-card.test.py covers the counts, the pending facts, the withdrawal receipt, both
refusals, determinism, the empty-corpus case, the committed-page identity, and both documents'
prohibitions. Every fixture is synthetic.

Closes #27

rockachopa merged commit d7026c0ed3 into main

2026-10-28 12:02:28 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#89