0 Fork 0

Files

T

Quality gates / quality (pull_request) Canceled after 0s

Details

feat(corpus): publish the dataset card and labeling guide from the manifest (#27 )

The corpus had an append-only manifest, a reviewer tool, and an evaluation harness, but nothing a
stranger could read: what is in the corpus, what a label means, and how leakage is prevented lived in
code and in reviewers' heads.

scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back
from real records — active samples, contributors, per-split contributor and sample counts, class
balance over active samples only — and every fact the pipeline does not collect (devices, contexts,
licenses) is printed as pending rather than estimated. The generator refuses to render a card for a
corpus where one contributor straddles two splits or where a recorded split is not the one the
subject key derives, and it never publishes a subject key or a withdrawn sample's labels. --check
fails when the committed page is not the generator's own output, so the card cannot drift into a
hand-written document.

docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, and photo
quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to
a human decision. Exclusions are named, and the split policy is stated where it is enforced.

The demo's new DATASET CARD scene builds a throwaway corpus with the real ingest command from
committed synthetic fixtures, publishes the card, and shows the refusal a planted contributor earns,
then re-checks the committed page. Its captions are release gates in build_release.py, which also runs
tests/dataset-card.test.py before a release exists.

tests/dataset-card.test.py covers the counts, the pending facts, the withdrawal receipt, both
refusals, determinism, the empty-corpus case, the committed-page identity, and both documents'
prohibitions. Every fixture is synthetic.

Closes #27

2026-10-28 11:35:41 +00:00

4.7 KiB

Raw Permalink Blame History

Labeling guide — Timmy corpus review

This is the guide a human reviewer follows when labelling a contributed photo in scripts/review_corpus.py. It exists so a label in the corpus means the same thing to every reviewer, and so a model trained on those labels cannot quietly inherit a clinical claim nobody made. The corpus it describes is summarised, from the manifest and never by hand, in DATASET_CARD.md.

A label is an observation. It is not a diagnosis, not a severity assessment, and not a statement about the person's health.

What a reviewer may record

Three fields, all of them visible in the photo:

Field	Vocabulary	Rule
`bristolType`	1–7	The Bristol form the sample most resembles, by shape and surface — not by how the person says they feel.
`color`	`brown`, `green`, `yellow`, `pale`, `red`, `black`	The broad visible color of the sample in the photo.
`quality`	`good`, `fair`, `poor`	Whether the photo supports a confident form and color call at all.

quality is about the photo, not the person: glare, blur, framing, and distance. A poor photo belongs in the corpus as a poor photo; it is not a reason to guess a higher-quality label, and review_corpus.py excludes an unclear sample from training instead of filling the gap.

Color is not blood

The single most damaging mistake available here. Visible color describes what the photo looks like; it does not establish that blood is present:

red means the sample looks reddish in this photo. It is not blood, and it is not "bleeding". A reviewer never writes "blood" into a label, a comment, or an exclusion reason.
black means the sample looks black or very dark in this photo. It is not "melena", not "GI bleed", and not evidence of anything.
Neither field may be used to reassure. A brown label is not a statement that there is no bleeding; a photo cannot rule that out and the app does not claim it can.
Blood, black or tarry stool, pain, urgency, fever, or vomiting are user-reported facts, asked in words and routed to the deterministic red-flag path. They are never inferred from an image and never stored as a label by this tool.

If the photo genuinely shows something a reviewer believes needs urgent attention, the correct action is to stop labelling and escalate to the maintainers — not to encode a clinical judgement in a class.

Never diagnose

Do not diagnose, rule out, or name a disease, condition, or cause in any label, comment, or exclusion reason.
Do not infer symptoms, pain, urgency, frequency, hydration, diet, medication, or pregnancy.
Do not grade severity ("mild", "concerning", "abnormal"). The vocabulary is form, color, and photo quality only.
A model suggestion is provisional input, never the label. The reviewer's own committed decision is the label; review_corpus.py refuses to reveal the model's answer until the reviewer has committed theirs, and records any disagreement beside the proposal rather than averaging it away.

Exclusions

Exclude a sample when it cannot be labelled honestly:

Situation	Action
No stool visible, or the photo is of something else	Exclude: `not-stool`.
Photo too blurred, glared, or distant to judge form	Exclude: `unreadable-photo`.
Two samples in one frame, or ambiguous framing	Exclude: `ambiguous-frame`.
Reviewer believes the image needs urgent clinical attention	Stop. Escalate to maintainers; do not label, do not exclude quietly.
Contributor withdrew consent	Not a review decision: the manifest tombstones the contribution and the queue drops it.

An exclusion is recorded with a reason from the list above. An excluded sample is never training data, and re-including it requires a new review, not an edit.

Splits: one contributor, one split

All of a contributor's photos stay in one split, decided by their salted pseudonymous subject key (< 205 train, < 230 validation, otherwise test — the ingestion rule in scripts/ingest_training_photo.py). A reviewer never chooses a split, and a reviewer never moves a sample between splits to balance a class. If a class is thin, the honest answer is more consented contributors, not a leaked one. scripts/dataset_card.py refuses to publish a card for a corpus where one subject key appears in two splits.

Reporting a problem with this guide

If a label feels impossible to assign without a clinical judgement, that is a gap in this guide, not a licence to improvise. Open an issue against stackchain/timmy-talking-turd describing the case, without attaching a real medical image.

4.7 KiB Raw Permalink Blame History Unescape Escape