timmy
9dabad495b
Quality gates / quality (pull_request) Canceled after 0s
The corpus had an append-only manifest, a reviewer tool, and an evaluation harness, but nothing a stranger could read: what is in the corpus, what a label means, and how leakage is prevented lived in code and in reviewers' heads. scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back from real records — active samples, contributors, per-split contributor and sample counts, class balance over active samples only — and every fact the pipeline does not collect (devices, contexts, licenses) is printed as pending rather than estimated. The generator refuses to render a card for a corpus where one contributor straddles two splits or where a recorded split is not the one the subject key derives, and it never publishes a subject key or a withdrawn sample's labels. --check fails when the committed page is not the generator's own output, so the card cannot drift into a hand-written document. docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, and photo quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to a human decision. Exclusions are named, and the split policy is stated where it is enforced. The demo's new DATASET CARD scene builds a throwaway corpus with the real ingest command from committed synthetic fixtures, publishes the card, and shows the refusal a planted contributor earns, then re-checks the committed page. Its captions are release gates in build_release.py, which also runs tests/dataset-card.test.py before a release exists. tests/dataset-card.test.py covers the counts, the pending facts, the withdrawal receipt, both refusals, determinism, the empty-corpus case, the committed-page identity, and both documents' prohibitions. Every fixture is synthetic. Closes #27
85 lines
4.7 KiB
Markdown
85 lines
4.7 KiB
Markdown
# Labeling guide — Timmy corpus review
|
||
|
||
This is the guide a human reviewer follows when labelling a contributed photo in
|
||
[`scripts/review_corpus.py`](../scripts/review_corpus.py). It exists so a label in the corpus means
|
||
the same thing to every reviewer, and so a model trained on those labels cannot quietly inherit a
|
||
clinical claim nobody made. The corpus it describes is summarised, from the manifest and never by
|
||
hand, in [DATASET_CARD.md](DATASET_CARD.md).
|
||
|
||
A label is an observation. It is **not** a diagnosis, not a severity assessment, and not a statement
|
||
about the person's health.
|
||
|
||
## What a reviewer may record
|
||
|
||
Three fields, all of them visible in the photo:
|
||
|
||
| Field | Vocabulary | Rule |
|
||
|---|---|---|
|
||
| `bristolType` | 1–7 | The Bristol form the sample most resembles, by shape and surface — not by how the person says they feel. |
|
||
| `color` | `brown`, `green`, `yellow`, `pale`, `red`, `black` | The **broad visible color** of the sample in the photo. |
|
||
| `quality` | `good`, `fair`, `poor` | Whether the photo supports a confident form and color call at all. |
|
||
|
||
`quality` is about the photo, not the person: glare, blur, framing, and distance. A `poor` photo
|
||
belongs in the corpus as a `poor` photo; it is not a reason to guess a higher-quality label, and
|
||
`review_corpus.py` excludes an unclear sample from training instead of filling the gap.
|
||
|
||
## Color is not blood
|
||
|
||
The single most damaging mistake available here. Visible color describes what the photo looks like;
|
||
it does not establish that blood is present:
|
||
|
||
- `red` means **the sample looks reddish in this photo**. It is **not blood**, and it is not
|
||
"bleeding". A reviewer never writes "blood" into a label, a comment, or an exclusion reason.
|
||
- `black` means **the sample looks black or very dark in this photo**. It is not "melena", not
|
||
"GI bleed", and not evidence of anything.
|
||
- Neither field may be used to reassure. A `brown` label is **not** a statement that there is no
|
||
bleeding; a photo cannot rule that out and the app does not claim it can.
|
||
- Blood, black or tarry stool, pain, urgency, fever, or vomiting are **user-reported** facts, asked
|
||
in words and routed to the deterministic red-flag path. They are never inferred from an image and
|
||
never stored as a label by this tool.
|
||
|
||
If the photo genuinely shows something a reviewer believes needs urgent attention, the correct action
|
||
is to stop labelling and escalate to the maintainers — not to encode a clinical judgement in a class.
|
||
|
||
## Never diagnose
|
||
|
||
- Do not diagnose, rule out, or name a disease, condition, or cause in any label, comment, or
|
||
exclusion reason.
|
||
- Do not infer symptoms, pain, urgency, frequency, hydration, diet, medication, or pregnancy.
|
||
- Do not grade severity ("mild", "concerning", "abnormal"). The vocabulary is form, color, and photo
|
||
quality only.
|
||
- A model suggestion is **provisional input**, never the label. The reviewer's own committed decision
|
||
is the label; `review_corpus.py` refuses to reveal the model's answer until the reviewer has
|
||
committed theirs, and records any disagreement beside the proposal rather than averaging it away.
|
||
|
||
## Exclusions
|
||
|
||
Exclude a sample when it cannot be labelled honestly:
|
||
|
||
| Situation | Action |
|
||
|---|---|
|
||
| No stool visible, or the photo is of something else | Exclude: `not-stool`. |
|
||
| Photo too blurred, glared, or distant to judge form | Exclude: `unreadable-photo`. |
|
||
| Two samples in one frame, or ambiguous framing | Exclude: `ambiguous-frame`. |
|
||
| Reviewer believes the image needs urgent clinical attention | **Stop.** Escalate to maintainers; do not label, do not exclude quietly. |
|
||
| Contributor withdrew consent | Not a review decision: the manifest tombstones the contribution and the queue drops it. |
|
||
|
||
An exclusion is recorded with a reason from the list above. An excluded sample is never training
|
||
data, and re-including it requires a new review, not an edit.
|
||
|
||
## Splits: one contributor, one split
|
||
|
||
All of a contributor's photos stay in **one split**, decided by their salted pseudonymous subject key
|
||
(`< 205` train, `< 230` validation, otherwise test — the ingestion rule in
|
||
`scripts/ingest_training_photo.py`). A reviewer never chooses a split, and a reviewer never moves a
|
||
sample between splits to balance a class. If a class is thin, the honest answer is more consented
|
||
contributors, not a leaked one. `scripts/dataset_card.py` refuses to publish a card for a corpus
|
||
where one subject key appears in two splits.
|
||
|
||
## Reporting a problem with this guide
|
||
|
||
If a label feels impossible to assign without a clinical judgement, that is a gap in this guide, not
|
||
a licence to improvise. Open an issue against
|
||
[stackchain/timmy-talking-turd](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd)
|
||
describing the case, without attaching a real medical image.
|