timmy
9dabad495b
Quality gates / quality (pull_request) Canceled after 0s
The corpus had an append-only manifest, a reviewer tool, and an evaluation harness, but nothing a stranger could read: what is in the corpus, what a label means, and how leakage is prevented lived in code and in reviewers' heads. scripts/dataset_card.py generates docs/DATASET_CARD.md from manifest.jsonl. Every count is read back from real records — active samples, contributors, per-split contributor and sample counts, class balance over active samples only — and every fact the pipeline does not collect (devices, contexts, licenses) is printed as pending rather than estimated. The generator refuses to render a card for a corpus where one contributor straddles two splits or where a recorded split is not the one the subject key derives, and it never publishes a subject key or a withdrawn sample's labels. --check fails when the committed page is not the generator's own output, so the card cannot drift into a hand-written document. docs/LABELING_GUIDE.md is the reviewer's rulebook: visible form, broad visible color, and photo quality; visible color is not blood; nothing is diagnosed; a model suggestion is provisional input to a human decision. Exclusions are named, and the split policy is stated where it is enforced. The demo's new DATASET CARD scene builds a throwaway corpus with the real ingest command from committed synthetic fixtures, publishes the card, and shows the refusal a planted contributor earns, then re-checks the committed page. Its captions are release gates in build_release.py, which also runs tests/dataset-card.test.py before a release exists. tests/dataset-card.test.py covers the counts, the pending facts, the withdrawal receipt, both refusals, determinism, the empty-corpus case, the committed-page identity, and both documents' prohibitions. Every fixture is synthetic. Closes #27
105 lines
3.6 KiB
Markdown
105 lines
3.6 KiB
Markdown
# Dataset card — Timmy consented stool-image corpus
|
|
|
|
Generated by [`scripts/dataset_card.py`](../scripts/dataset_card.py) from the append-only
|
|
corpus manifest. Do not edit this page by hand: regenerate it, or `--check` fails. Every count
|
|
below is read back from `manifest.jsonl`; a fact the pipeline does not collect is printed as
|
|
`pending` rather than estimated, so nothing here is invented.
|
|
|
|
| Field | Value |
|
|
|---|---|
|
|
| Corpus | `/srv/timmy-corpus` |
|
|
| Corpus present on this machine | no |
|
|
| Last manifest record | pending — no record yet |
|
|
| Active samples | 0 |
|
|
| Contributors | 0 |
|
|
| Withdrawals | 0 |
|
|
| Contexts | pending — not collected by ingest — no setting, time-of-day, or location fields are recorded |
|
|
| Devices | pending — not collected by ingest — no camera, model, or OS fields are recorded |
|
|
| Licenses | pending — no contributor grant is recorded yet, so no license is asserted |
|
|
| Regenerate | `python3 scripts/dataset_card.py --corpus /srv/timmy-corpus` |
|
|
|
|
## What this corpus is for
|
|
|
|
Consented photos of stool, labelled by a human reviewer, for research on a stool-form
|
|
classifier. The corpus may not be used to diagnose disease, to identify a person, or to
|
|
train a model that speaks to a patient with clinical authority. A model's output stays
|
|
provisional until the person logging confirms it: see [PRODUCT-DECISIONS.md](PRODUCT-DECISIONS.md).
|
|
|
|
This corpus holds no consented images yet. Every count of zero below is a real count of
|
|
an empty manifest, not a placeholder, and the pending facts stay pending until the
|
|
pipeline collects them.
|
|
|
|
## Contributor-level split policy
|
|
|
|
Splits are derived from the salted pseudonymous subject key, never assigned per photo: the
|
|
first byte of the key decides the split, `< 205` train, `< 230` validation, otherwise test
|
|
(the ingestion rule in `scripts/ingest_training_photo.py`). One contributor's photos all land
|
|
in one split, and this generator refuses a manifest where a subject key appears in two splits
|
|
or where a recorded split is not the one its subject key derives. Both edges are stated here
|
|
so the policy on this page is the policy in the code.
|
|
|
|
| Split | Contributors | Active samples |
|
|
|---|---:|---:|
|
|
| `train` | 0 | 0 |
|
|
| `validation` | 0 | 0 |
|
|
| `test` | 0 | 0 |
|
|
|
|
## Class balance (active samples only)
|
|
|
|
**Type**
|
|
|
|
| Type | Active samples |
|
|
|---|---:|
|
|
| Type 1 | 0 |
|
|
| Type 2 | 0 |
|
|
| Type 3 | 0 |
|
|
| Type 4 | 0 |
|
|
| Type 5 | 0 |
|
|
| Type 6 | 0 |
|
|
| Type 7 | 0 |
|
|
|
|
**Color**
|
|
|
|
| Color | Active samples |
|
|
|---|---:|
|
|
| `brown` | 0 |
|
|
| `green` | 0 |
|
|
| `yellow` | 0 |
|
|
| `pale` | 0 |
|
|
| `red` | 0 |
|
|
| `black` | 0 |
|
|
|
|
**Quality**
|
|
|
|
| Quality | Active samples |
|
|
|---|---:|
|
|
| `good` | 0 |
|
|
| `fair` | 0 |
|
|
| `poor` | 0 |
|
|
|
|
**Review status**
|
|
|
|
| Review status | Active samples |
|
|
|---|---:|
|
|
| `user-confirmed` | 0 |
|
|
| `clinician-reviewed` | 0 |
|
|
|
|
A withdrawn sample is excluded from every table above. Labels are what a reviewer saw, not a
|
|
diagnosis: [LABELING_GUIDE.md](LABELING_GUIDE.md) says what a label may and may not assert.
|
|
|
|
## Exclusions and withdrawals
|
|
|
|
A withdrawal appends a tombstone, deletes the derived image and any raw copy, and stays in the
|
|
history. Contribution ids and reasons are published here because a deletion receipt is
|
|
checkable; subject keys and labels are never published.
|
|
|
|
No withdrawals are recorded.
|
|
|
|
## Limitations
|
|
|
|
- Composition (devices, contexts, licenses) is pending, so the corpus is not yet a
|
|
representative sample of anything and must not be described as one.
|
|
- Review status is a human decision, not a clinical gold standard; `user-confirmed` means the
|
|
person who took the photo agreed with the proposal.
|
|
- A held-out split measures generalization across contributors, not across a population.
|