Timmy
c5fa2123e7
Quality gates / quality (pull_request) Canceled after 0s
The corpus manifest records what was ingested and whether the person still allows it; it does not record whether the labels are right. This adds the labeling control the data-flow threat model named: scripts/review_corpus.py, a command-line tool with no app screen. - The review queue shows the contributor's proposal and carries no model field at all: blindness is structural, because build_queue() never receives predictions. - reveal() refuses a contribution with no committed review, so the model answer cannot steer the human who labels it. Revealing appends nothing. - A review is one append-only line carrying the proposal it judged; a correction must change it. An adjudication is a second reviewer's line that names the record it resolves; the review it overrules survives untouched. - A withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and the refusal says it was withdrawn rather than that it is already reviewed. - trainable_records() returns only active samples with a human decision that was not an exclusion, so unreviewed proposals are never training data. Tests are written first against every refusal (tests/corpus-review.test.py, 21 cases); the release builder runs the suite and gates four on-camera captions, and the demo now runs the real queue -> review -> reveal -> adjudicate -> withdraw sequence against the committed synthetic sample and draws the evidence card from that output. docs/REVIEWER-TOOL.md is the contract; no model was measured.
70 lines
4.1 KiB
Markdown
70 lines
4.1 KiB
Markdown
# Corpus manifest contract
|
|
|
|
The training corpus is described by an **append-only** `manifest.jsonl`. One line is one event;
|
|
nothing is ever rewritten in place, so the trail can be audited after a withdrawal. The
|
|
implementation lives in [`scripts/corpus_manifest.py`](../scripts/corpus_manifest.py) and is
|
|
driven by [`scripts/ingest_training_photo.py`](../scripts/ingest_training_photo.py).
|
|
|
|
## Sample record (`recordType: "sample"`, schema version 2)
|
|
|
|
| Field | Meaning |
|
|
|---|---|
|
|
| `contributionId` | Opaque id minted by the app's contribution receipt. The same id addresses every derived artifact and every later tombstone. |
|
|
| `consent.version`, `consent.recordedAt` | The consent version the contributor accepted, and when. Duplicated as `consentVersion` for older readers. |
|
|
| `source.kind`, `source.inputSha256` | How the bytes arrived (`cli-ingest`) and the digest of the caller's input. The caller's path is never recorded. |
|
|
| `hashes.rawInput`, `hashes.derived` | Digest of the supplied bytes and of the re-encoded image that is actually stored. |
|
|
| `derivation` | `format`, `maxDimension`, `metadataStripped`, and the `pipeline` version that produced the derived image. |
|
|
| `review.status`, `review.reviewedAt` | `user-confirmed` or `clinician-reviewed`. A label set is never unlabelled-by-default. |
|
|
| `split` | `train`, `validation`, or `test`, derived from the salted subject key so one contributor never straddles splits. |
|
|
| `subjectKey` | 16-hex pseudonym derived from a salted subject id. Never a name, email, or account. |
|
|
| `lifecycle.state`, `lifecycle.updatedAt` | `active` while the sample is usable for training. |
|
|
| `imageId`, `relativePath` | Derived image digest and its dataset-relative path. |
|
|
|
|
## Tombstone record (`recordType: "tombstone"`)
|
|
|
|
A withdrawal never deletes history; it appends a tombstone that:
|
|
|
|
- keeps `contributionId` and `imageId` so the withdrawal can be matched to the original sample,
|
|
- records `lifecycle.state: "revoked"`, the timestamp, and the reason,
|
|
- lists `deletedArtifacts` — every derived copy it removed, addressed by contribution id with the
|
|
digest that identifies the bytes,
|
|
- deliberately drops `labels`, `subjectKey`, and `relativePath`: a deletion receipt keeps
|
|
identifiers and hashes, not the health label or the person.
|
|
|
|
`revoke()` removes the derived image and any raw pre-derivation copy from the active dataset
|
|
before appending the tombstone, and refuses any path that resolves outside the dataset root.
|
|
|
|
## Lifecycle rules (all covered by `tests/corpus-manifest.test.py`)
|
|
|
|
- `active` → `revoked` only, and only with a recorded reason.
|
|
- `revoked` → `active` is refused: a revoked contribution cannot return to the active dataset.
|
|
- Revoking an unknown contribution, or revoking twice, is refused and appends nothing.
|
|
- A contribution id can be ingested once; the same derived image cannot carry two label sets.
|
|
- `active_records()` returns only samples that are still `active` and were never tombstoned, so a
|
|
trainer that reads through it cannot silently train on withdrawn data.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
export TIMMY_DATASET_SALT=... # >= 8 chars, never committed
|
|
python3 scripts/ingest_training_photo.py \
|
|
--input photo.jpg --output-dir /srv/timmy-corpus \
|
|
--subject-id subject-123 --consent-version research-v1 \
|
|
--contribution-id "$CONTRIBUTION_ID" \
|
|
--bristol-type 4 --color brown --quality good --reviewer user-confirmed
|
|
|
|
python3 scripts/ingest_training_photo.py --revoke "$CONTRIBUTION_ID" \
|
|
--output-dir /srv/timmy-corpus --reason "subject withdrawal"
|
|
```
|
|
|
|
`npm run test:corpus` runs the contract suite. Every fixture is synthetic; no real medical image
|
|
is read, written, or committed.
|
|
|
|
## Review and adjudication
|
|
|
|
A manifest says what was ingested and whether the person still allows it. It does not say whether
|
|
the labels are right. That decision lives in a separate append-only ledger,
|
|
`<dataset>/reviews.jsonl`, written by [scripts/review_corpus.py](review_corpus.py) — see
|
|
[REVIEWER-TOOL.md](REVIEWER-TOOL.md). The trainer reads `trainable_records()`, which returns only
|
|
samples that are still active, carry a human decision, and were not excluded.
|