timmy
860c84f0fe
Quality gates / quality (pull_request) Failing after 9m42s
Promote the ingest script into an auditable corpus pipeline. Each sample now records consent (version + timestamp), source kind and raw-input digest, both hashes, the derivation pipeline, review status, split, and lifecycle state, with the caller's path never written down. Withdrawal is a new `--revoke <contribution-id> --reason` operation: it deletes the derived image and any raw pre-derivation copy, then appends a tombstone that addresses each removed artifact by contribution id and digest while dropping labels, subject key, and paths. History is append-only, so the original sample record survives the withdrawal it documents. Illegal moves are refused instead of silently applied: revoked -> active, revoking an unknown or already-revoked contribution, an unknown state, a duplicate contribution id, and a second label set for the same derived image. The duplicate guard runs before any byte is written, so a refused ingest cannot strand an image in the active dataset. Tests written first and watched fail (tests/corpus-manifest.test.py): provenance, tombstone + byte removal, active-record filtering, and the transition guards. The transition helper itself passed on first write and is reported as a hold-out check rather than a RED cycle. Wired into the quality workflow and check:syntax; contract documented in docs/CORPUS-MANIFEST.md. No browser surface changed, so record_release_demo.mjs is untouched and claims nothing new: the nearest honest visible effect is the withdrawn-receipt tombstone already recorded for #24.
3.6 KiB
3.6 KiB
Corpus manifest contract
The training corpus is described by an append-only manifest.jsonl. One line is one event;
nothing is ever rewritten in place, so the trail can be audited after a withdrawal. The
implementation lives in scripts/corpus_manifest.py and is
driven by scripts/ingest_training_photo.py.
Sample record (recordType: "sample", schema version 2)
| Field | Meaning |
|---|---|
contributionId |
Opaque id minted by the app's contribution receipt. The same id addresses every derived artifact and every later tombstone. |
consent.version, consent.recordedAt |
The consent version the contributor accepted, and when. Duplicated as consentVersion for older readers. |
source.kind, source.inputSha256 |
How the bytes arrived (cli-ingest) and the digest of the caller's input. The caller's path is never recorded. |
hashes.rawInput, hashes.derived |
Digest of the supplied bytes and of the re-encoded image that is actually stored. |
derivation |
format, maxDimension, metadataStripped, and the pipeline version that produced the derived image. |
review.status, review.reviewedAt |
user-confirmed or clinician-reviewed. A label set is never unlabelled-by-default. |
split |
train, validation, or test, derived from the salted subject key so one contributor never straddles splits. |
subjectKey |
16-hex pseudonym derived from a salted subject id. Never a name, email, or account. |
lifecycle.state, lifecycle.updatedAt |
active while the sample is usable for training. |
imageId, relativePath |
Derived image digest and its dataset-relative path. |
Tombstone record (recordType: "tombstone")
A withdrawal never deletes history; it appends a tombstone that:
- keeps
contributionIdandimageIdso the withdrawal can be matched to the original sample, - records
lifecycle.state: "revoked", the timestamp, and the reason, - lists
deletedArtifacts— every derived copy it removed, addressed by contribution id with the digest that identifies the bytes, - deliberately drops
labels,subjectKey, andrelativePath: a deletion receipt keeps identifiers and hashes, not the health label or the person.
revoke() removes the derived image and any raw pre-derivation copy from the active dataset
before appending the tombstone, and refuses any path that resolves outside the dataset root.
Lifecycle rules (all covered by tests/corpus-manifest.test.py)
active→revokedonly, and only with a recorded reason.revoked→activeis refused: a revoked contribution cannot return to the active dataset.- Revoking an unknown contribution, or revoking twice, is refused and appends nothing.
- A contribution id can be ingested once; the same derived image cannot carry two label sets.
active_records()returns only samples that are stillactiveand were never tombstoned, so a trainer that reads through it cannot silently train on withdrawn data.
Usage
export TIMMY_DATASET_SALT=... # >= 8 chars, never committed
python3 scripts/ingest_training_photo.py \
--input photo.jpg --output-dir /srv/timmy-corpus \
--subject-id subject-123 --consent-version research-v1 \
--contribution-id "$CONTRIBUTION_ID" \
--bristol-type 4 --color brown --quality good --reviewer user-confirmed
python3 scripts/ingest_training_photo.py --revoke "$CONTRIBUTION_ID" \
--output-dir /srv/timmy-corpus --reason "subject withdrawal"
npm run test:corpus runs the contract suite. Every fixture is synthetic; no real medical image
is read, written, or committed.