0 Fork 0

Files

T

Quality gates / quality (pull_request) Canceled after 0s

Details

feat(review): blind reviewer and adjudication tool with an append-only label ledger (#26 )

The corpus manifest records what was ingested and whether the person still allows it; it does not
record whether the labels are right. This adds the labeling control the data-flow threat model
named: scripts/review_corpus.py, a command-line tool with no app screen.

- The review queue shows the contributor's proposal and carries no model field at all: blindness is
structural, because build_queue() never receives predictions.
- reveal() refuses a contribution with no committed review, so the model answer cannot steer the
human who labels it. Revealing appends nothing.
- A review is one append-only line carrying the proposal it judged; a correction must change it. An
adjudication is a second reviewer's line that names the record it resolves; the review it
overrules survives untouched.
- A withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and the
refusal says it was withdrawn rather than that it is already reviewed.
- trainable_records() returns only active samples with a human decision that was not an exclusion,
so unreviewed proposals are never training data.

Tests are written first against every refusal (tests/corpus-review.test.py, 21 cases); the release
builder runs the suite and gates four on-camera captions, and the demo now runs the real queue ->
review -> reveal -> adjudicate -> withdraw sequence against the committed synthetic sample and draws
the evidence card from that output. docs/REVIEWER-TOOL.md is the contract; no model was measured.

2026-10-27 11:43:47 +00:00

4.1 KiB

Raw Blame History

Corpus manifest contract

The training corpus is described by an append-only manifest.jsonl. One line is one event; nothing is ever rewritten in place, so the trail can be audited after a withdrawal. The implementation lives in scripts/corpus_manifest.py and is driven by scripts/ingest_training_photo.py.

Sample record (`recordType: "sample"`, schema version 2)

Field	Meaning
`contributionId`	Opaque id minted by the app's contribution receipt. The same id addresses every derived artifact and every later tombstone.
`consent.version`, `consent.recordedAt`	The consent version the contributor accepted, and when. Duplicated as `consentVersion` for older readers.
`source.kind`, `source.inputSha256`	How the bytes arrived (`cli-ingest`) and the digest of the caller's input. The caller's path is never recorded.
`hashes.rawInput`, `hashes.derived`	Digest of the supplied bytes and of the re-encoded image that is actually stored.
`derivation`	`format`, `maxDimension`, `metadataStripped`, and the `pipeline` version that produced the derived image.
`review.status`, `review.reviewedAt`	`user-confirmed` or `clinician-reviewed`. A label set is never unlabelled-by-default.
`split`	`train`, `validation`, or `test`, derived from the salted subject key so one contributor never straddles splits.
`subjectKey`	16-hex pseudonym derived from a salted subject id. Never a name, email, or account.
`lifecycle.state`, `lifecycle.updatedAt`	`active` while the sample is usable for training.
`imageId`, `relativePath`	Derived image digest and its dataset-relative path.

Tombstone record (`recordType: "tombstone"`)

A withdrawal never deletes history; it appends a tombstone that:

keeps contributionId and imageId so the withdrawal can be matched to the original sample,
records lifecycle.state: "revoked", the timestamp, and the reason,
lists deletedArtifacts — every derived copy it removed, addressed by contribution id with the digest that identifies the bytes,
deliberately drops labels, subjectKey, and relativePath: a deletion receipt keeps identifiers and hashes, not the health label or the person.

revoke() removes the derived image and any raw pre-derivation copy from the active dataset before appending the tombstone, and refuses any path that resolves outside the dataset root.

Lifecycle rules (all covered by `tests/corpus-manifest.test.py`)

active → revoked only, and only with a recorded reason.
revoked → active is refused: a revoked contribution cannot return to the active dataset.
Revoking an unknown contribution, or revoking twice, is refused and appends nothing.
A contribution id can be ingested once; the same derived image cannot carry two label sets.
active_records() returns only samples that are still active and were never tombstoned, so a trainer that reads through it cannot silently train on withdrawn data.

Usage

export TIMMY_DATASET_SALT=...                       # >= 8 chars, never committed
python3 scripts/ingest_training_photo.py \
  --input photo.jpg --output-dir /srv/timmy-corpus \
  --subject-id subject-123 --consent-version research-v1 \
  --contribution-id "$CONTRIBUTION_ID" \
  --bristol-type 4 --color brown --quality good --reviewer user-confirmed

python3 scripts/ingest_training_photo.py --revoke "$CONTRIBUTION_ID" \
  --output-dir /srv/timmy-corpus --reason "subject withdrawal"

npm run test:corpus runs the contract suite. Every fixture is synthetic; no real medical image is read, written, or committed.

Review and adjudication

A manifest says what was ingested and whether the person still allows it. It does not say whether the labels are right. That decision lives in a separate append-only ledger, <dataset>/reviews.jsonl, written by scripts/review_corpus.py — see REVIEWER-TOOL.md. The trainer reads trainable_records(), which returns only samples that are still active, carry a human decision, and were not excluded.

4.1 KiB Raw Blame History