0 Fork 0

Files

T

Quality gates / quality (pull_request) Canceled after 0s

Details

feat(review): blind reviewer and adjudication tool with an append-only label ledger (#26 )

The corpus manifest records what was ingested and whether the person still allows it; it does not
record whether the labels are right. This adds the labeling control the data-flow threat model
named: scripts/review_corpus.py, a command-line tool with no app screen.

- The review queue shows the contributor's proposal and carries no model field at all: blindness is
structural, because build_queue() never receives predictions.
- reveal() refuses a contribution with no committed review, so the model answer cannot steer the
human who labels it. Revealing appends nothing.
- A review is one append-only line carrying the proposal it judged; a correction must change it. An
adjudication is a second reviewer's line that names the record it resolves; the review it
overrules survives untouched.
- A withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and the
refusal says it was withdrawn rather than that it is already reviewed.
- trainable_records() returns only active samples with a human decision that was not an exclusion,
so unreviewed proposals are never training data.

Tests are written first against every refusal (tests/corpus-review.test.py, 21 cases); the release
builder runs the suite and gates four on-camera captions, and the demo now runs the real queue ->
review -> reveal -> adjudicate -> withdraw sequence against the committed synthetic sample and draws
the evidence card from that output. docs/REVIEWER-TOOL.md is the contract; no model was measured.

2026-10-27 11:43:47 +00:00

5.2 KiB

Raw Permalink Blame History

Blind review and adjudication tool

Contract for scripts/review_corpus.py, the labeling control named in the photo data-flow threat model and the gate in front of the training corpus. Issue #26.

The tool decides which samples a trainer is allowed to read. It is a command-line tool, not a screen in the app: there is no reviewer UI, and nothing here implies one.

The four invariants

The model answer is withheld until a human commits. build_queue() takes no predictions at all, so the queue cannot leak an answer it never receives. reveal() refuses to return anything for a contribution with no committed review.
History is append-only. A review is a line in reviews.jsonl. An adjudication is another line that names the record it resolves (resolvesRecord, resolves). The review being overruled is never edited, and no record is deleted.
A withdrawal beats every other answer. A revoked contribution cannot be queued, reviewed, revealed, adjudicated, or trained on. The refusal says it was withdrawn — not that it is already reviewed — because the person's withdrawal is the fact that matters.
Unreviewed and excluded samples are not training data. trainable_records() returns only samples that are still active, carry a human decision, and were not excluded.

Review ledger records

One JSON object per line in <dataset>/reviews.jsonl:

{"schemaVersion":1,"recordType":"review","contributionId":"...","imageId":"<sha256>","proposed":{"bristolType":4,"color":"brown","quality":"good"},"reviewer":"reviewer-a","decision":"corrected","labels":{"bristolType":5,"color":"green","quality":"fair"},"committedAt":"2026-10-27T09:00:00+00:00"}
{"schemaVersion":1,"recordType":"adjudication","contributionId":"...","imageId":"<sha256>","resolvesRecord":1,"resolves":{"reviewer":"reviewer-a","decision":"corrected"},"reviewer":"reviewer-b","decision":"upheld","labels":{"bristolType":5,"color":"green","quality":"fair"},"adjudicatedAt":"2026-10-27T10:00:00+00:00"}

proposed copies the labels the reviewer judged, so a later manifest change cannot rewrite what was reviewed.
decision is agreed, corrected, or excluded for a review, and upheld, corrected, or excluded for an adjudication. An exclusion carries exclusionReason and no labels.
The ledger never carries subjectKey, note text, symptom answers, or image bytes.

A model answer file

reveal reads predictions only to show what the reviewer could not see. A file may be one JSON document or one object per line:

{"model":"<name>","answers":{"<contributionId>":{"bristolType":6,"color":"brown","quality":"good","confidence":0.71}}}

Keys may be contribution ids or derived image digests. reveal is a read: it appends nothing. When no answer file is supplied, model is null — the tool does not invent an answer.

Usage

python3 scripts/review_corpus.py queue --dataset /srv/timmy-corpus
python3 scripts/review_corpus.py review --dataset /srv/timmy-corpus \
  --contribution "$CONTRIBUTION_ID" --reviewer reviewer-a --bristol 5 --color green --quality fair
python3 scripts/review_corpus.py reveal --dataset /srv/timmy-corpus \
  --contribution "$CONTRIBUTION_ID" --predictions answers.json
python3 scripts/review_corpus.py adjudicate --dataset /srv/timmy-corpus \
  --contribution "$CONTRIBUTION_ID" --reviewer reviewer-b --upheld
python3 scripts/review_corpus.py trainable --dataset /srv/timmy-corpus

Every mutating command prints the record it appended. A refusal exits 2 and prints one line of plain copy on stderr.

Refusals (all covered by `tests/corpus-review.test.py`)

Attempt	Result
Review or reveal a withdrawn contribution	refused: `was withdrawn and cannot be reviewed, revealed, or trained on`
Reveal before any review is committed	refused: `stays hidden until a review is committed`
Review the same contribution twice	refused; adjudication is the only way to change a decision
Correct a label to the value it already has	refused: a correction must change it
Review with a blank reviewer id	refused
Exclude without a reason	refused
Review a contribution that has no active sample	refused
Queue an active sample whose image is missing	refused: a reviewer cannot judge a file that is not there
Adjudicate your own review	refused: adjudication needs a second reviewer
Adjudicate twice	refused
Adjudicate a contribution with no review	refused

Known limits

One review per contribution, resolved by at most one adjudication. Re-review after an adjudication would need a new record type; it is not implemented.
The tool does not measure or validate a model. It records what a human decided and what a model answered, so the disagreement can be counted later.
It does not train, split, or export a dataset; trainable_records() is the list a trainer may read, not a trainer.

npm run test:review runs the contract suite. Every fixture is the committed synthetic image; no real medical image is read, written, or committed.

5.2 KiB Raw Permalink Blame History