# Blind review and adjudication tool

Contract for `scripts/review_corpus.py`, the labeling control named in the
[photo data-flow threat model](PHOTO-DATAFLOW-THREAT-MODEL.md) and the gate in front of the
[training corpus](CORPUS-MANIFEST.md). Issue [#26](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/26).

The tool decides which samples a trainer is allowed to read. It is a command-line tool, not a
screen in the app: there is no reviewer UI, and nothing here implies one.

## The four invariants

1. **The model answer is withheld until a human commits.** `build_queue()` takes no predictions at
   all, so the queue cannot leak an answer it never receives. `reveal()` refuses to return anything
   for a contribution with no committed review.
2. **History is append-only.** A review is a line in `reviews.jsonl`. An adjudication is another
   line that names the record it resolves (`resolvesRecord`, `resolves`). The review being
   overruled is never edited, and no record is deleted.
3. **A withdrawal beats every other answer.** A revoked contribution cannot be queued, reviewed,
   revealed, adjudicated, or trained on. The refusal says it was withdrawn — not that it is
   already reviewed — because the person's withdrawal is the fact that matters.
4. **Unreviewed and excluded samples are not training data.** `trainable_records()` returns only
   samples that are still active, carry a human decision, and were not excluded.

## Review ledger records

One JSON object per line in `<dataset>/reviews.jsonl`:

```json
{"schemaVersion":1,"recordType":"review","contributionId":"...","imageId":"<sha256>","proposed":{"bristolType":4,"color":"brown","quality":"good"},"reviewer":"reviewer-a","decision":"corrected","labels":{"bristolType":5,"color":"green","quality":"fair"},"committedAt":"2026-10-27T09:00:00+00:00"}
{"schemaVersion":1,"recordType":"adjudication","contributionId":"...","imageId":"<sha256>","resolvesRecord":1,"resolves":{"reviewer":"reviewer-a","decision":"corrected"},"reviewer":"reviewer-b","decision":"upheld","labels":{"bristolType":5,"color":"green","quality":"fair"},"adjudicatedAt":"2026-10-27T10:00:00+00:00"}
```

- `proposed` copies the labels the reviewer judged, so a later manifest change cannot rewrite what
  was reviewed.
- `decision` is `agreed`, `corrected`, or `excluded` for a review, and `upheld`, `corrected`, or
  `excluded` for an adjudication. An exclusion carries `exclusionReason` and no labels.
- The ledger never carries `subjectKey`, note text, symptom answers, or image bytes.

## A model answer file

`reveal` reads predictions only to show what the reviewer could not see. A file may be one JSON
document or one object per line:

```json
{"model":"<name>","answers":{"<contributionId>":{"bristolType":6,"color":"brown","quality":"good","confidence":0.71}}}
```

Keys may be contribution ids or derived image digests. `reveal` is a read: it appends nothing.
When no answer file is supplied, `model` is `null` — the tool does not invent an answer.

## Usage

```bash
python3 scripts/review_corpus.py queue --dataset /srv/timmy-corpus
python3 scripts/review_corpus.py review --dataset /srv/timmy-corpus \
  --contribution "$CONTRIBUTION_ID" --reviewer reviewer-a --bristol 5 --color green --quality fair
python3 scripts/review_corpus.py reveal --dataset /srv/timmy-corpus \
  --contribution "$CONTRIBUTION_ID" --predictions answers.json
python3 scripts/review_corpus.py adjudicate --dataset /srv/timmy-corpus \
  --contribution "$CONTRIBUTION_ID" --reviewer reviewer-b --upheld
python3 scripts/review_corpus.py trainable --dataset /srv/timmy-corpus
```

Every mutating command prints the record it appended. A refusal exits `2` and prints one line of
plain copy on stderr.

## Refusals (all covered by `tests/corpus-review.test.py`)

| Attempt | Result |
|---|---|
| Review or reveal a withdrawn contribution | refused: `was withdrawn and cannot be reviewed, revealed, or trained on` |
| Reveal before any review is committed | refused: `stays hidden until a review is committed` |
| Review the same contribution twice | refused; adjudication is the only way to change a decision |
| Correct a label to the value it already has | refused: a correction must change it |
| Review with a blank reviewer id | refused |
| Exclude without a reason | refused |
| Review a contribution that has no active sample | refused |
| Queue an active sample whose image is missing | refused: a reviewer cannot judge a file that is not there |
| Adjudicate your own review | refused: adjudication needs a second reviewer |
| Adjudicate twice | refused |
| Adjudicate a contribution with no review | refused |

## Known limits

- One review per contribution, resolved by at most one adjudication. Re-review after an
  adjudication would need a new record type; it is not implemented.
- The tool does not measure or validate a model. It records what a human decided and what a model
  answered, so the disagreement can be counted later.
- It does not train, split, or export a dataset; `trainable_records()` is the list a trainer may
  read, not a trainer.

`npm run test:review` runs the contract suite. Every fixture is the committed synthetic image; no
real medical image is read, written, or committed.