Timmy
c5fa2123e7
Quality gates / quality (pull_request) Canceled after 0s
The corpus manifest records what was ingested and whether the person still allows it; it does not record whether the labels are right. This adds the labeling control the data-flow threat model named: scripts/review_corpus.py, a command-line tool with no app screen. - The review queue shows the contributor's proposal and carries no model field at all: blindness is structural, because build_queue() never receives predictions. - reveal() refuses a contribution with no committed review, so the model answer cannot steer the human who labels it. Revealing appends nothing. - A review is one append-only line carrying the proposal it judged; a correction must change it. An adjudication is a second reviewer's line that names the record it resolves; the review it overrules survives untouched. - A withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and the refusal says it was withdrawn rather than that it is already reviewed. - trainable_records() returns only active samples with a human decision that was not an exclusion, so unreviewed proposals are never training data. Tests are written first against every refusal (tests/corpus-review.test.py, 21 cases); the release builder runs the suite and gates four on-camera captions, and the demo now runs the real queue -> review -> reveal -> adjudicate -> withdraw sequence against the committed synthetic sample and draws the evidence card from that output. docs/REVIEWER-TOOL.md is the contract; no model was measured.
94 lines
5.2 KiB
Markdown
94 lines
5.2 KiB
Markdown
# Blind review and adjudication tool
|
|
|
|
Contract for `scripts/review_corpus.py`, the labeling control named in the
|
|
[photo data-flow threat model](PHOTO-DATAFLOW-THREAT-MODEL.md) and the gate in front of the
|
|
[training corpus](CORPUS-MANIFEST.md). Issue [#26](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/26).
|
|
|
|
The tool decides which samples a trainer is allowed to read. It is a command-line tool, not a
|
|
screen in the app: there is no reviewer UI, and nothing here implies one.
|
|
|
|
## The four invariants
|
|
|
|
1. **The model answer is withheld until a human commits.** `build_queue()` takes no predictions at
|
|
all, so the queue cannot leak an answer it never receives. `reveal()` refuses to return anything
|
|
for a contribution with no committed review.
|
|
2. **History is append-only.** A review is a line in `reviews.jsonl`. An adjudication is another
|
|
line that names the record it resolves (`resolvesRecord`, `resolves`). The review being
|
|
overruled is never edited, and no record is deleted.
|
|
3. **A withdrawal beats every other answer.** A revoked contribution cannot be queued, reviewed,
|
|
revealed, adjudicated, or trained on. The refusal says it was withdrawn — not that it is
|
|
already reviewed — because the person's withdrawal is the fact that matters.
|
|
4. **Unreviewed and excluded samples are not training data.** `trainable_records()` returns only
|
|
samples that are still active, carry a human decision, and were not excluded.
|
|
|
|
## Review ledger records
|
|
|
|
One JSON object per line in `<dataset>/reviews.jsonl`:
|
|
|
|
```json
|
|
{"schemaVersion":1,"recordType":"review","contributionId":"...","imageId":"<sha256>","proposed":{"bristolType":4,"color":"brown","quality":"good"},"reviewer":"reviewer-a","decision":"corrected","labels":{"bristolType":5,"color":"green","quality":"fair"},"committedAt":"2026-10-27T09:00:00+00:00"}
|
|
{"schemaVersion":1,"recordType":"adjudication","contributionId":"...","imageId":"<sha256>","resolvesRecord":1,"resolves":{"reviewer":"reviewer-a","decision":"corrected"},"reviewer":"reviewer-b","decision":"upheld","labels":{"bristolType":5,"color":"green","quality":"fair"},"adjudicatedAt":"2026-10-27T10:00:00+00:00"}
|
|
```
|
|
|
|
- `proposed` copies the labels the reviewer judged, so a later manifest change cannot rewrite what
|
|
was reviewed.
|
|
- `decision` is `agreed`, `corrected`, or `excluded` for a review, and `upheld`, `corrected`, or
|
|
`excluded` for an adjudication. An exclusion carries `exclusionReason` and no labels.
|
|
- The ledger never carries `subjectKey`, note text, symptom answers, or image bytes.
|
|
|
|
## A model answer file
|
|
|
|
`reveal` reads predictions only to show what the reviewer could not see. A file may be one JSON
|
|
document or one object per line:
|
|
|
|
```json
|
|
{"model":"<name>","answers":{"<contributionId>":{"bristolType":6,"color":"brown","quality":"good","confidence":0.71}}}
|
|
```
|
|
|
|
Keys may be contribution ids or derived image digests. `reveal` is a read: it appends nothing.
|
|
When no answer file is supplied, `model` is `null` — the tool does not invent an answer.
|
|
|
|
## Usage
|
|
|
|
```bash
|
|
python3 scripts/review_corpus.py queue --dataset /srv/timmy-corpus
|
|
python3 scripts/review_corpus.py review --dataset /srv/timmy-corpus \
|
|
--contribution "$CONTRIBUTION_ID" --reviewer reviewer-a --bristol 5 --color green --quality fair
|
|
python3 scripts/review_corpus.py reveal --dataset /srv/timmy-corpus \
|
|
--contribution "$CONTRIBUTION_ID" --predictions answers.json
|
|
python3 scripts/review_corpus.py adjudicate --dataset /srv/timmy-corpus \
|
|
--contribution "$CONTRIBUTION_ID" --reviewer reviewer-b --upheld
|
|
python3 scripts/review_corpus.py trainable --dataset /srv/timmy-corpus
|
|
```
|
|
|
|
Every mutating command prints the record it appended. A refusal exits `2` and prints one line of
|
|
plain copy on stderr.
|
|
|
|
## Refusals (all covered by `tests/corpus-review.test.py`)
|
|
|
|
| Attempt | Result |
|
|
|---|---|
|
|
| Review or reveal a withdrawn contribution | refused: `was withdrawn and cannot be reviewed, revealed, or trained on` |
|
|
| Reveal before any review is committed | refused: `stays hidden until a review is committed` |
|
|
| Review the same contribution twice | refused; adjudication is the only way to change a decision |
|
|
| Correct a label to the value it already has | refused: a correction must change it |
|
|
| Review with a blank reviewer id | refused |
|
|
| Exclude without a reason | refused |
|
|
| Review a contribution that has no active sample | refused |
|
|
| Queue an active sample whose image is missing | refused: a reviewer cannot judge a file that is not there |
|
|
| Adjudicate your own review | refused: adjudication needs a second reviewer |
|
|
| Adjudicate twice | refused |
|
|
| Adjudicate a contribution with no review | refused |
|
|
|
|
## Known limits
|
|
|
|
- One review per contribution, resolved by at most one adjudication. Re-review after an
|
|
adjudication would need a new record type; it is not implemented.
|
|
- The tool does not measure or validate a model. It records what a human decided and what a model
|
|
answered, so the disagreement can be counted later.
|
|
- It does not train, split, or export a dataset; `trainable_records()` is the list a trainer may
|
|
read, not a trainer.
|
|
|
|
`npm run test:review` runs the contract suite. Every fixture is the committed synthetic image; no
|
|
real medical image is read, written, or committed.
|