Blind reviewer and adjudication tool for corpus labels (#26) #88

2026-10-27T11:44:04Z

rockachopa commented

2026-10-27 11:44:04 +00:00

Closes #26

What this adds

scripts/review_corpus.py — the labeling control the photo data-flow threat model names, as a command-line tool with no app screen. The corpus manifest says what was ingested and whether the person still allows it; this ledger says whether a human agreed with the label, and what a model answered afterwards.

Acceptance criteria

Review is blind to the model answer until the human commits. build_queue() takes no predictions at all — blindness is structural, not a convention. reveal() refuses with stays hidden until a review is committed when no review exists, and appends nothing when it is allowed.
Disagreement and adjudication are preserved. A review is one append-only line carrying the proposal it judged plus the human decision. An adjudication is a second reviewer's line that names the record it resolves (resolvesRecord, resolves); the review it overrules is still in the ledger, byte for byte. A reviewer cannot adjudicate their own review, and a contribution adjudicates once.
The tool cannot display withdrawn contributions. active_records() gates every path: a revoked contribution is absent from the queue, and review, reveal, adjudicate, and trainable all refuse with was withdrawn and cannot be reviewed, revealed, or trained on. trainable_records() additionally drops unreviewed and excluded samples, so unreviewed proposals are never training data.

TDD

tests/corpus-review.test.py (21 cases) was written first and watched fail before the module existed, then implemented one behaviour slice at a time (queue → commit → reveal → adjudicate → trainable → CLI). Two real defects were found by tests rather than by review:

A withdrawn contribution reported already reviewed because the duplicate check ran before the lifecycle check. The withdrawal now outranks every other refusal.
load_predictions read a pretty-printed answer file line by line and crashed. A document is now parsed whole; JSONL is the fallback.

Release evidence

npm test — 227 pass. python3 tests/corpus-review.test.py — 21 pass. node --test tests/release-demo.test.js — 24 pass. npm run check:syntax, npm run check:diff clean.
The release builder now runs the review suite and gates four captions, including The queue shows the proposal and hides the model answer; revealing it is refused until a reviewer commits.
scripts/record_release_demo.mjs runs the real queue → reveal (refused) → review → reveal → adjudicate → trainable → revoke sequence against the committed synthetic sample and draws a REVIEW LEDGER card from that output. Verified by a local recording: the card paints opaque and unclipped, and OCR of the frame at ~296s reads REVIEW LEDGER / 1 pending · proposal Type 4 · brown · good / model answer withheld. The answer file on screen is labelled synthetic answer file · no model was measured.
No app behaviour changed; the three CLI evidence cards now share one renderer (evidenceCard) instead of three copies of the same ~40 lines.

Boundary

No medical boundary moved, no model weights, no real medical images, no infrastructure purchased, no inference exposed. The tool records decisions; it does not train, split, or score. Docs: docs/REVIEWER-TOOL.md.

Not merged

Left open for review as instructed.

Closes #26 ## What this adds `scripts/review_corpus.py` — the labeling control the photo data-flow threat model names, as a command-line tool with no app screen. The corpus manifest says what was ingested and whether the person still allows it; this ledger says whether a human agreed with the label, and what a model answered afterwards. ## Acceptance criteria - [x] **Review is blind to the model answer until the human commits.** `build_queue()` takes no predictions at all — blindness is structural, not a convention. `reveal()` refuses with `stays hidden until a review is committed` when no review exists, and appends nothing when it is allowed. - [x] **Disagreement and adjudication are preserved.** A review is one append-only line carrying the proposal it judged plus the human decision. An adjudication is a second reviewer's line that names the record it resolves (`resolvesRecord`, `resolves`); the review it overrules is still in the ledger, byte for byte. A reviewer cannot adjudicate their own review, and a contribution adjudicates once. - [x] **The tool cannot display withdrawn contributions.** `active_records()` gates every path: a revoked contribution is absent from the queue, and review, reveal, adjudicate, and trainable all refuse with `was withdrawn and cannot be reviewed, revealed, or trained on`. `trainable_records()` additionally drops unreviewed and excluded samples, so unreviewed proposals are never training data. ## TDD `tests/corpus-review.test.py` (21 cases) was written first and watched fail before the module existed, then implemented one behaviour slice at a time (queue → commit → reveal → adjudicate → trainable → CLI). Two real defects were found by tests rather than by review: 1. A withdrawn contribution reported `already reviewed` because the duplicate check ran before the lifecycle check. The withdrawal now outranks every other refusal. 2. `load_predictions` read a pretty-printed answer file line by line and crashed. A document is now parsed whole; JSONL is the fallback. ## Release evidence - `npm test` — 227 pass. `python3 tests/corpus-review.test.py` — 21 pass. `node --test tests/release-demo.test.js` — 24 pass. `npm run check:syntax`, `npm run check:diff` clean. - The release builder now runs the review suite and gates four captions, including `The queue shows the proposal and hides the model answer; revealing it is refused until a reviewer commits`. - `scripts/record_release_demo.mjs` runs the real `queue → reveal (refused) → review → reveal → adjudicate → trainable → revoke` sequence against the committed synthetic sample and draws a `REVIEW LEDGER` card from that output. Verified by a local recording: the card paints opaque and unclipped, and OCR of the frame at ~296s reads `REVIEW LEDGER / 1 pending · proposal Type 4 · brown · good / model answer withheld`. The answer file on screen is labelled `synthetic answer file · no model was measured`. - No app behaviour changed; the three CLI evidence cards now share one renderer (`evidenceCard`) instead of three copies of the same ~40 lines. ## Boundary No medical boundary moved, no model weights, no real medical images, no infrastructure purchased, no inference exposed. The tool records decisions; it does not train, split, or score. Docs: `docs/REVIEWER-TOOL.md`. ## Not merged Left open for review as instructed.

rockachopa added 1 commit 2026-10-27 11:44:04 +00:00

feat(review): blind reviewer and adjudication tool with an append-only label ledger (#26 )

Quality gates / quality (pull_request) Canceled after 0s

Details

c5fa2123e7

The corpus manifest records what was ingested and whether the person still allows it; it does not
record whether the labels are right. This adds the labeling control the data-flow threat model
named: scripts/review_corpus.py, a command-line tool with no app screen.

- The review queue shows the contributor's proposal and carries no model field at all: blindness is
structural, because build_queue() never receives predictions.
- reveal() refuses a contribution with no committed review, so the model answer cannot steer the
human who labels it. Revealing appends nothing.
- A review is one append-only line carrying the proposal it judged; a correction must change it. An
adjudication is a second reviewer's line that names the record it resolves; the review it
overrules survives untouched.
- A withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and the
refusal says it was withdrawn rather than that it is already reviewed.
- trainable_records() returns only active samples with a human decision that was not an exclusion,
so unreviewed proposals are never training data.

Tests are written first against every refusal (tests/corpus-review.test.py, 21 cases); the release
builder runs the suite and gates four on-camera captions, and the demo now runs the real queue ->
review -> reveal -> adjudicate -> withdraw sequence against the committed synthetic sample and draws
the evidence card from that output. docs/REVIEWER-TOOL.md is the contract; no model was measured.

timmy added 1 commit 2026-10-27 12:05:51 +00:00

chore(review): drop the trailing blank line that fails check:diff

Quality gates / quality (pull_request) Canceled after 0s

Details

d54665dbd9

Release builder runs `git diff --check`; scripts/review_corpus.py ended with an extra newline, so the gate failed on the PR head. Whitespace only; no behaviour change.

timmy merged commit b191d0427d into main

2026-10-27 12:06:00 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#88