feat(review): blind reviewer and adjudication tool with an append-only label ledger (#26)
Quality gates / quality (pull_request) Canceled after 0s
Quality gates / quality (pull_request) Canceled after 0s
The corpus manifest records what was ingested and whether the person still allows it; it does not record whether the labels are right. This adds the labeling control the data-flow threat model named: scripts/review_corpus.py, a command-line tool with no app screen. - The review queue shows the contributor's proposal and carries no model field at all: blindness is structural, because build_queue() never receives predictions. - reveal() refuses a contribution with no committed review, so the model answer cannot steer the human who labels it. Revealing appends nothing. - A review is one append-only line carrying the proposal it judged; a correction must change it. An adjudication is a second reviewer's line that names the record it resolves; the review it overrules survives untouched. - A withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and the refusal says it was withdrawn rather than that it is already reviewed. - trainable_records() returns only active samples with a human decision that was not an exclusion, so unreviewed proposals are never training data. Tests are written first against every refusal (tests/corpus-review.test.py, 21 cases); the release builder runs the suite and gates four on-camera captions, and the demo now runs the real queue -> review -> reveal -> adjudicate -> withdraw sequence against the committed synthetic sample and draws the evidence card from that output. docs/REVIEWER-TOOL.md is the contract; no model was measured.
This commit is contained in:
1 parent
2533f83af3
commit
c5fa2123e7
10 files changed
+1296
-109
No files matched your search
@@ -68,6 +68,10 @@ DEMO_REQUIRED_CAPTIONS = (
|
||||
"An older backup is upgraded on import, and the notice says which version the file came from",
|
||||
"A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched",
|
||||
"A restored log still says it began as an AI suggestion you corrected — provenance is part of the record",
|
||||
"The review tool is a command line, not a screen in the app — this runs it against the committed synthetic sample",
|
||||
"The queue shows the proposal and hides the model answer; revealing it is refused until a reviewer commits",
|
||||
"The model said Type 6 and the reviewer said Type 5: the disagreement is recorded, not smoothed over",
|
||||
"Withdrawing the contribution empties the queue and the training set, and the tool refuses to review it at all",
|
||||
)
|
||||
|
||||
|
||||
@@ -166,6 +170,9 @@ def main() -> int:
|
||||
run(["python3", "scripts/build_vision_corpus.py", "--verify"], tree)
|
||||
# Issue #25's corpus pipeline is a CLI, so its own suite is a release gate, not just CI.
|
||||
run(["python3", "tests/corpus-manifest.test.py", "-v"], tree)
|
||||
# Issue #26's review tool decides which samples a trainer may read, so the blindness and
|
||||
# withdrawal rules it enforces are a release gate too.
|
||||
run(["python3", "tests/corpus-review.test.py", "-v"], tree)
|
||||
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
|
||||
# arithmetic are exercised before this release is allowed to exist.
|
||||
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
|
||||
@@ -261,7 +268,7 @@ def main() -> int:
|
||||
run(["bash", "-n", "deploy/worker/entrypoint.sh"], tree)
|
||||
run(["bash", "-n", "deploy/worker/build_image.sh"], tree)
|
||||
run(["bash", "-n", "deploy/worker/run_worker_container.sh"], tree)
|
||||
run(["python3", "-m", "py_compile", "scripts/ingest_training_photo.py", "scripts/build_release.py", "scripts/build_vision_corpus.py", "scripts/corpus_manifest.py"], tree)
|
||||
run(["python3", "-m", "py_compile", "scripts/ingest_training_photo.py", "scripts/build_release.py", "scripts/build_vision_corpus.py", "scripts/corpus_manifest.py", "scripts/review_corpus.py"], tree)
|
||||
run(["git", "diff", "--check", commit], tree)
|
||||
|
||||
names = tracked_files(tree)
|
||||
@@ -297,6 +304,8 @@ def main() -> int:
|
||||
"- Training contribution is a separate opt-in (issue #24): a journal save records no contribution, the consent version and the receipt are on screen before the control can be pressed, the receipt is queued on this device only (this build transmits nothing) and carries no note text, symptom answers, or photo bytes, and a withdrawal keeps a tombstone naming every derived artifact by contribution id\n"
|
||||
"- Auditable corpus pipeline (issue #25): every ingested training sample now records its consent version and timestamp, source kind and the raw-input digest (the caller's file path is never stored), both the raw and derived hashes, the derivation pipeline, the human review status, the split, and its lifecycle state in an append-only manifest; withdrawing a contribution deletes the derived image and any raw pre-derivation copy from the active dataset, keeps the original record as its own history, and appends a tombstone that names every removed artifact by contribution id and digest while dropping labels and the subject key; illegal lifecycle moves (re-ingesting a contribution, re-labelling a derived image, revoking twice, revoking an unknown contribution, reviving a revoked one, revoking without a reason) are refused before any byte is written\n"
|
||||
"- Scope note: the corpus pipeline is a command-line tool. There is no corpus screen in the app, and this release does not imply one — the recorded demo runs the real ingest and revoke commands against the committed synthetic fixture and shows their actual output\n"
|
||||
"- Blind reviewer and adjudication tool (issue #26): the review queue shows the proposal and withholds every model field, revealing a model answer is refused until a human has committed their own labels, a correction must actually change the proposal, the ledger is append-only, and a second reader's adjudication is appended beside the review it resolves rather than replacing it; a withdrawn contribution cannot be queued, reviewed, revealed, adjudicated, or trained on, and unreviewed or excluded samples are never training data\n"
|
||||
"- Scope note: the review tool is a command line, not a screen in the app. There is no reviewer screen, and this release does not imply one — the recorded demo runs the real queue, review, reveal, adjudicate, and trainable commands against the committed synthetic sample and shows their actual output, with a synthetic answer file that no model produced\n"
|
||||
"- Known limitation: open-weight VLM acceptance is proven, Bristol accuracy is not clinically validated\n"
|
||||
"- Package excludes model weights, medical-image corpora, raw research pages, generated screenshots, and video production assets\n"
|
||||
)
|
||||
@@ -341,7 +350,7 @@ def main() -> int:
|
||||
"codec": "h264",
|
||||
"pixel_format": "yuv420p",
|
||||
"recorded_from_working_app": True,
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything",
|
||||
"fixture_data": "synthetic-Type-4, deterministic vision suggestion reviewed, confirmed, and corrected to Type 2, user-reported hard-stool follow-up with non-diagnostic self-care and a confirmed black/tarry red flag, the CC0 synthetic vision corpus replaying a non-stool refusal, an unreadable-fields abstention, a confident reply about a poor-quality photo abstaining, an out-of-range fail-closed reply, and a blank-plus-out-of-range fail-closed reply, offline vision status, fictional multi-day sample logs driving the explainable pattern summary and its citation sheet, a sparse fictional window showing the insufficient-data copy, a grounded local chat answer citing each counted log and opening the cited record, a refusal that borrows no journal evidence, and a deterministic Hermes chat reply whose positional refs render as sources, plus the opt-in training contribution path where nothing is queued after ordinary saves, the consent version and receipt precede the confirm control, one receipt is queued on the device only, and withdrawal leaves a tombstone naming each derived artifact, plus a versioned backup that carries the logs and the withdrawn receipt, an older version 1 file upgraded with the version named on screen, a file from a newer Timmy refused without changing the ledger, and a restored log still showing it began as a corrected AI suggestion, plus the corpus audit card, where the real ingest and revoke commands run against the committed synthetic fixture and the card shows their actual output: the provenance record with consent, source digest, both hashes, derivation, review and lifecycle; the derived image deleted from disk; the original record kept as history beside a tombstone that drops labels and the subject key; zero active samples for a trainer; and a second revocation refused without appending anything, plus the review ledger card, where the real review commands run against the committed synthetic sample and the card shows their actual output: one pending proposal with the model answer withheld, a refused reveal before commitment, a committed Type 5 correction beside the Type 4 proposal, the revealed synthetic Type 6 answer recorded as a three-field disagreement, a second reader's adjudication pointing at the review it resolved, one trainable sample, and a withdrawal that empties the queue and the training set and makes review refused",
|
||||
},
|
||||
"gates": {
|
||||
"unit_security": "passed",
|
||||
@@ -359,6 +368,7 @@ def main() -> int:
|
||||
"grounded_chat_citations": "passed",
|
||||
"training_contribution_consent": "passed",
|
||||
"corpus_manifest_provenance": "passed",
|
||||
"reviewer_adjudication_blind": "passed",
|
||||
"calibration_harness_gates": "passed",
|
||||
"portable_backup_restore": "passed",
|
||||
"dependency_audit_high": "passed",
|
||||
|
||||
Reference in new issue
Block a user