Build contributor-held-out evaluation harness and calibration report (#29) #86

2026-10-25T11:05:52Z

rockachopa commented

2026-10-25 11:05:52 +00:00

Closes #29

What this does

scripts/calibration_report.py is the comparison and release-gate harness the ROADMAP gates were written for. It reads prediction records and refuses anything it cannot honestly score.

Requirement (#29)	Where
macro-F1, per-class precision/recall, confusion	`evaluate()` — class 1–7 rows plus a `truth->predicted` matrix
calibration error	`ece` over ten equal-width confidence bins, weighted by bin share
coverage and precision-at-coverage	selective-classification metrics over accepted suggestions; abstentions split into `no-suggestion` and `low-confidence`
contributor-held-out splits	`assert_contributor_held_out()` — a subject key in two splits is refused
reproducible splits	`expected_split()` re-derives the split from the pseudonymous key with the existing ingestion rule (first byte: <205 train, <230 validation, else test), so a corpus cannot re-label its own splits
compare VLM bootstrap, frozen encoder, specialist checkpoints	`compare()` groups by `model`, fixes one threshold per candidate on validation, then scores that candidate on test

Threshold protocol

select_threshold() maximises precision-at-coverage among the cuts that clear the coverage floor, breaking ties toward wider coverage. It reads the validation split only and refuses test or train input: choosing a threshold on the split the gate is reported on manufactures whatever result the report needs. This is the executable half of #32; the policy numbers remain #32's decision.

Gates and exit codes

Limits are the ROADMAP values and are inclusive: macroF1 >= 0.80, precisionAtCoverage >= 0.90, ece <= 0.05. Exit 0 = GO, 3 = BLOCKED, 2 = refused (malformed, leaky, or unreproducible input — nothing is scored and no report is written).

Classes that abstain away are named, not scored

Macro-F1 averages over classes present in the accepted suggestions. A class whose every photo abstained appears in excludedClasses instead of counting as a silent zero, so abstention can never inflate or deflate the score unnoticed.

Tests

Test-first, vertical slices (RED before each implementation; two cycle-4 guard tests were hold-out checks, noted in the receipt):

python3 tests/calibration-harness.test.py -v — 19 cases: macro-F1 math on a hand-computable case, abstention/coverage, ECE, the split and leak refusals, threshold selection on both sides of the coverage floor, gate verdicts, inclusive limits, CLI exit codes, and fail-closed refusals.
npm test — 219 pass. python3 tests/corpus-manifest.test.py -v — 8 pass.
Browser acceptance: ui, photo, mobile-capture, sleek, manual-fallback, provisional, followup, vision-refusal, worker-readiness, pattern-summary, photo-lifecycle, training-contribution, portability — all pass.
npm run check:syntax, npm run check:diff — pass.

Demonstration path

scripts/record_release_demo.mjs now records the harness itself (the neighbouring corpus scene's pattern): a throwaway run of the real CLI against committed fixtures, then an opaque card drawn from the JSON report the run wrote.

Blocked: the three-candidate fixture set → each candidate fails a different gate.
Passing: the engineered fixture clears every gate — the passing branch is proven, not implied.
Refused: a corpus that leaks one contributor across two splits → exit 2, refusal text on camera.

The recorder throws if any verdict, split label, or card value disagrees with the run, if the card is not fully readable on screen, or if the card omits the measured macro-F1 and the refusal line. The four captions are release gates in DEMO_REQUIRED_CAPTIONS, and were verified painted in a real 377s recording (captionsVerified, 77/77).

Privacy and safety boundary

No model weights, no inference, no provider call, no infrastructure, no production service touched.
No image bytes anywhere: a prediction record is a Bristol label, a confidence, and a pseudonymous key.
The fixtures are invented numbers (tests/fixtures/calibration/README.md says so, and so do the captions): no candidate model has been measured against these gates, and the harness is not evidence about accuracy. The non-diagnostic boundary and the prefill block are unchanged.

Not in this PR

Merging (left for review).
Any claim that the bootstrap VLM clears a gate — it does not, and the fixture set records exactly that.
Calibrated production thresholds (#32) and the reviewer tool (#26).

Closes #29 ## What this does `scripts/calibration_report.py` is the comparison and release-gate harness the ROADMAP gates were written for. It reads prediction records and refuses anything it cannot honestly score. | Requirement (#29) | Where | |---|---| | macro-F1, per-class precision/recall, confusion | `evaluate()` — class 1–7 rows plus a `truth->predicted` matrix | | calibration error | `ece` over ten equal-width confidence bins, weighted by bin share | | coverage and precision-at-coverage | selective-classification metrics over *accepted* suggestions; abstentions split into `no-suggestion` and `low-confidence` | | contributor-held-out splits | `assert_contributor_held_out()` — a subject key in two splits is refused | | reproducible splits | `expected_split()` re-derives the split from the pseudonymous key with the **existing ingestion rule** (first byte: <205 train, <230 validation, else test), so a corpus cannot re-label its own splits | | compare VLM bootstrap, frozen encoder, specialist checkpoints | `compare()` groups by `model`, fixes one threshold per candidate on validation, then scores that candidate on test | ### Threshold protocol `select_threshold()` maximises precision-at-coverage among the cuts that clear the coverage floor, breaking ties toward wider coverage. It reads the **validation split only** and refuses test or train input: choosing a threshold on the split the gate is reported on manufactures whatever result the report needs. This is the executable half of #32; the policy numbers remain #32's decision. ### Gates and exit codes Limits are the ROADMAP values and are inclusive: `macroF1 >= 0.80`, `precisionAtCoverage >= 0.90`, `ece <= 0.05`. Exit 0 = `GO`, 3 = `BLOCKED`, 2 = refused (malformed, leaky, or unreproducible input — nothing is scored and no report is written). ### Classes that abstain away are named, not scored Macro-F1 averages over classes present in the accepted suggestions. A class whose every photo abstained appears in `excludedClasses` instead of counting as a silent zero, so abstention can never inflate or deflate the score unnoticed. ## Tests Test-first, vertical slices (RED before each implementation; two cycle-4 guard tests were hold-out checks, noted in the receipt): - `python3 tests/calibration-harness.test.py -v` — **19 cases**: macro-F1 math on a hand-computable case, abstention/coverage, ECE, the split and leak refusals, threshold selection on both sides of the coverage floor, gate verdicts, inclusive limits, CLI exit codes, and fail-closed refusals. - `npm test` — 219 pass. `python3 tests/corpus-manifest.test.py -v` — 8 pass. - Browser acceptance: ui, photo, mobile-capture, sleek, manual-fallback, provisional, followup, vision-refusal, worker-readiness, pattern-summary, photo-lifecycle, training-contribution, portability — all pass. - `npm run check:syntax`, `npm run check:diff` — pass. ## Demonstration path `scripts/record_release_demo.mjs` now records the harness itself (the neighbouring corpus scene's pattern): a throwaway run of the **real CLI** against committed fixtures, then an opaque card drawn from the JSON report the run wrote. - Blocked: the three-candidate fixture set → each candidate fails a different gate. - Passing: the engineered fixture clears every gate — the passing branch is proven, not implied. - Refused: a corpus that leaks one contributor across two splits → exit 2, refusal text on camera. The recorder throws if any verdict, split label, or card value disagrees with the run, if the card is not fully readable on screen, or if the card omits the measured macro-F1 and the refusal line. The four captions are release gates in `DEMO_REQUIRED_CAPTIONS`, and were verified painted in a real 377s recording (`captionsVerified`, 77/77). ## Privacy and safety boundary - No model weights, no inference, no provider call, no infrastructure, no production service touched. - No image bytes anywhere: a prediction record is a Bristol label, a confidence, and a pseudonymous key. - The fixtures are **invented numbers** (`tests/fixtures/calibration/README.md` says so, and so do the captions): no candidate model has been measured against these gates, and the harness is not evidence about accuracy. The non-diagnostic boundary and the prefill block are unchanged. ## Not in this PR - Merging (left for review). - Any claim that the bootstrap VLM clears a gate — it does not, and the fixture set records exactly that. - Calibrated production thresholds (#32) and the reviewer tool (#26).

rockachopa added 1 commit 2026-10-25 11:05:52 +00:00

feat(eval): score candidates against the release gates on held-out contributors (#29 )

Quality gates / quality (pull_request) Successful in 13m25s

Details

179f8fe6e9

The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no
tool behind them. scripts/calibration_report.py now computes them from prediction records and
refuses input it cannot honestly score:

- a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus
  cannot re-label its own splits; a contributor spanning two splits is refused before scoring
- the threshold is selected on validation only, and selecting on test or train is refused
- coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected
  calibration error are computed on accepted suggestions, with classes that abstained away named
  in excludedClasses instead of being scored as a silent zero
- exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused

The release demo records the real harness against committed synthetic fixtures: a blocked
three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused.
Every number on the card is read back from the report the run wrote, and the captions state that
no model was measured, because a fixture is not evidence about accuracy.

The harness suite is wired into CI, check:syntax, and the release builder's own gates.

rockachopa reviewed

2026-10-25 12:02:03 +00:00

rockachopa left a comment

Reviewed against the merge criteria, not just the author's summary.

Verified independently on timmy/29-evaluation-harness @ 179f8fe:

python3 tests/calibration-harness.test.py -v — 19/19 pass (macro-F1, abstention/coverage, ECE binning, split reproducibility, leak refusal, threshold-on-validation-only, inclusive limits, CLI exit codes 0/2/3).
npm test — 219/219 pass. npm run check:syntax, npm run check:diff — clean.
Diff is 14 files, +1085/-2, scoped to the harness, its fixtures, the release gate wiring, and the demo scene. No secrets, no model weights, no image bytes, no real medical data; fixtures are declared invented in tests/fixtures/calibration/README.md and on camera.
The demo scene drives the real CLI and throws if the card disagrees with the run, so the release video cannot show a number the harness did not produce.

Substantive points I checked rather than took on faith: the split rule re-derives from the pseudonymous key using the existing ingestion byte edges (205/230), threshold selection refuses train/test input, and classes that abstain away are reported in excludedClasses instead of scoring as silent zeros. The known limitation stands and is stated on camera: the fixture scores are invented arithmetic checks and are not a measurement of any model.

Reviewed against the merge criteria, not just the author's summary. Verified independently on `timmy/29-evaluation-harness` @ 179f8fe: - `python3 tests/calibration-harness.test.py -v` — 19/19 pass (macro-F1, abstention/coverage, ECE binning, split reproducibility, leak refusal, threshold-on-validation-only, inclusive limits, CLI exit codes 0/2/3). - `npm test` — 219/219 pass. `npm run check:syntax`, `npm run check:diff` — clean. - Diff is 14 files, +1085/-2, scoped to the harness, its fixtures, the release gate wiring, and the demo scene. No secrets, no model weights, no image bytes, no real medical data; fixtures are declared invented in `tests/fixtures/calibration/README.md` and on camera. - The demo scene drives the real CLI and throws if the card disagrees with the run, so the release video cannot show a number the harness did not produce. Substantive points I checked rather than took on faith: the split rule re-derives from the pseudonymous key using the existing ingestion byte edges (205/230), threshold selection refuses train/test input, and classes that abstain away are reported in `excludedClasses` instead of scoring as silent zeros. The known limitation stands and is stated on camera: the fixture scores are invented arithmetic checks and are not a measurement of any model.

rockachopa merged commit a2293c6e34 into main

2026-10-25 12:02:13 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#86