timmy
179f8fe6e9
The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no tool behind them. scripts/calibration_report.py now computes them from prediction records and refuses input it cannot honestly score: - a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus cannot re-label its own splits; a contributor spanning two splits is refused before scoring - the threshold is selected on validation only, and selecting on test or train is refused - coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected calibration error are computed on accepted suggestions, with classes that abstained away named in excludedClasses instead of being scored as a silent zero - exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused The release demo records the real harness against committed synthetic fixtures: a blocked three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused. Every number on the card is read back from the report the run wrote, and the captions state that no model was measured, because a fixture is not evidence about accuracy. The harness suite is wired into CI, check:syntax, and the release builder's own gates.
Synthetic calibration fixtures
These files prove the evaluation harness in scripts/calibration_report.py (issue #29) computes the
release gates correctly. They are not measurements. No model produced them, no contributor
supplied them, and no number here says anything about how well Timmy labels a real photo.
Every row is a promise in the same shape the harness accepts:
{"model":"…","subjectKey":"<16 hex>","split":"validation","truth":{"bristolType":4},
"prediction":{"bristolType":4,"confidence":0.97}}
subjectKey values are fictional but obey the ingestion split rule (the first byte of the key picks
the split), so the harness's contributor-held-out check is exercised for real. prediction: null
means the candidate produced no suggestion at all.
| File | What it is for | Harness outcome |
|---|---|---|
predictions.jsonl |
Three candidates with invented scores: a bootstrap that calls almost everything Type 4, a frozen encoder that is precise but badly calibrated, and a specialist that looks strong on validation and slips on test. | BLOCKED — each candidate fails a different gate. |
predictions-go.jsonl |
One engineered candidate that clears every gate, including a low-confidence mistake on validation that the threshold is supposed to cut away. | GO — proves the passing branch and threshold selection, not a result. |
predictions-leaky.jsonl |
One contributor whose photos sit in both validation and test. |
Refused with exit 2 — a leaked corpus is never scored. |
Regenerating
The numbers are hand-chosen so the expected outcome is obvious by inspection; there is no generator
to re-run. If you change a row, update the expectation in tests/calibration-harness.test.py and the
caption in scripts/record_release_demo.mjs together, and read the harness output back before
claiming what it says.
No image, no diagnosis, and no patient data lives in this directory.