feat(eval): score candidates against the release gates on held-out contributors (#29)
Quality gates / quality (pull_request) Successful in 13m25s

The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no
tool behind them. scripts/calibration_report.py now computes them from prediction records and
refuses input it cannot honestly score:

- a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus
  cannot re-label its own splits; a contributor spanning two splits is refused before scoring
- the threshold is selected on validation only, and selecting on test or train is refused
- coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected
  calibration error are computed on accepted suggestions, with classes that abstained away named
  in excludedClasses instead of being scored as a silent zero
- exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused

The release demo records the real harness against committed synthetic fixtures: a blocked
three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused.
Every number on the card is read back from the report the run wrote, and the captions state that
no model was measured, because a fixture is not evidence about accuracy.

The harness suite is wired into CI, check:syntax, and the release builder's own gates.
This commit is contained in:
timmy committed 2026-10-25 11:05:28 +00:00
1 parent 9f69730780
commit 179f8fe6e9
14 files changed
+1085 -2

No files matched your search

+1
View File
@@ -34,6 +34,7 @@ jobs:
npm test
python3 tests/staging-deploy.test.py -v
python3 tests/corpus-manifest.test.py -v
python3 tests/calibration-harness.test.py -v
- name: Mobile browser acceptance
run: |
npm start > /tmp/timmy-server.log 2>&1 &