feat(eval): score candidates against the release gates on held-out contributors (#29)
Quality gates / quality (pull_request) Successful in 13m25s
Quality gates / quality (pull_request) Successful in 13m25s
The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no tool behind them. scripts/calibration_report.py now computes them from prediction records and refuses input it cannot honestly score: - a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus cannot re-label its own splits; a contributor spanning two splits is refused before scoring - the threshold is selected on validation only, and selecting on test or train is refused - coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected calibration error are computed on accepted suggestions, with classes that abstained away named in excludedClasses instead of being scored as a silent zero - exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused The release demo records the real harness against committed synthetic fixtures: a blocked three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused. Every number on the card is read back from the report the run wrote, and the captions state that no model was measured, because a fixture is not evidence about accuracy. The harness suite is wired into CI, check:syntax, and the release builder's own gates.
This commit is contained in:
1 parent
9f69730780
commit
179f8fe6e9
14 files changed
+1085
-2
No files matched your search
@@ -34,6 +34,7 @@ jobs:
|
||||
npm test
|
||||
python3 tests/staging-deploy.test.py -v
|
||||
python3 tests/corpus-manifest.test.py -v
|
||||
python3 tests/calibration-harness.test.py -v
|
||||
- name: Mobile browser acceptance
|
||||
run: |
|
||||
npm start > /tmp/timmy-server.log 2>&1 &
|
||||
|
||||
Reference in new issue
Block a user