timmy
179f8fe6e9
Quality gates / quality (pull_request) Successful in 13m25s
The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no tool behind them. scripts/calibration_report.py now computes them from prediction records and refuses input it cannot honestly score: - a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus cannot re-label its own splits; a contributor spanning two splits is refused before scoring - the threshold is selected on validation only, and selecting on test or train is refused - coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected calibration error are computed on accepted suggestions, with classes that abstained away named in excludedClasses instead of being scored as a silent zero - exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused The release demo records the real harness against committed synthetic fixtures: a blocked three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused. Every number on the card is read back from the report the run wrote, and the captions state that no model was measured, because a fixture is not evidence about accuracy. The harness suite is wired into CI, check:syntax, and the release builder's own gates.
43 lines
6.3 KiB
JSON
43 lines
6.3 KiB
JSON
{"model":"vlm-bootstrap","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":4,"confidence":0.55}}
|
|
{"model":"vlm-bootstrap","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":4,"confidence":0.52}}
|
|
{"model":"vlm-bootstrap","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.61}}
|
|
{"model":"vlm-bootstrap","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.72}}
|
|
{"model":"vlm-bootstrap","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":4,"confidence":0.64}}
|
|
{"model":"vlm-bootstrap","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":4,"confidence":0.58}}
|
|
{"model":"vlm-bootstrap","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":4,"confidence":0.53}}
|
|
{"model":"vlm-bootstrap","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":4,"confidence":0.57}}
|
|
{"model":"vlm-bootstrap","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":4,"confidence":0.54}}
|
|
{"model":"vlm-bootstrap","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.63}}
|
|
{"model":"vlm-bootstrap","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.7}}
|
|
{"model":"vlm-bootstrap","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":4,"confidence":0.62}}
|
|
{"model":"vlm-bootstrap","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":4,"confidence":0.59}}
|
|
{"model":"vlm-bootstrap","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":4,"confidence":0.51}}
|
|
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.62}}
|
|
{"model":"frozen-encoder","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.58}}
|
|
{"model":"frozen-encoder","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.66}}
|
|
{"model":"frozen-encoder","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.71}}
|
|
{"model":"frozen-encoder","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.57}}
|
|
{"model":"frozen-encoder","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.64}}
|
|
{"model":"frozen-encoder","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":6,"confidence":0.55}}
|
|
{"model":"frozen-encoder","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.6}}
|
|
{"model":"frozen-encoder","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.59}}
|
|
{"model":"frozen-encoder","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.63}}
|
|
{"model":"frozen-encoder","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.69}}
|
|
{"model":"frozen-encoder","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":6,"confidence":0.56}}
|
|
{"model":"frozen-encoder","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.61}}
|
|
{"model":"frozen-encoder","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.54}}
|
|
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.88}}
|
|
{"model":"specialist-candidate","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.86}}
|
|
{"model":"specialist-candidate","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.9}}
|
|
{"model":"specialist-candidate","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.94}}
|
|
{"model":"specialist-candidate","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.87}}
|
|
{"model":"specialist-candidate","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.91}}
|
|
{"model":"specialist-candidate","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.85}}
|
|
{"model":"specialist-candidate","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.86}}
|
|
{"model":"specialist-candidate","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.84}}
|
|
{"model":"specialist-candidate","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.89}}
|
|
{"model":"specialist-candidate","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.92}}
|
|
{"model":"specialist-candidate","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.88}}
|
|
{"model":"specialist-candidate","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":3,"confidence":0.83}}
|
|
{"model":"specialist-candidate","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.87}}
|