feat(eval): score candidates against the release gates on held-out contributors (#29)
Quality gates / quality (pull_request) Successful in 13m25s
Quality gates / quality (pull_request) Successful in 13m25s
The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no tool behind them. scripts/calibration_report.py now computes them from prediction records and refuses input it cannot honestly score: - a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus cannot re-label its own splits; a contributor spanning two splits is refused before scoring - the threshold is selected on validation only, and selecting on test or train is refused - coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected calibration error are computed on accepted suggestions, with classes that abstained away named in excludedClasses instead of being scored as a silent zero - exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused The release demo records the real harness against committed synthetic fixtures: a blocked three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused. Every number on the card is read back from the report the run wrote, and the captions state that no model was measured, because a fixture is not evidence about accuracy. The harness suite is wired into CI, check:syntax, and the release builder's own gates.
This commit is contained in:
timmy committed
1 parent
9f69730780
commit
179f8fe6e9
14 files changed
+1085
-2
No files matched your search
@@ -34,6 +34,7 @@ jobs:
|
|||||||
npm test
|
npm test
|
||||||
python3 tests/staging-deploy.test.py -v
|
python3 tests/staging-deploy.test.py -v
|
||||||
python3 tests/corpus-manifest.test.py -v
|
python3 tests/corpus-manifest.test.py -v
|
||||||
|
python3 tests/calibration-harness.test.py -v
|
||||||
- name: Mobile browser acceptance
|
- name: Mobile browser acceptance
|
||||||
run: |
|
run: |
|
||||||
npm start > /tmp/timmy-server.log 2>&1 &
|
npm start > /tmp/timmy-server.log 2>&1 &
|
||||||
|
|||||||
@@ -100,6 +100,7 @@ npm run test:manual-fallback # vision-offline manual path + in-session draft re
|
|||||||
npm run test:vision-refusal # corpus refusals through a real server and a real browser
|
npm run test:vision-refusal # corpus refusals through a real server and a real browser
|
||||||
npm run test:grounded-chat # chat answers cite the ledger, or refuse to answer
|
npm run test:grounded-chat # chat answers cite the ledger, or refuse to answer
|
||||||
npm run test:photo-lifecycle # photo presence, explicit reveal, remove-photo-keeps-log, delete-all
|
npm run test:photo-lifecycle # photo presence, explicit reveal, remove-photo-keeps-log, delete-all
|
||||||
|
npm run test:calibration # the release-gate harness: gates, threshold protocol, refused corpora
|
||||||
npm audit --audit-level=high
|
npm audit --audit-level=high
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -132,6 +133,8 @@ The positive photo-prefill interface is covered by a deterministic provider fixt
|
|||||||
|
|
||||||
Timmy does not diagnose disease, identify bleeding, infer pain/urgency, recommend treatment, or clear foods. The selected VLM is a bootstrap worker, not the final classifier. See [research/SELF-HOSTED-STOOL-VISION.md](research/SELF-HOSTED-STOOL-VISION.md) for the measured receipts, hardware gate, consented-data pipeline, specialist-classifier plan, and sources.
|
Timmy does not diagnose disease, identify bleeding, infer pain/urgency, recommend treatment, or clear foods. The selected VLM is a bootstrap worker, not the final classifier. See [research/SELF-HOSTED-STOOL-VISION.md](research/SELF-HOSTED-STOOL-VISION.md) for the measured receipts, hardware gate, consented-data pipeline, specialist-classifier plan, and sources.
|
||||||
|
|
||||||
|
The calibration harness ([scripts/calibration_report.py](scripts/calibration_report.py), [contract](docs/EVALUATION-HARNESS.md)) computes the ROADMAP release gates — macro-F1, precision-at-coverage, and calibration error — from prediction files, chooses a threshold on validation only, and refuses a corpus that leaks one contributor across two splits. Its committed fixtures are invented numbers that exercise the arithmetic: **no candidate model has been measured against these gates yet, and the harness itself is not evidence about accuracy.**
|
||||||
|
|
||||||
## Architecture
|
## Architecture
|
||||||
|
|
||||||
```text
|
```text
|
||||||
|
|||||||
@@ -0,0 +1,113 @@
|
|||||||
|
# Evaluation harness contract
|
||||||
|
|
||||||
|
Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
|
||||||
|
precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
|
||||||
|
splits that are contributor-held-out and reproducible, comparing candidate models.
|
||||||
|
|
||||||
|
The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
|
||||||
|
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
|
||||||
|
|
||||||
|
**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
|
||||||
|
labels, predictions, and confidences.
|
||||||
|
|
||||||
|
## Input
|
||||||
|
|
||||||
|
One prediction record per line (JSONL), one file or many:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
|
||||||
|
"truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Rule |
|
||||||
|
|---|---|
|
||||||
|
| `model` | Non-empty. Candidates are grouped by this value, so one file may carry several. |
|
||||||
|
| `subjectKey` | The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account. |
|
||||||
|
| `split` | `train`, `validation`, or `test`. |
|
||||||
|
| `truth.bristolType` | Whole number 1–7. |
|
||||||
|
| `prediction` | `{bristolType: 1–7, confidence: 0–1}`, or `null` when the candidate offered nothing. |
|
||||||
|
|
||||||
|
A record that breaks any row above is refused (exit 2) rather than scored.
|
||||||
|
|
||||||
|
## Splits are contributor-held-out and reproducible
|
||||||
|
|
||||||
|
The split rule is the ingestion rule from `scripts/ingest_training_photo.py`: the first byte of the
|
||||||
|
subject key picks the bucket — below `205` is `train`, below `230` is `validation`, otherwise `test`.
|
||||||
|
Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the
|
||||||
|
manifest, and a corpus cannot quietly re-label itself.
|
||||||
|
|
||||||
|
`assert_contributor_held_out()` refuses:
|
||||||
|
|
||||||
|
- a `split` that does not match the one the subject key derives (not reproducible), and
|
||||||
|
- one subject key appearing in more than one split (contributor leak) — a held-out score where the
|
||||||
|
model has already seen that person is not held out.
|
||||||
|
|
||||||
|
Several photos from one contributor inside one split are fine; that is the normal case.
|
||||||
|
|
||||||
|
## Metrics
|
||||||
|
|
||||||
|
The harness is a *selective* classifier evaluation. A suggestion is **accepted** when it exists and
|
||||||
|
its confidence is at or above the threshold; everything else is an abstention. Two abstention
|
||||||
|
reasons are counted separately: `no-suggestion` and `low-confidence`.
|
||||||
|
|
||||||
|
| Metric | Definition |
|
||||||
|
|---|---|
|
||||||
|
| `coverage` | accepted / total in the scored split. |
|
||||||
|
| `precisionAtCoverage` | correct / accepted. The gate metric. |
|
||||||
|
| `perClass[label]` | `precision`, `recall`, `f1`, `support`, `predicted` over accepted suggestions. |
|
||||||
|
| `macroF1` | Mean of `perClass[*].f1` over the classes present in **accepted** truth. A class whose every photo abstained cannot be scored, so it is named in `excludedClasses` instead of being counted as a silent zero. |
|
||||||
|
| `confusion` | `"truth->predicted": count` over accepted suggestions. |
|
||||||
|
| `ece` | Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share. |
|
||||||
|
|
||||||
|
Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated
|
||||||
|
by abstaining out of its worst bins.
|
||||||
|
|
||||||
|
## Threshold protocol
|
||||||
|
|
||||||
|
`select_threshold()` takes the coverage floor and returns the threshold with the highest
|
||||||
|
precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage.
|
||||||
|
It reads the **validation split only** and raises otherwise: choosing a threshold on the data the gate
|
||||||
|
is measured on manufactures whatever precision the report needs.
|
||||||
|
|
||||||
|
`compare()` therefore fixes one threshold per candidate on validation, scores that candidate on test,
|
||||||
|
and never looks back at validation scores.
|
||||||
|
|
||||||
|
## Gates
|
||||||
|
|
||||||
|
Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.
|
||||||
|
|
||||||
|
| Gate | Limit |
|
||||||
|
|---|---|
|
||||||
|
| `macroF1` | ≥ 0.80 |
|
||||||
|
| `precisionAtCoverage` | ≥ 0.90 |
|
||||||
|
| `ece` | ≤ 0.05 |
|
||||||
|
|
||||||
|
A candidate must clear all three on its held-out split. `verdict` is `GO` only when at least one
|
||||||
|
candidate does; `recommended` is the passing candidate with the highest macro-F1 (then precision).
|
||||||
|
Otherwise the verdict is `BLOCKED` and no candidate is recommended.
|
||||||
|
|
||||||
|
## Run it
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 scripts/calibration_report.py \
|
||||||
|
--predictions tests/fixtures/calibration/predictions.jsonl \
|
||||||
|
--min-coverage 0.5 --json /tmp/calibration-report.json
|
||||||
|
|
||||||
|
npm run test:calibration # the harness contract suite
|
||||||
|
```
|
||||||
|
|
||||||
|
| Exit code | Meaning |
|
||||||
|
|---|---|
|
||||||
|
| 0 | `GO` — a candidate cleared every gate. |
|
||||||
|
| 3 | `BLOCKED` — the release gate refuses every candidate. |
|
||||||
|
| 2 | Refused — malformed, leaky, or unreproducible input. Nothing was scored or written. |
|
||||||
|
|
||||||
|
`--json PATH` writes the full report (per-class rows, confusion matrix, calibration bins, gate detail).
|
||||||
|
The report is only written after a successful run.
|
||||||
|
|
||||||
|
## Fixtures
|
||||||
|
|
||||||
|
[`tests/fixtures/calibration/`](../tests/fixtures/calibration/README.md) holds synthetic predictions
|
||||||
|
that exercise `BLOCKED`, `GO`, and the leaked-corpus refusal. They are invented numbers on fabricated
|
||||||
|
subject keys: **no model produced them and they say nothing about the real classifier.** Recording a
|
||||||
|
video or writing a report from them means saying so.
|
||||||
+3
-2
@@ -18,12 +18,13 @@
|
|||||||
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
|
"test:grounded-chat": "node tests/grounded-chat.acceptance.mjs",
|
||||||
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
|
"test:photo-lifecycle": "node tests/photo-lifecycle.acceptance.mjs",
|
||||||
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
|
"test:training-contribution": "node tests/training-contribution.acceptance.mjs",
|
||||||
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
"check:syntax": "node --check app.js && node --check server.mjs && node --check service-worker.js && node --check src/analysis.js && node --check src/domain.js && node --check src/draft.js && node --check src/suggestion.js && node --check src/followup.js && node --check src/vision-readiness.js && node --check src/hermes-agent-service.js && node --check src/vision-config.js && node --check src/vision-service.js && node --check src/vision-corpus.js && node --check src/summary.js && node --check src/photo-lifecycle.js && node --check src/grounded-chat.js && node --check src/training-contribution.js && node --check scripts/record_release_demo.mjs && node --check tests/staging.acceptance.mjs && node --check tests/manual-fallback.acceptance.mjs && node --check tests/provisional-suggestion.acceptance.mjs && node --check tests/followup.acceptance.mjs && node --check tests/vision-refusal.acceptance.mjs && bash -n scripts/bootstrap_selfhost_smolvlm.sh && bash -n scripts/run_selfhost_smolvlm.sh && python3 -m py_compile scripts/corpus_manifest.py scripts/ingest_training_photo.py scripts/build_release.py scripts/deploy_staging.py scripts/build_vision_corpus.py scripts/calibration_report.py && node --check scripts/record_worker_benchmark.mjs && node --check tests/worker-readiness.acceptance.mjs && node --check tests/pattern-summary.acceptance.mjs && node --check tests/photo-lifecycle.acceptance.mjs && node --check tests/training-contribution.acceptance.mjs && node --check tests/portability.acceptance.mjs && node --check tests/grounded-chat.acceptance.mjs && bash -n scripts/worker_ready.sh && bash -n deploy/worker/entrypoint.sh && bash -n deploy/worker/build_image.sh && bash -n deploy/worker/run_worker_container.sh",
|
||||||
"check:diff": "bash scripts/check_diff.sh",
|
"check:diff": "bash scripts/check_diff.sh",
|
||||||
"start": "node server.mjs",
|
"start": "node server.mjs",
|
||||||
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
|
"test:worker-readiness": "node tests/worker-readiness.acceptance.mjs",
|
||||||
"test:portability": "node tests/portability.acceptance.mjs",
|
"test:portability": "node tests/portability.acceptance.mjs",
|
||||||
"test:corpus": "python3 tests/corpus-manifest.test.py"
|
"test:corpus": "python3 tests/corpus-manifest.test.py",
|
||||||
|
"test:calibration": "python3 tests/calibration-harness.test.py"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"playwright": "^1.62.1"
|
"playwright": "^1.62.1"
|
||||||
|
|||||||
@@ -59,6 +59,10 @@ DEMO_REQUIRED_CAPTIONS = (
|
|||||||
"Every ingested sample records consent, source digest, both hashes, derivation, review, and lifecycle — the caller file path is never stored",
|
"Every ingested sample records consent, source digest, both hashes, derivation, review, and lifecycle — the caller file path is never stored",
|
||||||
"Withdrawing deletes the derived image and any raw copy, keeps the original record, and appends a tombstone — the trainer sees zero active samples",
|
"Withdrawing deletes the derived image and any raw copy, keeps the original record, and appends a tombstone — the trainer sees zero active samples",
|
||||||
"Revoking the same contribution twice is refused and appends nothing: the manifest only ever grows",
|
"Revoking the same contribution twice is refused and appends nothing: the manifest only ever grows",
|
||||||
|
"The calibration harness is a command-line release gate, not a screen in the app — this runs it against committed synthetic predictions",
|
||||||
|
"Every score on this card is invented to exercise the arithmetic: no model produced them and none of them describes the real classifier",
|
||||||
|
"Each candidate is scored on test at a threshold chosen on validation only; picking a threshold on the split you report is refused",
|
||||||
|
"One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it",
|
||||||
"A backup is versioned, and it carries the withdrawn receipt as a tombstone rather than silence",
|
"A backup is versioned, and it carries the withdrawn receipt as a tombstone rather than silence",
|
||||||
"An older backup is upgraded on import, and the notice says which version the file came from",
|
"An older backup is upgraded on import, and the notice says which version the file came from",
|
||||||
"A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched",
|
"A file from a newer Timmy is refused with an instruction, and the ledger it would have replaced is untouched",
|
||||||
@@ -161,6 +165,9 @@ def main() -> int:
|
|||||||
run(["python3", "scripts/build_vision_corpus.py", "--verify"], tree)
|
run(["python3", "scripts/build_vision_corpus.py", "--verify"], tree)
|
||||||
# Issue #25's corpus pipeline is a CLI, so its own suite is a release gate, not just CI.
|
# Issue #25's corpus pipeline is a CLI, so its own suite is a release gate, not just CI.
|
||||||
run(["python3", "tests/corpus-manifest.test.py", "-v"], tree)
|
run(["python3", "tests/corpus-manifest.test.py", "-v"], tree)
|
||||||
|
# Issue #29's evaluation harness decides GO/BLOCKED for a model, so its refusal and gate
|
||||||
|
# arithmetic are exercised before this release is allowed to exist.
|
||||||
|
run(["python3", "tests/calibration-harness.test.py", "-v"], tree)
|
||||||
server_env = dict(os.environ)
|
server_env = dict(os.environ)
|
||||||
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
|
server_env.update({"HOST": "127.0.0.1", "PORT": "4173", "TIMMY_VISION_ENABLED": "false"})
|
||||||
# A leftover server from another checkout answers on this port and the readiness probe cannot
|
# A leftover server from another checkout answers on this port and the readiness probe cannot
|
||||||
@@ -350,6 +357,7 @@ def main() -> int:
|
|||||||
"grounded_chat_citations": "passed",
|
"grounded_chat_citations": "passed",
|
||||||
"training_contribution_consent": "passed",
|
"training_contribution_consent": "passed",
|
||||||
"corpus_manifest_provenance": "passed",
|
"corpus_manifest_provenance": "passed",
|
||||||
|
"calibration_harness_gates": "passed",
|
||||||
"portable_backup_restore": "passed",
|
"portable_backup_restore": "passed",
|
||||||
"dependency_audit_high": "passed",
|
"dependency_audit_high": "passed",
|
||||||
"syntax": "passed",
|
"syntax": "passed",
|
||||||
|
|||||||
@@ -0,0 +1,378 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Contributor-held-out evaluation harness for the Timmy Bristol classifier.
|
||||||
|
|
||||||
|
This is the comparison and release-gate tool for issue #29: it turns candidate predictions into the
|
||||||
|
numbers the ROADMAP gates are written in — macro-F1, per-class precision/recall, a confusion matrix,
|
||||||
|
calibration error, coverage, and precision-at-coverage — and it refuses a corpus whose splits are
|
||||||
|
not contributor-held-out or not reproducible.
|
||||||
|
|
||||||
|
Nothing here trains, scores, or diagnoses: it reads predictions that something else produced, and it
|
||||||
|
reports what those predictions would have done on held-out contributors. Every fixture used to test
|
||||||
|
it is synthetic.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
BRISTOL_CLASSES = tuple(range(1, 8))
|
||||||
|
SPLITS = ("train", "validation", "test")
|
||||||
|
# The split rule is the ingestion rule (scripts/ingest_training_photo.py): the bucket of the first
|
||||||
|
# byte of the pseudonymous subject key. Keeping it here means a split can be re-derived from the
|
||||||
|
# key alone, so a corpus cannot quietly re-label its own splits.
|
||||||
|
_TRAIN_EDGE = 205
|
||||||
|
_VALIDATION_EDGE = 230
|
||||||
|
_SHA_LIKE = re.compile(r"^[0-9a-f]{16}$")
|
||||||
|
|
||||||
|
# The ROADMAP release gates. A candidate must clear every one on its held-out split; a limit is
|
||||||
|
# inclusive, so a model exactly on the line passes.
|
||||||
|
RELEASE_GATES = {"macroF1": 0.80, "precisionAtCoverage": 0.90, "ece": 0.05}
|
||||||
|
|
||||||
|
|
||||||
|
class EvaluationError(Exception):
|
||||||
|
"""The corpus or the caller asked for a comparison that cannot be trusted."""
|
||||||
|
|
||||||
|
|
||||||
|
def expected_split(subject_key: str) -> str:
|
||||||
|
"""The split a pseudonymous subject key must belong to, by the ingestion rule."""
|
||||||
|
if not isinstance(subject_key, str) or not _SHA_LIKE.match(subject_key):
|
||||||
|
raise EvaluationError("subject key must be the 16-character pseudonymous digest")
|
||||||
|
bucket = int(subject_key[:2], 16)
|
||||||
|
if bucket < _TRAIN_EDGE:
|
||||||
|
return "train"
|
||||||
|
if bucket < _VALIDATION_EDGE:
|
||||||
|
return "validation"
|
||||||
|
return "test"
|
||||||
|
|
||||||
|
|
||||||
|
def _calibration_error(accepted: list[tuple[int, int, float]], bins: int = 10) -> tuple[float, list[dict]]:
|
||||||
|
"""Expected calibration error over the accepted suggestions, by equal-width confidence bin."""
|
||||||
|
if not accepted:
|
||||||
|
return 0.0, []
|
||||||
|
edges = [index / bins for index in range(bins + 1)]
|
||||||
|
buckets: list[list[tuple[int, int, float]]] = [[] for _ in range(bins)]
|
||||||
|
for truth, predicted, confidence in accepted:
|
||||||
|
index = min(int(confidence * bins), bins - 1)
|
||||||
|
buckets[index].append((truth, predicted, confidence))
|
||||||
|
error = 0.0
|
||||||
|
detail: list[dict] = []
|
||||||
|
for index, bucket in enumerate(buckets):
|
||||||
|
if not bucket:
|
||||||
|
continue
|
||||||
|
accuracy = sum(1 for truth, predicted, _ in bucket if truth == predicted) / len(bucket)
|
||||||
|
mean_confidence = sum(confidence for _, _, confidence in bucket) / len(bucket)
|
||||||
|
share = len(bucket) / len(accepted)
|
||||||
|
error += share * abs(accuracy - mean_confidence)
|
||||||
|
detail.append(
|
||||||
|
{
|
||||||
|
"lower": edges[index],
|
||||||
|
"upper": edges[index + 1],
|
||||||
|
"count": len(bucket),
|
||||||
|
"accuracy": accuracy,
|
||||||
|
"meanConfidence": mean_confidence,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return error, detail
|
||||||
|
|
||||||
|
|
||||||
|
def assert_contributor_held_out(records: list[dict]) -> None:
|
||||||
|
"""Refuse a corpus whose splits are not contributor-held-out and reproducible.
|
||||||
|
|
||||||
|
A split must be re-derivable from the pseudonymous subject key alone (the ingestion rule), and
|
||||||
|
one contributor's photos must all land in the same split. Otherwise a held-out score is not
|
||||||
|
held out: the model has already seen that person's body in training.
|
||||||
|
"""
|
||||||
|
splits_by_subject: dict[str, set[str]] = {}
|
||||||
|
for entry in records:
|
||||||
|
subject = entry.get("subjectKey")
|
||||||
|
if not isinstance(subject, str) or not subject:
|
||||||
|
raise EvaluationError("every prediction must carry the pseudonymous subject key")
|
||||||
|
split = entry.get("split")
|
||||||
|
if split not in SPLITS:
|
||||||
|
raise EvaluationError(f"split must be one of {', '.join(SPLITS)}: {split!r}")
|
||||||
|
splits_by_subject.setdefault(subject, set()).add(split)
|
||||||
|
for subject, splits in splits_by_subject.items():
|
||||||
|
if len(splits) > 1:
|
||||||
|
raise EvaluationError(
|
||||||
|
f"contributor {subject} leaks across {', '.join(sorted(splits))}: splits must be contributor-held-out"
|
||||||
|
)
|
||||||
|
for entry in records:
|
||||||
|
subject = entry["subjectKey"]
|
||||||
|
required = expected_split(subject)
|
||||||
|
if entry["split"] != required:
|
||||||
|
raise EvaluationError(
|
||||||
|
f"split {entry['split']!r} for contributor {subject} is not reproducible: the subject key derives {required!r}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def compare(models: dict[str, list[dict]], *, min_coverage: float = 0.5, gates: dict | None = None) -> dict:
|
||||||
|
"""Compare candidate models against the release gates.
|
||||||
|
|
||||||
|
Each candidate gets one threshold, chosen on its own validation split, and is then measured on
|
||||||
|
its test split. The gates are the ROADMAP release gates; all limits are inclusive.
|
||||||
|
"""
|
||||||
|
limits = dict(gates or RELEASE_GATES)
|
||||||
|
rows: list[dict] = []
|
||||||
|
for model_id in sorted(models):
|
||||||
|
records = models[model_id]
|
||||||
|
assert_contributor_held_out(records)
|
||||||
|
test = [entry for entry in records if entry.get("split") == "test"]
|
||||||
|
validation = [entry for entry in records if entry.get("split") == "validation"]
|
||||||
|
if not test:
|
||||||
|
raise EvaluationError(f"{model_id} has no test-split predictions to measure a release on")
|
||||||
|
if not validation:
|
||||||
|
raise EvaluationError(f"{model_id} has no validation-split predictions to fix a threshold on")
|
||||||
|
selection = select_threshold(validation, min_coverage=min_coverage)
|
||||||
|
scored = evaluate(test, threshold=selection["threshold"])
|
||||||
|
failed = []
|
||||||
|
if scored["macroF1"] < limits["macroF1"]:
|
||||||
|
failed.append("macroF1")
|
||||||
|
if scored["precisionAtCoverage"] < limits["precisionAtCoverage"]:
|
||||||
|
failed.append("precisionAtCoverage")
|
||||||
|
if scored["ece"] > limits["ece"]:
|
||||||
|
failed.append("ece")
|
||||||
|
rows.append(
|
||||||
|
{
|
||||||
|
"model": model_id,
|
||||||
|
**scored,
|
||||||
|
"selectionSplit": "validation",
|
||||||
|
"evaluatedSplit": "test",
|
||||||
|
"minCoverage": min_coverage,
|
||||||
|
"gate": {"passed": not failed, "failed": failed},
|
||||||
|
}
|
||||||
|
)
|
||||||
|
ordered = sorted(
|
||||||
|
rows,
|
||||||
|
key=lambda row: (
|
||||||
|
not row["gate"]["passed"],
|
||||||
|
-row["macroF1"],
|
||||||
|
-row["precisionAtCoverage"],
|
||||||
|
row["model"],
|
||||||
|
),
|
||||||
|
)
|
||||||
|
passing = [row for row in ordered if row["gate"]["passed"]]
|
||||||
|
return {
|
||||||
|
"gates": limits,
|
||||||
|
"models": ordered,
|
||||||
|
"recommended": passing[0]["model"] if passing else None,
|
||||||
|
"verdict": "GO" if passing else "BLOCKED",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def select_threshold(records: list[dict], *, min_coverage: float = 0.5) -> dict:
|
||||||
|
"""Choose the abstention threshold that maximises precision at a required coverage.
|
||||||
|
|
||||||
|
Selection reads the validation split only. Fixing a threshold on the same data the release gate
|
||||||
|
is measured on would manufacture whatever precision the report needs.
|
||||||
|
"""
|
||||||
|
if not records:
|
||||||
|
raise EvaluationError("threshold selection needs validation predictions")
|
||||||
|
if not 0.0 <= min_coverage <= 1.0:
|
||||||
|
raise EvaluationError(f"a coverage floor must be between 0 and 1: {min_coverage}")
|
||||||
|
if any(entry.get("split") != "validation" for entry in records):
|
||||||
|
raise EvaluationError("threshold selection must use the validation split only")
|
||||||
|
assert_contributor_held_out(records)
|
||||||
|
|
||||||
|
candidates = {0.0}
|
||||||
|
candidates.update(
|
||||||
|
entry["prediction"]["confidence"] for entry in records if entry["prediction"] is not None
|
||||||
|
)
|
||||||
|
best: dict | None = None
|
||||||
|
for threshold in sorted(candidates):
|
||||||
|
report = evaluate(records, threshold=threshold)
|
||||||
|
# Float comparison: a coverage exactly on the floor must not be lost to rounding.
|
||||||
|
if report["coverage"] + 1e-12 < min_coverage:
|
||||||
|
continue
|
||||||
|
if best is None:
|
||||||
|
best = report
|
||||||
|
continue
|
||||||
|
more_precise = round(report["precisionAtCoverage"], 12) > round(best["precisionAtCoverage"], 12)
|
||||||
|
equally_precise_but_wider = (
|
||||||
|
round(report["precisionAtCoverage"], 12) == round(best["precisionAtCoverage"], 12)
|
||||||
|
and report["coverage"] > best["coverage"]
|
||||||
|
)
|
||||||
|
if more_precise or equally_precise_but_wider:
|
||||||
|
best = report
|
||||||
|
if best is None:
|
||||||
|
raise EvaluationError(f"no threshold reaches the {min_coverage:.0%} coverage floor on validation")
|
||||||
|
return {**best, "selectionSplit": "validation", "minCoverage": min_coverage}
|
||||||
|
|
||||||
|
|
||||||
|
def evaluate(records: list[dict], *, threshold: float = 0.0) -> dict:
|
||||||
|
"""Score one model's accepted suggestions at a fixed abstention threshold."""
|
||||||
|
accepted: list[tuple[int, int, float]] = []
|
||||||
|
abstentions = {"low-confidence": 0, "no-suggestion": 0}
|
||||||
|
for entry in records:
|
||||||
|
truth = entry["truth"]["bristolType"]
|
||||||
|
prediction = entry["prediction"]
|
||||||
|
if prediction is None:
|
||||||
|
abstentions["no-suggestion"] += 1
|
||||||
|
continue
|
||||||
|
if prediction["confidence"] < threshold:
|
||||||
|
abstentions["low-confidence"] += 1
|
||||||
|
continue
|
||||||
|
accepted.append((truth, prediction["bristolType"], prediction["confidence"]))
|
||||||
|
|
||||||
|
confusion: dict[str, int] = {}
|
||||||
|
for truth, predicted, _ in accepted:
|
||||||
|
key = f"{truth}->{predicted}"
|
||||||
|
confusion[key] = confusion.get(key, 0) + 1
|
||||||
|
|
||||||
|
truth_classes = {truth for truth, _, _ in accepted}
|
||||||
|
per_class: dict[int, dict] = {}
|
||||||
|
for label in sorted(truth_classes):
|
||||||
|
true_positive = sum(1 for truth, predicted, _ in accepted if truth == label and predicted == label)
|
||||||
|
predicted_positive = sum(1 for _, predicted, _ in accepted if predicted == label)
|
||||||
|
actual_positive = sum(1 for truth, _, _ in accepted if truth == label)
|
||||||
|
precision = true_positive / predicted_positive if predicted_positive else 0.0
|
||||||
|
recall = true_positive / actual_positive if actual_positive else 0.0
|
||||||
|
f1 = 2 * precision * recall / (precision + recall) if precision + recall else 0.0
|
||||||
|
per_class[label] = {
|
||||||
|
"precision": precision,
|
||||||
|
"recall": recall,
|
||||||
|
"f1": f1,
|
||||||
|
"support": actual_positive,
|
||||||
|
"predicted": predicted_positive,
|
||||||
|
}
|
||||||
|
|
||||||
|
macro_f1 = sum(row["f1"] for row in per_class.values()) / len(per_class) if per_class else 0.0
|
||||||
|
ece, calibration_bins = _calibration_error(accepted)
|
||||||
|
truth_classes_in_split = {entry["truth"]["bristolType"] for entry in records}
|
||||||
|
return {
|
||||||
|
"threshold": threshold,
|
||||||
|
"counts": {"total": len(records), "accepted": len(accepted), "abstained": len(records) - len(accepted)},
|
||||||
|
"abstentions": abstentions,
|
||||||
|
"coverage": len(accepted) / len(records) if records else 0.0,
|
||||||
|
"precisionAtCoverage": (
|
||||||
|
sum(1 for truth, predicted, _ in accepted if truth == predicted) / len(accepted) if accepted else 0.0
|
||||||
|
),
|
||||||
|
"perClass": per_class,
|
||||||
|
"excludedClasses": sorted(truth_classes_in_split - set(per_class)),
|
||||||
|
"macroF1": macro_f1,
|
||||||
|
"ece": ece,
|
||||||
|
"calibrationBins": calibration_bins,
|
||||||
|
"confusion": dict(sorted(confusion.items())),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_prediction(entry: object, where: str) -> dict:
|
||||||
|
"""Normalise and bound one prediction record, or refuse it."""
|
||||||
|
if not isinstance(entry, dict):
|
||||||
|
raise EvaluationError(f"{where} is not a prediction object")
|
||||||
|
model = entry.get("model")
|
||||||
|
if not isinstance(model, str) or not model.strip():
|
||||||
|
raise EvaluationError(f"{where} needs the model that produced the prediction")
|
||||||
|
truth = entry.get("truth")
|
||||||
|
if not isinstance(truth, dict) or not isinstance(truth.get("bristolType"), int):
|
||||||
|
raise EvaluationError(f"{where} needs truth.bristolType as a whole number")
|
||||||
|
if truth["bristolType"] not in BRISTOL_CLASSES:
|
||||||
|
raise EvaluationError(f"{where} truth.bristolType {truth['bristolType']} is outside Bristol 1-7")
|
||||||
|
split = entry.get("split")
|
||||||
|
if split not in SPLITS:
|
||||||
|
raise EvaluationError(f"{where} split must be one of {', '.join(SPLITS)}")
|
||||||
|
subject = entry.get("subjectKey")
|
||||||
|
if not isinstance(subject, str) or not _SHA_LIKE.match(subject):
|
||||||
|
raise EvaluationError(f"{where} needs the 16-character pseudonymous subject key")
|
||||||
|
prediction = entry.get("prediction")
|
||||||
|
if prediction is not None:
|
||||||
|
if not isinstance(prediction, dict):
|
||||||
|
raise EvaluationError(f"{where} prediction must be an object or null")
|
||||||
|
if prediction.get("bristolType") not in BRISTOL_CLASSES:
|
||||||
|
raise EvaluationError(f"{where} predicted bristolType must be 1-7")
|
||||||
|
confidence = prediction.get("confidence")
|
||||||
|
if not isinstance(confidence, (int, float)) or isinstance(confidence, bool) or not 0.0 <= confidence <= 1.0:
|
||||||
|
raise EvaluationError(f"{where} confidence must be a number between 0 and 1")
|
||||||
|
return {
|
||||||
|
"model": model.strip(),
|
||||||
|
"subjectKey": subject,
|
||||||
|
"split": split,
|
||||||
|
"truth": {"bristolType": truth["bristolType"]},
|
||||||
|
"prediction": None
|
||||||
|
if prediction is None
|
||||||
|
else {"bristolType": prediction["bristolType"], "confidence": float(prediction["confidence"])},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def load_predictions(paths: list[str]) -> list[dict]:
|
||||||
|
"""Read prediction JSONL files, refusing anything the gates could not honestly be computed from."""
|
||||||
|
records: list[dict] = []
|
||||||
|
for raw_path in paths:
|
||||||
|
path = Path(raw_path)
|
||||||
|
if not path.is_file():
|
||||||
|
raise EvaluationError(f"predictions file not found: {path}")
|
||||||
|
for number, line in enumerate(path.read_text(encoding="utf-8").splitlines(), 1):
|
||||||
|
if not line.strip():
|
||||||
|
continue
|
||||||
|
try:
|
||||||
|
entry = json.loads(line)
|
||||||
|
except json.JSONDecodeError as exc:
|
||||||
|
raise EvaluationError(f"{path}:{number} is not valid JSON: {exc.msg}") from exc
|
||||||
|
records.append(_validate_prediction(entry, f"{path}:{number}"))
|
||||||
|
if not records:
|
||||||
|
raise EvaluationError("no prediction records were supplied")
|
||||||
|
return records
|
||||||
|
|
||||||
|
|
||||||
|
def group_by_model(records: list[dict]) -> dict[str, list[dict]]:
|
||||||
|
models: dict[str, list[dict]] = {}
|
||||||
|
for entry in records:
|
||||||
|
models.setdefault(entry["model"], []).append(
|
||||||
|
{key: value for key, value in entry.items() if key != "model"}
|
||||||
|
)
|
||||||
|
return models
|
||||||
|
|
||||||
|
|
||||||
|
def format_report(report: dict, *, source: str) -> str:
|
||||||
|
"""Render the comparison as a compact receipt a human can read in a terminal."""
|
||||||
|
lines = [
|
||||||
|
"TIMMY CALIBRATION REPORT",
|
||||||
|
f"source: {source}",
|
||||||
|
f"threshold: chosen on validation only · scored on test · coverage floor {report['models'][0]['minCoverage']:.0%}"
|
||||||
|
if report["models"]
|
||||||
|
else "threshold: nothing to score",
|
||||||
|
"candidates: " + ", ".join(row["model"] for row in report["models"]),
|
||||||
|
"",
|
||||||
|
f"{'model':<26}{'thr':>5}{'macroF1':>9}{'prec@cov':>10}{'coverage':>10}{'ECE':>7} gate",
|
||||||
|
]
|
||||||
|
for row in report["models"]:
|
||||||
|
gate = "PASS" if row["gate"]["passed"] else "FAIL: " + ", ".join(row["gate"]["failed"])
|
||||||
|
lines.append(
|
||||||
|
f"{row['model']:<26}{row['threshold']:>5.2f}{row['macroF1']:>9.3f}"
|
||||||
|
f"{row['precisionAtCoverage']:>10.3f}{row['coverage']:>10.2f}{row['ece']:>7.3f} {gate}"
|
||||||
|
)
|
||||||
|
limits = report["gates"]
|
||||||
|
lines += [
|
||||||
|
"",
|
||||||
|
"verdict: " + report["verdict"] + (f" — recommended {report['recommended']}" if report["recommended"] else ""),
|
||||||
|
f"gates: macroF1 >= {limits['macroF1']:.2f} · precision@coverage >= {limits['precisionAtCoverage']:.2f} · ECE <= {limits['ece']:.2f}",
|
||||||
|
]
|
||||||
|
return "\n".join(lines)
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
parser = argparse.ArgumentParser(description="Score Bristol candidates against the release gates.")
|
||||||
|
parser.add_argument("--predictions", action="append", required=True, metavar="PATH", help="prediction JSONL (repeatable)")
|
||||||
|
parser.add_argument("--min-coverage", type=float, default=0.5, help="coverage floor for threshold selection")
|
||||||
|
parser.add_argument("--json", metavar="PATH", help="write the full report as JSON here")
|
||||||
|
parser.add_argument("--source", default=None, help="what the predictions are, recorded in the receipt")
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
try:
|
||||||
|
records = load_predictions(args.predictions)
|
||||||
|
report = compare(group_by_model(records), min_coverage=args.min_coverage)
|
||||||
|
except EvaluationError as exc:
|
||||||
|
print(f"calibration harness refused: {exc}", file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
source = args.source or f"{len(records)} prediction records from {len(args.predictions)} file(s)"
|
||||||
|
print(format_report(report, source=source))
|
||||||
|
if args.json:
|
||||||
|
Path(args.json).write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
|
||||||
|
return 0 if report["verdict"] == "GO" else 3
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -710,6 +710,145 @@ await caption('Every ingested sample records consent, source digest, both hashes
|
|||||||
await caption('Withdrawing deletes the derived image and any raw copy, keeps the original record, and appends a tombstone — the trainer sees zero active samples', 3800);
|
await caption('Withdrawing deletes the derived image and any raw copy, keeps the original record, and appends a tombstone — the trainer sees zero active samples', 3800);
|
||||||
await caption('Revoking the same contribution twice is refused and appends nothing: the manifest only ever grows', 3200);
|
await caption('Revoking the same contribution twice is refused and appends nothing: the manifest only ever grows', 3200);
|
||||||
await page.evaluate(() => document.querySelector('#corpus-audit')?.remove());
|
await page.evaluate(() => document.querySelector('#corpus-audit')?.remove());
|
||||||
|
// Issue #29 — the release gate is a command-line harness, not a screen in the app. This scene runs
|
||||||
|
// the real harness over the committed synthetic prediction fixtures and shows the receipt it wrote:
|
||||||
|
// a blocked comparison, a passing fixture, and a refused leaked corpus. Every number and every
|
||||||
|
// refusal line on the card is read back from those runs; the scores are invented fixtures, and the
|
||||||
|
// captions say so, because a fixture is not a measurement of any model.
|
||||||
|
async function calibrationCli(args) {
|
||||||
|
try {
|
||||||
|
const result = await execFileAsync('python3', ['scripts/calibration_report.py', ...args], { cwd: process.cwd() });
|
||||||
|
return { code: 0, stdout: result.stdout, stderr: result.stderr };
|
||||||
|
} catch (error) {
|
||||||
|
return {
|
||||||
|
code: typeof error.code === 'number' ? error.code : 1,
|
||||||
|
stdout: error.stdout || '',
|
||||||
|
stderr: error.stderr || '',
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
async function calibrationGate() {
|
||||||
|
const directory = await mkdtemp(path.join(os.tmpdir(), 'timmy-calibration-demo-'));
|
||||||
|
const reportPath = path.join(directory, 'report.json');
|
||||||
|
const source = 'invented scores, not measurements of any model';
|
||||||
|
try {
|
||||||
|
const blocked = await calibrationCli([
|
||||||
|
'--predictions', 'tests/fixtures/calibration/predictions.jsonl',
|
||||||
|
'--json', reportPath, '--source', source,
|
||||||
|
]);
|
||||||
|
if (blocked.code !== 3) throw new Error('the synthetic candidate fixtures must block the release');
|
||||||
|
const report = JSON.parse(await readFile(reportPath, 'utf8'));
|
||||||
|
if (report.verdict !== 'BLOCKED' || report.recommended !== null) {
|
||||||
|
throw new Error('a blocked gate must recommend nobody');
|
||||||
|
}
|
||||||
|
if (report.models.length !== 3) throw new Error('the comparison must score all three candidate fixtures');
|
||||||
|
for (const row of report.models) {
|
||||||
|
if (!row.gate.failed.length) throw new Error(`${row.model}: a fixture candidate must fail at least one gate`);
|
||||||
|
if (row.selectionSplit !== 'validation' || row.evaluatedSplit !== 'test') {
|
||||||
|
throw new Error(`${row.model}: the threshold must come from validation and the score from test`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const go = await calibrationCli([
|
||||||
|
'--predictions', 'tests/fixtures/calibration/predictions-go.jsonl',
|
||||||
|
'--source', 'engineered fixture — proves the passing branch, not a result',
|
||||||
|
]);
|
||||||
|
if (go.code !== 0 || !/verdict: GO/.test(go.stdout)) throw new Error('the engineered fixture must prove the passing branch');
|
||||||
|
const leak = await calibrationCli(['--predictions', 'tests/fixtures/calibration/predictions-leaky.jsonl']);
|
||||||
|
if (leak.code !== 2 || !/leaks across/.test(leak.stderr)) throw new Error('a leaked corpus must be refused before scoring');
|
||||||
|
return { report, source, blocked, go, leak };
|
||||||
|
} finally {
|
||||||
|
await rm(directory, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const gate = await calibrationGate();
|
||||||
|
await tap('[data-view="home"]', 450);
|
||||||
|
const gateRows = [
|
||||||
|
['command', 'python3 scripts/calibration_report.py --predictions <fixtures>'],
|
||||||
|
['split', 'threshold chosen on validation · scored on test'],
|
||||||
|
['floor', `coverage floor ${Math.round(gate.report.models[0].minCoverage * 100)}%`],
|
||||||
|
['—', ''],
|
||||||
|
];
|
||||||
|
for (const row of gate.report.models) {
|
||||||
|
gateRows.push([
|
||||||
|
row.model,
|
||||||
|
`thr ${row.threshold.toFixed(2)} · F1 ${row.macroF1.toFixed(3)} · P@C ${row.precisionAtCoverage.toFixed(3)} · cov ${row.coverage.toFixed(2)}`,
|
||||||
|
]);
|
||||||
|
gateRows.push(['gate', `FAIL: ${row.gate.failed.join(', ')} · ECE ${row.ece.toFixed(3)}`]);
|
||||||
|
}
|
||||||
|
gateRows.push(['—', '']);
|
||||||
|
gateRows.push(['verdict', `${gate.report.verdict} · no candidate clears every gate`]);
|
||||||
|
gateRows.push(['gates', 'macroF1 ≥ 0.80 · precision@coverage ≥ 0.90 · ECE ≤ 0.05']);
|
||||||
|
gateRows.push(['—', '']);
|
||||||
|
gateRows.push(['engineered', 'GO — gate-plumbing-fixture clears every gate (fixture, not a result)']);
|
||||||
|
gateRows.push(['leaked', gate.leak.stderr.trim().replace(/^calibration harness refused:\s*/, '')]);
|
||||||
|
gateRows.push(['data', 'synthetic predictions · no model was measured']);
|
||||||
|
const gateCard = await page.evaluate(rows => {
|
||||||
|
document.querySelector('#calibration-report')?.remove();
|
||||||
|
const panel = document.createElement('div');
|
||||||
|
panel.id = 'calibration-report';
|
||||||
|
Object.assign(panel.style, {
|
||||||
|
position: 'fixed', inset: '96px 14px 40px 14px', zIndex: '15000',
|
||||||
|
background: '#181412', color: '#f6f1e6', borderRadius: '18px',
|
||||||
|
border: '1px solid rgba(245,201,91,.45)', padding: '13px 13px 11px',
|
||||||
|
font: '600 9.5px/1.45 ui-monospace, SFMono-Regular, Menlo, monospace',
|
||||||
|
display: 'flex', flexDirection: 'column', gap: '1px', overflow: 'hidden',
|
||||||
|
boxShadow: '0 18px 40px rgba(0,0,0,.42)',
|
||||||
|
});
|
||||||
|
const head = document.createElement('div');
|
||||||
|
head.textContent = 'CALIBRATION HARNESS';
|
||||||
|
Object.assign(head.style, { font: '800 13.5px/1.2 system-ui', letterSpacing: '.12em', color: '#f5c95b', marginBottom: '2px' });
|
||||||
|
const sub = document.createElement('div');
|
||||||
|
sub.textContent = 'command line, not app UI — real run, synthetic predictions';
|
||||||
|
Object.assign(sub.style, { font: '700 9px/1.3 system-ui', color: '#c9bfae', marginBottom: '8px' });
|
||||||
|
panel.append(head, sub);
|
||||||
|
for (const [key, value] of rows) {
|
||||||
|
const row = document.createElement('div');
|
||||||
|
if (key === '—') { row.style.height = '6px'; panel.append(row); continue; }
|
||||||
|
Object.assign(row.style, { display: 'flex', gap: '6px', alignItems: 'baseline' });
|
||||||
|
const k = document.createElement('span');
|
||||||
|
k.textContent = key;
|
||||||
|
k.style.cssText = 'flex:0 0 74px;color:#8fd7c8;text-transform:uppercase;font-size:8.5px;letter-spacing:.06em';
|
||||||
|
const v = document.createElement('span');
|
||||||
|
v.textContent = value;
|
||||||
|
v.style.cssText = 'flex:1 1 auto;word-break:break-word;color:#f6f1e6';
|
||||||
|
row.append(k, v);
|
||||||
|
panel.append(row);
|
||||||
|
}
|
||||||
|
document.body.append(panel);
|
||||||
|
panel.animate([{ opacity: 0, transform: 'translateY(12px)' }, { opacity: 1, transform: 'translateY(0)' }], { duration: 320, fill: 'forwards' });
|
||||||
|
return { height: panel.clientHeight, scroll: panel.scrollHeight, text: panel.innerText };
|
||||||
|
}, gateRows);
|
||||||
|
await sleep(420);
|
||||||
|
const gateProof = await page.evaluate(() => {
|
||||||
|
const panel = document.querySelector('#calibration-report');
|
||||||
|
if (!panel) return { ok: false, reason: 'panel missing' };
|
||||||
|
const rect = panel.getBoundingClientRect();
|
||||||
|
const topmost = document.elementFromPoint(Math.round(rect.left + rect.width / 2), Math.round(rect.top + rect.height / 2));
|
||||||
|
// Same rule as the corpus card: a translucent card lets the screen behind it ghost through.
|
||||||
|
const opaque = !/^rgba\(.*,\s*(0?\.\d+|0)\)$/.test(getComputedStyle(panel).backgroundColor);
|
||||||
|
return {
|
||||||
|
ok: opaque && rect.top >= 60 && rect.bottom <= window.innerHeight && rect.width > 300 && rect.height > 300
|
||||||
|
&& (topmost === panel || panel.contains(topmost)) && panel.scrollHeight <= panel.clientHeight + 1,
|
||||||
|
top: Math.round(rect.top), height: Math.round(rect.height), width: Math.round(rect.width),
|
||||||
|
scroll: panel.scrollHeight, client: panel.clientHeight, opaque,
|
||||||
|
covered: topmost === panel || panel.contains(topmost),
|
||||||
|
};
|
||||||
|
});
|
||||||
|
if (!gateProof.ok) throw new Error(`Calibration card is not fully readable on screen: ${JSON.stringify(gateProof)}`);
|
||||||
|
// The card is drawn from the report; prove the values on it are the ones the harness measured.
|
||||||
|
for (const row of gate.report.models) {
|
||||||
|
if (!gateCard.text.includes(row.macroF1.toFixed(3))) {
|
||||||
|
throw new Error(`the card must show the macro-F1 the harness measured for ${row.model}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!gateCard.text.includes('leaks across')) throw new Error('the card must show the refusal the leaked corpus earned');
|
||||||
|
if (!gateCard.text.includes('GO — gate-plumbing-fixture')) throw new Error('the card must show the passing branch it proved');
|
||||||
|
await caption('The calibration harness is a command-line release gate, not a screen in the app — this runs it against committed synthetic predictions', 3400);
|
||||||
|
await caption('Every score on this card is invented to exercise the arithmetic: no model produced them and none of them describes the real classifier', 3600);
|
||||||
|
await caption('Each candidate is scored on test at a threshold chosen on validation only; picking a threshold on the split you report is refused', 3600);
|
||||||
|
await caption('One contributor spread across two splits is refused before anything is scored, and a passing verdict is proven only by a fixture engineered to clear it', 3800);
|
||||||
|
await page.evaluate(() => document.querySelector('#calibration-report')?.remove());
|
||||||
|
|
||||||
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
|
// Back to the privacy view for the portability scene: the privacy entry lives on the journal view.
|
||||||
await tap('[data-view="calendar"]', 550);
|
await tap('[data-view="calendar"]', 550);
|
||||||
await tap('[data-view="privacy"]', 600);
|
await tap('[data-view="privacy"]', 600);
|
||||||
|
|||||||
@@ -0,0 +1,306 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Tests for the contributor-held-out evaluation harness.
|
||||||
|
|
||||||
|
Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
|
||||||
|
precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
|
||||||
|
splits that are contributor-held-out and reproducible, comparing candidate models.
|
||||||
|
|
||||||
|
Every fixture in this file is synthetic: fictional pseudonymous subject keys and invented
|
||||||
|
prediction scores. No real photo, label, or patient record is read or written.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib.util
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
from pathlib import Path
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
HARNESS = ROOT / "scripts" / "calibration_report.py"
|
||||||
|
|
||||||
|
|
||||||
|
def load_harness():
|
||||||
|
spec = importlib.util.spec_from_file_location("calibration_report", HARNESS)
|
||||||
|
assert spec and spec.loader, "scripts/calibration_report.py must exist"
|
||||||
|
module = importlib.util.module_from_spec(spec)
|
||||||
|
spec.loader.exec_module(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
harness = load_harness()
|
||||||
|
|
||||||
|
|
||||||
|
def record(subject_key: str, truth: int, predicted: int | None, confidence: float, *, split: str | None = None, model: str = "frozen-encoder") -> dict:
|
||||||
|
return {
|
||||||
|
"model": model,
|
||||||
|
"subjectKey": subject_key,
|
||||||
|
"split": split or harness.expected_split(subject_key),
|
||||||
|
"truth": {"bristolType": truth},
|
||||||
|
"prediction": None if predicted is None else {"bristolType": predicted, "confidence": confidence},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class ComparisonTests(unittest.TestCase):
|
||||||
|
def models(self):
|
||||||
|
return {
|
||||||
|
"specialist-candidate": [
|
||||||
|
record("cd00000000000001", 4, 4, 1.0),
|
||||||
|
record("ce00000000000002", 3, 3, 1.0),
|
||||||
|
record("e600000000000001", 4, 4, 1.0),
|
||||||
|
record("e700000000000002", 6, 6, 1.0),
|
||||||
|
],
|
||||||
|
"vlm-bootstrap": [
|
||||||
|
record("d000000000000004", 4, 4, 1.0),
|
||||||
|
record("e600000000000003", 4, 2, 1.0),
|
||||||
|
record("e700000000000004", 6, 6, 1.0),
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_measures_each_candidate_on_test_at_a_threshold_chosen_on_validation(self):
|
||||||
|
verdict = harness.compare(self.models())
|
||||||
|
self.assertEqual([row["model"] for row in verdict["models"]], ["specialist-candidate", "vlm-bootstrap"])
|
||||||
|
specialist = verdict["models"][0]
|
||||||
|
self.assertEqual(specialist["selectionSplit"], "validation")
|
||||||
|
self.assertEqual(specialist["threshold"], 0.0)
|
||||||
|
self.assertEqual(specialist["evaluatedSplit"], "test")
|
||||||
|
self.assertAlmostEqual(specialist["macroF1"], 1.0, places=6)
|
||||||
|
self.assertTrue(specialist["gate"]["passed"])
|
||||||
|
self.assertEqual(specialist["gate"]["failed"], [])
|
||||||
|
self.assertEqual(verdict["recommended"], "specialist-candidate")
|
||||||
|
self.assertEqual(verdict["verdict"], "GO")
|
||||||
|
|
||||||
|
def test_a_candidate_that_is_only_half_right_fails_the_macro_f1_precision_and_calibration_gates(self):
|
||||||
|
vlm = next(row for row in harness.compare(self.models())["models"] if row["model"] == "vlm-bootstrap")
|
||||||
|
self.assertAlmostEqual(vlm["precisionAtCoverage"], 0.5, places=6)
|
||||||
|
self.assertFalse(vlm["gate"]["passed"])
|
||||||
|
self.assertEqual(vlm["gate"]["failed"], ["macroF1", "precisionAtCoverage", "ece"])
|
||||||
|
|
||||||
|
def test_blocks_the_release_when_no_candidate_clears_every_gate(self):
|
||||||
|
models = {"vlm-bootstrap": self.models()["vlm-bootstrap"]}
|
||||||
|
verdict = harness.compare(models)
|
||||||
|
self.assertEqual(verdict["verdict"], "BLOCKED")
|
||||||
|
self.assertIsNone(verdict["recommended"])
|
||||||
|
self.assertEqual(verdict["gates"], {"macroF1": 0.80, "precisionAtCoverage": 0.90, "ece": 0.05})
|
||||||
|
|
||||||
|
def test_gate_limits_are_inclusive_so_a_candidate_exactly_on_the_line_passes(self):
|
||||||
|
records = [
|
||||||
|
record("cd00000000000001", 4, 4, 0.9),
|
||||||
|
record("e600000000000001", 4, 4, 0.9),
|
||||||
|
record("e700000000000002", 3, 2, 0.9),
|
||||||
|
]
|
||||||
|
verdict = harness.compare({"frozen-encoder": records}, gates={"macroF1": 0.0, "precisionAtCoverage": 0.5, "ece": 1.0})
|
||||||
|
row = verdict["models"][0]
|
||||||
|
self.assertAlmostEqual(row["precisionAtCoverage"], 0.5, places=6)
|
||||||
|
self.assertTrue(row["gate"]["passed"], row["gate"])
|
||||||
|
|
||||||
|
def test_refuses_a_candidate_with_no_held_out_predictions(self):
|
||||||
|
with self.assertRaisesRegex(harness.EvaluationError, "no test-split predictions"):
|
||||||
|
harness.compare({"frozen-encoder": [record("cd00000000000001", 4, 4, 1.0)]})
|
||||||
|
|
||||||
|
|
||||||
|
class CliTests(unittest.TestCase):
|
||||||
|
def write_predictions(self, directory: Path, name: str, rows: list[dict]) -> Path:
|
||||||
|
path = directory / name
|
||||||
|
path.write_text("\n".join(json.dumps(row) for row in rows) + "\n", encoding="utf-8")
|
||||||
|
return path
|
||||||
|
|
||||||
|
def run_cli(self, *args: str) -> subprocess.CompletedProcess:
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, str(HARNESS), *args], cwd=ROOT, capture_output=True, text=True
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_reports_a_go_with_the_candidate_table_and_writes_the_json_report(self):
|
||||||
|
with tempfile.TemporaryDirectory() as raw:
|
||||||
|
directory = Path(raw)
|
||||||
|
predictions = self.write_predictions(directory, "predictions.jsonl", [
|
||||||
|
{**record("cd00000000000001", 4, 4, 1.0), "model": "specialist-candidate"},
|
||||||
|
{**record("e600000000000001", 4, 4, 1.0), "model": "specialist-candidate"},
|
||||||
|
{**record("e700000000000002", 6, 6, 1.0), "model": "specialist-candidate"},
|
||||||
|
])
|
||||||
|
output = directory / "report.json"
|
||||||
|
run = self.run_cli("--predictions", str(predictions), "--json", str(output), "--min-coverage", "0.5")
|
||||||
|
self.assertEqual(run.returncode, 0, run.stderr)
|
||||||
|
self.assertIn("verdict: GO", run.stdout)
|
||||||
|
self.assertIn("specialist-candidate", run.stdout)
|
||||||
|
self.assertIn("macroF1", run.stdout)
|
||||||
|
report = json.loads(output.read_text(encoding="utf-8"))
|
||||||
|
self.assertEqual(report["verdict"], "GO")
|
||||||
|
self.assertEqual(report["recommended"], "specialist-candidate")
|
||||||
|
self.assertEqual(report["models"][0]["evaluatedSplit"], "test")
|
||||||
|
self.assertEqual(report["models"][0]["selectionSplit"], "validation")
|
||||||
|
|
||||||
|
def test_exits_three_when_the_release_gate_blocks(self):
|
||||||
|
with tempfile.TemporaryDirectory() as raw:
|
||||||
|
directory = Path(raw)
|
||||||
|
predictions = self.write_predictions(directory, "predictions.jsonl", [
|
||||||
|
{**record("cd00000000000001", 4, 3, 1.0), "model": "vlm-bootstrap"},
|
||||||
|
{**record("e600000000000001", 4, 2, 1.0), "model": "vlm-bootstrap"},
|
||||||
|
])
|
||||||
|
run = self.run_cli("--predictions", str(predictions))
|
||||||
|
self.assertEqual(run.returncode, 3, run.stderr)
|
||||||
|
self.assertIn("verdict: BLOCKED", run.stdout)
|
||||||
|
|
||||||
|
|
||||||
|
def test_refuses_an_out_of_range_label_instead_of_scoring_it(self):
|
||||||
|
with tempfile.TemporaryDirectory() as raw:
|
||||||
|
directory = Path(raw)
|
||||||
|
output = directory / "report.json"
|
||||||
|
bad_truth = self.write_predictions(directory, "truth.jsonl", [
|
||||||
|
{**record("cd00000000000001", 9, 9, 1.0), "model": "frozen-encoder"},
|
||||||
|
])
|
||||||
|
run = self.run_cli("--predictions", str(bad_truth), "--json", str(output))
|
||||||
|
self.assertEqual(run.returncode, 2)
|
||||||
|
self.assertIn("outside Bristol 1-7", run.stderr)
|
||||||
|
self.assertFalse(output.exists(), "a refused run must not leave a report behind")
|
||||||
|
bad_prediction = self.write_predictions(directory, "prediction.jsonl", [
|
||||||
|
{**record("e600000000000001", 4, 9, 1.0), "model": "frozen-encoder"},
|
||||||
|
])
|
||||||
|
run = self.run_cli("--predictions", str(bad_prediction))
|
||||||
|
self.assertEqual(run.returncode, 2)
|
||||||
|
self.assertIn("predicted bristolType must be 1-7", run.stderr)
|
||||||
|
|
||||||
|
def test_refuses_a_corpus_that_leaks_a_contributor_across_splits(self):
|
||||||
|
with tempfile.TemporaryDirectory() as raw:
|
||||||
|
directory = Path(raw)
|
||||||
|
predictions = self.write_predictions(directory, "predictions.jsonl", [
|
||||||
|
{**record("cd00000000000001", 4, 4, 1.0, split="train"), "model": "frozen-encoder"},
|
||||||
|
{**record("cd00000000000001", 4, 4, 1.0, split="validation"), "model": "frozen-encoder"},
|
||||||
|
{**record("e600000000000001", 4, 4, 1.0), "model": "frozen-encoder"},
|
||||||
|
])
|
||||||
|
run = self.run_cli("--predictions", str(predictions))
|
||||||
|
self.assertEqual(run.returncode, 2)
|
||||||
|
self.assertIn("leaks across", run.stderr)
|
||||||
|
|
||||||
|
|
||||||
|
class ThresholdSelectionTests(unittest.TestCase):
|
||||||
|
def validation_records(self):
|
||||||
|
# 0xcd–0xe5 bucket to validation under the ingestion rule. Confident answers are right and
|
||||||
|
# the least confident one is wrong, which is what a threshold is supposed to exploit.
|
||||||
|
return [
|
||||||
|
record("cd00000000000001", 4, 4, 0.90),
|
||||||
|
record("ce00000000000002", 4, 4, 0.80),
|
||||||
|
record("cf00000000000003", 3, 3, 0.60),
|
||||||
|
record("d000000000000004", 7, 4, 0.40),
|
||||||
|
]
|
||||||
|
|
||||||
|
def test_picks_the_most_precise_threshold_that_still_clears_the_coverage_floor(self):
|
||||||
|
records = self.validation_records()
|
||||||
|
# At a 0.5 floor, 0.60 and 0.80 are both perfect; the wider cut wins, so 0.60.
|
||||||
|
floored = harness.select_threshold(records, min_coverage=0.5)
|
||||||
|
self.assertEqual(floored["threshold"], 0.6)
|
||||||
|
self.assertEqual(floored["counts"], {"total": 4, "accepted": 3, "abstained": 1})
|
||||||
|
self.assertAlmostEqual(floored["precisionAtCoverage"], 1.0, places=6)
|
||||||
|
self.assertAlmostEqual(floored["coverage"], 0.75, places=6)
|
||||||
|
self.assertEqual(floored["selectionSplit"], "validation")
|
||||||
|
# A 0.8 floor forbids that cut, so the harness falls back to the widest honest threshold
|
||||||
|
# instead of silently returning a threshold that violates the floor.
|
||||||
|
strict = harness.select_threshold(records, min_coverage=0.8)
|
||||||
|
self.assertEqual(strict["threshold"], 0.0)
|
||||||
|
self.assertAlmostEqual(strict["precisionAtCoverage"], 0.75, places=6)
|
||||||
|
|
||||||
|
def test_refuses_an_unreachable_coverage_floor(self):
|
||||||
|
with self.assertRaisesRegex(harness.EvaluationError, "coverage floor"):
|
||||||
|
harness.select_threshold(self.validation_records(), min_coverage=1.5)
|
||||||
|
|
||||||
|
def test_refuses_to_choose_a_threshold_on_test_or_train_data(self):
|
||||||
|
test_only = [record("e700000000000001", 4, 4, 0.9)]
|
||||||
|
with self.assertRaisesRegex(harness.EvaluationError, "validation split only"):
|
||||||
|
harness.select_threshold(test_only, min_coverage=0.5)
|
||||||
|
train_mixed = [record("aa00000000000001", 4, 4, 0.9), record("cd00000000000001", 3, 3, 0.8)]
|
||||||
|
with self.assertRaisesRegex(harness.EvaluationError, "validation split only"):
|
||||||
|
harness.select_threshold(train_mixed, min_coverage=0.5)
|
||||||
|
|
||||||
|
|
||||||
|
class ContributorHeldOutTests(unittest.TestCase):
|
||||||
|
def test_refuses_a_split_that_cannot_be_derived_from_the_subject_key(self):
|
||||||
|
# 0xaa = 170, below the train/validation edge, so this subject can only ever be train.
|
||||||
|
records = [record("aa00000000000001", 4, 4, 0.9, split="test")]
|
||||||
|
with self.assertRaisesRegex(harness.EvaluationError, "not reproducible"):
|
||||||
|
harness.assert_contributor_held_out(records)
|
||||||
|
|
||||||
|
|
||||||
|
def test_refuses_a_contributor_whose_photos_span_two_splits(self):
|
||||||
|
records = [
|
||||||
|
record("aa00000000000001", 4, 4, 0.9, split="train"),
|
||||||
|
record("aa00000000000001", 3, 3, 0.8, split="validation"),
|
||||||
|
]
|
||||||
|
with self.assertRaisesRegex(harness.EvaluationError, "leaks across"):
|
||||||
|
harness.assert_contributor_held_out(records)
|
||||||
|
|
||||||
|
def test_allows_one_contributor_to_supply_several_photos_in_the_same_split(self):
|
||||||
|
records = [
|
||||||
|
record("aa00000000000001", 4, 4, 0.9),
|
||||||
|
record("aa00000000000001", 3, 3, 0.8),
|
||||||
|
]
|
||||||
|
harness.assert_contributor_held_out(records) # several photos, one person, one split: fine
|
||||||
|
|
||||||
|
|
||||||
|
class CalibrationTests(unittest.TestCase):
|
||||||
|
def test_calibration_error_weights_each_confidence_bin_by_its_share_of_accepted_suggestions(self):
|
||||||
|
records = [
|
||||||
|
record("aa00000000000001", 4, 4, 0.95),
|
||||||
|
record("ab00000000000002", 4, 4, 0.90),
|
||||||
|
record("ac00000000000003", 3, 3, 0.55),
|
||||||
|
record("ad00000000000004", 7, 7, 0.70),
|
||||||
|
]
|
||||||
|
# Bins [0.9,1.0) hold both 0.95 and 0.90: mean confidence 0.925, accuracy 1.0, gap 0.075.
|
||||||
|
self.assertAlmostEqual(harness.evaluate(records, threshold=0.0)["ece"], 0.225, places=6)
|
||||||
|
# Raising the threshold to 0.6 drops the 0.55 sample, so it can no longer pull calibration.
|
||||||
|
self.assertAlmostEqual(harness.evaluate(records, threshold=0.6)["ece"], 0.15, places=6)
|
||||||
|
|
||||||
|
|
||||||
|
class AbstentionTests(unittest.TestCase):
|
||||||
|
def test_low_confidence_and_explicit_abstentions_shrink_coverage_but_never_count_as_errors(self):
|
||||||
|
records = [
|
||||||
|
record("aa00000000000001", 4, 4, 0.90),
|
||||||
|
record("ab00000000000002", 4, 3, 0.55),
|
||||||
|
record("ac00000000000003", 3, 3, 0.40),
|
||||||
|
record("ad00000000000004", 7, None, 0.0),
|
||||||
|
]
|
||||||
|
report = harness.evaluate(records, threshold=0.6)
|
||||||
|
self.assertEqual(report["counts"], {"total": 4, "accepted": 1, "abstained": 3})
|
||||||
|
self.assertAlmostEqual(report["coverage"], 0.25, places=6)
|
||||||
|
self.assertAlmostEqual(report["precisionAtCoverage"], 1.0, places=6)
|
||||||
|
# Only the accepted suggestion is scored; the class that abstained away is named, not hidden.
|
||||||
|
self.assertEqual(sorted(report["perClass"]), [4])
|
||||||
|
self.assertAlmostEqual(report["macroF1"], 1.0, places=6)
|
||||||
|
self.assertEqual(report["excludedClasses"], [3, 7])
|
||||||
|
self.assertEqual(report["abstentions"], {"low-confidence": 2, "no-suggestion": 1})
|
||||||
|
|
||||||
|
def test_a_threshold_above_every_confidence_abstains_on_everything_without_crashing(self):
|
||||||
|
records = [
|
||||||
|
record("aa00000000000001", 4, 4, 0.30),
|
||||||
|
record("ab00000000000002", 2, 2, 0.50),
|
||||||
|
]
|
||||||
|
report = harness.evaluate(records, threshold=0.95)
|
||||||
|
self.assertEqual(report["counts"], {"total": 2, "accepted": 0, "abstained": 2})
|
||||||
|
self.assertAlmostEqual(report["coverage"], 0.0, places=6)
|
||||||
|
self.assertAlmostEqual(report["macroF1"], 0.0, places=6)
|
||||||
|
self.assertEqual(report["perClass"], {})
|
||||||
|
|
||||||
|
|
||||||
|
class MacroF1Tests(unittest.TestCase):
|
||||||
|
def test_scores_each_bristol_class_and_averages_only_the_classes_present_in_truth(self):
|
||||||
|
records = [
|
||||||
|
record("aa00000000000001", 4, 4, 0.90),
|
||||||
|
record("ab00000000000002", 4, 3, 0.80),
|
||||||
|
record("ac00000000000003", 3, 3, 0.70),
|
||||||
|
record("ad00000000000004", 7, 7, 0.60),
|
||||||
|
]
|
||||||
|
report = harness.evaluate(records, threshold=0.0)
|
||||||
|
self.assertEqual(sorted(report["perClass"]), [3, 4, 7])
|
||||||
|
self.assertAlmostEqual(report["perClass"][4]["precision"], 1.0, places=6)
|
||||||
|
self.assertAlmostEqual(report["perClass"][4]["recall"], 0.5, places=6)
|
||||||
|
self.assertAlmostEqual(report["perClass"][4]["f1"], 0.666667, places=6)
|
||||||
|
self.assertAlmostEqual(report["perClass"][3]["precision"], 0.5, places=6)
|
||||||
|
self.assertAlmostEqual(report["perClass"][3]["recall"], 1.0, places=6)
|
||||||
|
self.assertAlmostEqual(report["perClass"][7]["f1"], 1.0, places=6)
|
||||||
|
self.assertAlmostEqual(report["macroF1"], 0.777778, places=6)
|
||||||
|
self.assertEqual(report["confusion"], {"4->4": 1, "4->3": 1, "3->3": 1, "7->7": 1})
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
unittest.main()
|
||||||
@@ -174,3 +174,20 @@ test('the deterministic symptom escalation regression suite runs in the default
|
|||||||
assert.match(builder, /"src\/domain\.js"/);
|
assert.match(builder, /"src\/domain\.js"/);
|
||||||
assert.match(builder, /"src\/hermes-agent-service\.js"/);
|
assert.match(builder, /"src\/hermes-agent-service\.js"/);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('the calibration harness is wired into the unit and syntax gates', async () => {
|
||||||
|
const [packageJsonText, workflow] = await Promise.all([
|
||||||
|
readFile(packagePath, 'utf8'),
|
||||||
|
readFile(workflowPath, 'utf8'),
|
||||||
|
]);
|
||||||
|
const packageJson = JSON.parse(packageJsonText);
|
||||||
|
|
||||||
|
// Issue #29: a release gate nothing runs is a claim, not a gate.
|
||||||
|
assert.equal(packageJson.scripts['test:calibration'], 'python3 tests/calibration-harness.test.py');
|
||||||
|
assert.match(packageJson.scripts['check:syntax'], /py_compile[^&]*scripts\/calibration_report\.py/);
|
||||||
|
assert.match(workflow, /python3 tests\/calibration-harness\.test\.py -v/);
|
||||||
|
// The release builder has to run it too, or the gate exists only in CI.
|
||||||
|
const builder = await readFile(new URL('../scripts/build_release.py', import.meta.url), 'utf8');
|
||||||
|
assert.match(builder, /tests\/calibration-harness\.test\.py/);
|
||||||
|
assert.match(builder, /"calibration_harness_gates": "passed"/);
|
||||||
|
});
|
||||||
Vendored
+31
@@ -0,0 +1,31 @@
|
|||||||
|
# Synthetic calibration fixtures
|
||||||
|
|
||||||
|
These files prove the evaluation harness in `scripts/calibration_report.py` (issue #29) computes the
|
||||||
|
release gates correctly. **They are not measurements.** No model produced them, no contributor
|
||||||
|
supplied them, and no number here says anything about how well Timmy labels a real photo.
|
||||||
|
|
||||||
|
Every row is a promise in the same shape the harness accepts:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{"model":"…","subjectKey":"<16 hex>","split":"validation","truth":{"bristolType":4},
|
||||||
|
"prediction":{"bristolType":4,"confidence":0.97}}
|
||||||
|
```
|
||||||
|
|
||||||
|
`subjectKey` values are fictional but obey the ingestion split rule (the first byte of the key picks
|
||||||
|
the split), so the harness's contributor-held-out check is exercised for real. `prediction: null`
|
||||||
|
means the candidate produced no suggestion at all.
|
||||||
|
|
||||||
|
| File | What it is for | Harness outcome |
|
||||||
|
|---|---|---|
|
||||||
|
| `predictions.jsonl` | Three candidates with invented scores: a bootstrap that calls almost everything Type 4, a frozen encoder that is precise but badly calibrated, and a specialist that looks strong on validation and slips on test. | `BLOCKED` — each candidate fails a different gate. |
|
||||||
|
| `predictions-go.jsonl` | One engineered candidate that clears every gate, including a low-confidence mistake on validation that the threshold is supposed to cut away. | `GO` — proves the passing branch and threshold selection, not a result. |
|
||||||
|
| `predictions-leaky.jsonl` | One contributor whose photos sit in both `validation` and `test`. | Refused with exit 2 — a leaked corpus is never scored. |
|
||||||
|
|
||||||
|
## Regenerating
|
||||||
|
|
||||||
|
The numbers are hand-chosen so the expected outcome is obvious by inspection; there is no generator
|
||||||
|
to re-run. If you change a row, update the expectation in `tests/calibration-harness.test.py` and the
|
||||||
|
caption in `scripts/record_release_demo.mjs` together, and read the harness output back before
|
||||||
|
claiming what it says.
|
||||||
|
|
||||||
|
No image, no diagnosis, and no patient data lives in this directory.
|
||||||
+14
@@ -0,0 +1,14 @@
|
|||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.98}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.98}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":3,"confidence":0.6}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.98}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.98}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.97}}
|
||||||
|
{"model":"gate-plumbing-fixture","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.98}}
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.7}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.7}}
|
||||||
+42
@@ -0,0 +1,42 @@
|
|||||||
|
{"model":"vlm-bootstrap","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":4,"confidence":0.55}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":4,"confidence":0.52}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.61}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.72}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":4,"confidence":0.64}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":4,"confidence":0.58}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":4,"confidence":0.53}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":4,"confidence":0.57}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":4,"confidence":0.54}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.63}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.7}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":4,"confidence":0.62}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":4,"confidence":0.59}}
|
||||||
|
{"model":"vlm-bootstrap","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":4,"confidence":0.51}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.62}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.58}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":4,"confidence":0.66}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.71}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.57}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.64}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":6,"confidence":0.55}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.6}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.59}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.63}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.69}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":6,"confidence":0.56}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.61}}
|
||||||
|
{"model":"frozen-encoder","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.54}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.88}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"ce2b3c4d5e6f7012","split":"validation","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.86}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"cf3c4d5e6f701223","split":"validation","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.9}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"d04d5e6f70122334","split":"validation","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.94}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"d15e6f7012233445","split":"validation","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.87}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"d26f701223344556","split":"validation","truth":{"bristolType":6},"prediction":{"bristolType":6,"confidence":0.91}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"d370122334455667","split":"validation","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.85}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"e671223344556677","split":"test","truth":{"bristolType":1},"prediction":{"bristolType":1,"confidence":0.86}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"e772334455667788","split":"test","truth":{"bristolType":2},"prediction":{"bristolType":2,"confidence":0.84}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"e883445566778899","split":"test","truth":{"bristolType":3},"prediction":{"bristolType":3,"confidence":0.89}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"e994556677889900","split":"test","truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.92}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"eaa5667788990011","split":"test","truth":{"bristolType":5},"prediction":{"bristolType":5,"confidence":0.88}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"ebb6778899001122","split":"test","truth":{"bristolType":6},"prediction":{"bristolType":3,"confidence":0.83}}
|
||||||
|
{"model":"specialist-candidate","subjectKey":"ecc7889900112233","split":"test","truth":{"bristolType":7},"prediction":{"bristolType":7,"confidence":0.87}}
|
||||||
@@ -389,3 +389,31 @@ test('the builder refuses to gate or record a release against a foreign server',
|
|||||||
assert.ok(identityCheck > 0, 'the served-tree check must be present');
|
assert.ok(identityCheck > 0, 'the served-tree check must be present');
|
||||||
assert.ok(firstFlow > identityCheck, 'the served-tree check must run before the first acceptance flow');
|
assert.ok(firstFlow > identityCheck, 'the served-tree check must run before the first acceptance flow');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('release demo replays the calibration harness against synthetic predictions instead of narrating the gates', async () => {
|
||||||
|
const demo = await readFile(demoPath, 'utf8');
|
||||||
|
const builder = await readFile(builderPath, 'utf8');
|
||||||
|
|
||||||
|
// Issue #29: the gate math is only proven by running the real harness on camera.
|
||||||
|
assert.match(demo, /scripts\/calibration_report\.py/);
|
||||||
|
assert.match(demo, /tests\/fixtures\/calibration\/predictions\.jsonl/);
|
||||||
|
assert.match(demo, /tests\/fixtures\/calibration\/predictions-go\.jsonl/);
|
||||||
|
assert.match(demo, /tests\/fixtures\/calibration\/predictions-leaky\.jsonl/);
|
||||||
|
// All three verdict paths are exercised: blocked, passing, and refused before scoring.
|
||||||
|
assert.match(demo, /the synthetic candidate fixtures must block the release/);
|
||||||
|
assert.match(demo, /the engineered fixture must prove the passing branch/);
|
||||||
|
assert.match(demo, /a leaked corpus must be refused before scoring/);
|
||||||
|
// The card is drawn from the report the run wrote, and gated for readability like the corpus card.
|
||||||
|
assert.match(demo, /id = 'calibration-report'/);
|
||||||
|
assert.match(demo, /Calibration card is not fully readable on screen/);
|
||||||
|
assert.match(demo, /the card must show the macro-F1 the harness measured/);
|
||||||
|
assert.match(demo, /the card must show the refusal the leaked corpus earned/);
|
||||||
|
// The scores are labelled as invented: a synthetic fixture is not a measurement of any model.
|
||||||
|
assert.match(demo, /invented scores, not measurements of any model/);
|
||||||
|
assert.match(demo, /synthetic predictions · no model was measured/);
|
||||||
|
// The scene cannot be dropped silently: its captions are release gates.
|
||||||
|
assert.match(builder, /The calibration harness is a command-line release gate, not a screen in the app/);
|
||||||
|
assert.match(builder, /Every score on this card is invented to exercise the arithmetic/);
|
||||||
|
assert.match(builder, /Each candidate is scored on test at a threshold chosen on validation only/);
|
||||||
|
assert.match(builder, /One contributor spread across two splits is refused before anything is scored/);
|
||||||
|
});
|
||||||
Reference in new issue
Block a user