timmy
209e21a951
Confidence was the only thing standing between a model answer and a prefilled journal entry, so a confident reading of a blurry photo could still be offered as fact. Quality is now part of the policy: `poor` abstains however confident the reply looked, with its own fixed copy. The confidence floor, accepted quality levels, and refusal copy move into src/abstention-policy.js so the parser, the corpus, and the browser cannot drift into three explanations of the same refusal. parseVisionResponse takes an optional policy, which is how the tests prove the parser carries no hidden limits of its own. evaluatePrefillGate turns the #29 calibration report into one release decision: GO plus a validation-selected threshold scored on the held-out test split, or prefill is blocked with each reason named. A missing report is a block, never a default allow. scripts/abstention_policy_receipt.mjs exposes that as a receipt (exit 0 / 3 / 2). The committed synthetic predictions currently report BLOCKED and the receipt says so. No candidate exists until #31, and the builder does not hard-gate on the receipt yet; the on-camera quality gate is the evidence this release carries. The corpus gains a case that replays a 0.94-confidence answer about the synthetic motion-blur fixture, and the release demo asserts in the browser that nothing is prefilled before captioning it. Verified: 226 unit tests, calibration harness (19), corpus manifest (8), vision-refusal, photo-first, and provisional acceptance, check:syntax, check:diff, and a full real-browser demo recording whose caption receipt confirms the new scene painted on screen.
5.4 KiB
Evaluation harness contract
Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over splits that are contributor-held-out and reproducible, comparing candidate models.
The implementation is scripts/calibration_report.py. It reads
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
The GO/BLOCKED verdict this report produces is consumed by the production abstention policy in
docs/ABSTENTION-POLICY.md: a candidate that has not cleared these gates
cannot authorise a visual prefill.
This harness does not train, does not diagnose, and never reads an image. It only sees Bristol labels, predictions, and confidences.
Input
One prediction record per line (JSONL), one file or many:
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
"truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
| Field | Rule |
|---|---|
model |
Non-empty. Candidates are grouped by this value, so one file may carry several. |
subjectKey |
The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account. |
split |
train, validation, or test. |
truth.bristolType |
Whole number 1–7. |
prediction |
{bristolType: 1–7, confidence: 0–1}, or null when the candidate offered nothing. |
A record that breaks any row above is refused (exit 2) rather than scored.
Splits are contributor-held-out and reproducible
The split rule is the ingestion rule from scripts/ingest_training_photo.py: the first byte of the
subject key picks the bucket — below 205 is train, below 230 is validation, otherwise test.
Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the
manifest, and a corpus cannot quietly re-label itself.
assert_contributor_held_out() refuses:
- a
splitthat does not match the one the subject key derives (not reproducible), and - one subject key appearing in more than one split (contributor leak) — a held-out score where the model has already seen that person is not held out.
Several photos from one contributor inside one split are fine; that is the normal case.
Metrics
The harness is a selective classifier evaluation. A suggestion is accepted when it exists and
its confidence is at or above the threshold; everything else is an abstention. Two abstention
reasons are counted separately: no-suggestion and low-confidence.
| Metric | Definition |
|---|---|
coverage |
accepted / total in the scored split. |
precisionAtCoverage |
correct / accepted. The gate metric. |
perClass[label] |
precision, recall, f1, support, predicted over accepted suggestions. |
macroF1 |
Mean of perClass[*].f1 over the classes present in accepted truth. A class whose every photo abstained cannot be scored, so it is named in excludedClasses instead of being counted as a silent zero. |
confusion |
"truth->predicted": count over accepted suggestions. |
ece |
Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share. |
Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated by abstaining out of its worst bins.
Threshold protocol
select_threshold() takes the coverage floor and returns the threshold with the highest
precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage.
It reads the validation split only and raises otherwise: choosing a threshold on the data the gate
is measured on manufactures whatever precision the report needs.
compare() therefore fixes one threshold per candidate on validation, scores that candidate on test,
and never looks back at validation scores.
Gates
Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.
| Gate | Limit |
|---|---|
macroF1 |
≥ 0.80 |
precisionAtCoverage |
≥ 0.90 |
ece |
≤ 0.05 |
A candidate must clear all three on its held-out split. verdict is GO only when at least one
candidate does; recommended is the passing candidate with the highest macro-F1 (then precision).
Otherwise the verdict is BLOCKED and no candidate is recommended.
Run it
python3 scripts/calibration_report.py \
--predictions tests/fixtures/calibration/predictions.jsonl \
--min-coverage 0.5 --json /tmp/calibration-report.json
npm run test:calibration # the harness contract suite
| Exit code | Meaning |
|---|---|
| 0 | GO — a candidate cleared every gate. |
| 3 | BLOCKED — the release gate refuses every candidate. |
| 2 | Refused — malformed, leaky, or unreproducible input. Nothing was scored or written. |
--json PATH writes the full report (per-class rows, confusion matrix, calibration bins, gate detail).
The report is only written after a successful run.
Fixtures
tests/fixtures/calibration/ holds synthetic predictions
that exercise BLOCKED, GO, and the leaked-corpus refusal. They are invented numbers on fabricated
subject keys: no model produced them and they say nothing about the real classifier. Recording a
video or writing a report from them means saying so.