timmy-talking-turd/docs/EVALUATION-HARNESS.md

Evaluation harness contract

Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over splits that are contributor-held-out and reproducible, comparing candidate models.

The implementation is scripts/calibration_report.py. It reads predictions something else produced, and it refuses to compute gates it cannot honestly compute.

This harness does not train, does not diagnose, and never reads an image. It only sees Bristol labels, predictions, and confidences.

Input

One prediction record per line (JSONL), one file or many:

{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
 "truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}

Field	Rule
model	Non-empty. Candidates are grouped by this value, so one file may carry several.
subjectKey	The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account.
split	train, validation, or test.
truth.bristolType	Whole number 1–7.
prediction	{bristolType: 1–7, confidence: 0–1}, or null when the candidate offered nothing.

A record that breaks any row above is refused (exit 2) rather than scored.

Splits are contributor-held-out and reproducible

The split rule is the ingestion rule from scripts/ingest_training_photo.py: the first byte of the subject key picks the bucket — below 205 is train, below 230 is validation, otherwise test. Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the manifest, and a corpus cannot quietly re-label itself.

assert_contributor_held_out() refuses:

• a split that does not match the one the subject key derives (not reproducible), and
• one subject key appearing in more than one split (contributor leak) — a held-out score where the model has already seen that person is not held out.

Several photos from one contributor inside one split are fine; that is the normal case.

Metrics

The harness is a selective classifier evaluation. A suggestion is accepted when it exists and its confidence is at or above the threshold; everything else is an abstention. Two abstention reasons are counted separately: no-suggestion and low-confidence.

Metric	Definition
coverage	accepted / total in the scored split.
precisionAtCoverage	correct / accepted. The gate metric.
perClass[label]	precision, recall, f1, support, predicted over accepted suggestions.
macroF1	Mean of perClass[*].f1 over the classes present in accepted truth. A class whose every photo abstained cannot be scored, so it is named in excludedClasses instead of being counted as a silent zero.
confusion	"truth->predicted": count over accepted suggestions.
ece	Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share.

Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated by abstaining out of its worst bins.

Threshold protocol

select_threshold() takes the coverage floor and returns the threshold with the highest precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage. It reads the validation split only and raises otherwise: choosing a threshold on the data the gate is measured on manufactures whatever precision the report needs.

compare() therefore fixes one threshold per candidate on validation, scores that candidate on test, and never looks back at validation scores.

Gates

Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.

Gate	Limit
macroF1	≥ 0.80
precisionAtCoverage	≥ 0.90
ece	≤ 0.05

A candidate must clear all three on its held-out split. verdict is GO only when at least one candidate does; recommended is the passing candidate with the highest macro-F1 (then precision). Otherwise the verdict is BLOCKED and no candidate is recommended.

Run it

python3 scripts/calibration_report.py \
  --predictions tests/fixtures/calibration/predictions.jsonl \
  --min-coverage 0.5 --json /tmp/calibration-report.json

npm run test:calibration    # the harness contract suite

Exit code	Meaning
0	GO — a candidate cleared every gate.
3	BLOCKED — the release gate refuses every candidate.
2	Refused — malformed, leaky, or unreproducible input. Nothing was scored or written.

--json PATH writes the full report (per-class rows, confusion matrix, calibration bins, gate detail). The report is only written after a successful run.

Fixtures

tests/fixtures/calibration/ holds synthetic predictions that exercise BLOCKED, GO, and the leaked-corpus refusal. They are invented numbers on fabricated subject keys: no model produced them and they say nothing about the real classifier. Recording a video or writing a report from them means saying so.