feat(eval): score candidates against the release gates on held-out contributors (#29)

The ROADMAP gates (macro-F1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05) were prose with no
tool behind them. scripts/calibration_report.py now computes them from prediction records and
refuses input it cannot honestly score:

- a split is re-derived from the pseudonymous subject key with the ingestion rule, so a corpus
  cannot re-label its own splits; a contributor spanning two splits is refused before scoring
- the threshold is selected on validation only, and selecting on test or train is refused
- coverage, precision-at-coverage, macro-F1, per-class precision/recall, confusion, and expected
  calibration error are computed on accepted suggestions, with classes that abstained away named
  in excludedClasses instead of being scored as a silent zero
- exit codes carry the verdict: 0 GO, 3 BLOCKED, 2 refused

The release demo records the real harness against committed synthetic fixtures: a blocked
three-candidate comparison, a fixture engineered to clear every gate, and a leaked corpus refused.
Every number on the card is read back from the report the run wrote, and the captions state that
no model was measured, because a fixture is not evidence about accuracy.

The harness suite is wired into CI, check:syntax, and the release builder's own gates.

docs/EVALUATION-HARNESS.md

@@ -0,0 +1,113 @@
+# Evaluation harness contract
+
+Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
+precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
+splits that are contributor-held-out and reproducible, comparing candidate models.
+
+The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
+predictions something else produced, and it refuses to compute gates it cannot honestly compute.
+
+**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
+labels, predictions, and confidences.
+
+## Input
+
+One prediction record per line (JSONL), one file or many:
+
+```json
+{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
+ "truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
+```
+
+| Field | Rule |
+|---|---|
+| `model` | Non-empty. Candidates are grouped by this value, so one file may carry several. |
+| `subjectKey` | The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account. |
+| `split` | `train`, `validation`, or `test`. |
+| `truth.bristolType` | Whole number 1–7. |
+| `prediction` | `{bristolType: 1–7, confidence: 0–1}`, or `null` when the candidate offered nothing. |
+
+A record that breaks any row above is refused (exit 2) rather than scored.
+
+## Splits are contributor-held-out and reproducible
+
+The split rule is the ingestion rule from `scripts/ingest_training_photo.py`: the first byte of the
+subject key picks the bucket — below `205` is `train`, below `230` is `validation`, otherwise `test`.
+Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the
+manifest, and a corpus cannot quietly re-label itself.
+
+`assert_contributor_held_out()` refuses:
+
+- a `split` that does not match the one the subject key derives (not reproducible), and
+- one subject key appearing in more than one split (contributor leak) — a held-out score where the
+  model has already seen that person is not held out.
+
+Several photos from one contributor inside one split are fine; that is the normal case.
+
+## Metrics
+
+The harness is a *selective* classifier evaluation. A suggestion is **accepted** when it exists and
+its confidence is at or above the threshold; everything else is an abstention. Two abstention
+reasons are counted separately: `no-suggestion` and `low-confidence`.
+
+| Metric | Definition |
+|---|---|
+| `coverage` | accepted / total in the scored split. |
+| `precisionAtCoverage` | correct / accepted. The gate metric. |
+| `perClass[label]` | `precision`, `recall`, `f1`, `support`, `predicted` over accepted suggestions. |
+| `macroF1` | Mean of `perClass[*].f1` over the classes present in **accepted** truth. A class whose every photo abstained cannot be scored, so it is named in `excludedClasses` instead of being counted as a silent zero. |
+| `confusion` | `"truth->predicted": count` over accepted suggestions. |
+| `ece` | Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share. |
+
+Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated
+by abstaining out of its worst bins.
+
+## Threshold protocol
+
+`select_threshold()` takes the coverage floor and returns the threshold with the highest
+precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage.
+It reads the **validation split only** and raises otherwise: choosing a threshold on the data the gate
+is measured on manufactures whatever precision the report needs.
+
+`compare()` therefore fixes one threshold per candidate on validation, scores that candidate on test,
+and never looks back at validation scores.
+
+## Gates
+
+Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.
+
+| Gate | Limit |
+|---|---|
+| `macroF1` | ≥ 0.80 |
+| `precisionAtCoverage` | ≥ 0.90 |
+| `ece` | ≤ 0.05 |
+
+A candidate must clear all three on its held-out split. `verdict` is `GO` only when at least one
+candidate does; `recommended` is the passing candidate with the highest macro-F1 (then precision).
+Otherwise the verdict is `BLOCKED` and no candidate is recommended.
+
+## Run it
+
+```bash
+python3 scripts/calibration_report.py \
+  --predictions tests/fixtures/calibration/predictions.jsonl \
+  --min-coverage 0.5 --json /tmp/calibration-report.json
+
+npm run test:calibration    # the harness contract suite
+```
+
+| Exit code | Meaning |
+|---|---|
+| 0 | `GO` — a candidate cleared every gate. |
+| 3 | `BLOCKED` — the release gate refuses every candidate. |
+| 2 | Refused — malformed, leaky, or unreproducible input. Nothing was scored or written. |
+
+`--json PATH` writes the full report (per-class rows, confusion matrix, calibration bins, gate detail).
+The report is only written after a successful run.
+
+## Fixtures
+
+[`tests/fixtures/calibration/`](../tests/fixtures/calibration/README.md) holds synthetic predictions
+that exercise `BLOCKED`, `GO`, and the leaked-corpus refusal. They are invented numbers on fabricated
+subject keys: **no model produced them and they say nothing about the real classifier.** Recording a
+video or writing a report from them means saying so.