# Evaluation harness contract

Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
splits that are contributor-held-out and reproducible, comparing candidate models.

The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
predictions something else produced, and it refuses to compute gates it cannot honestly compute.

The GO/BLOCKED verdict this report produces is consumed by the production abstention policy in
[`docs/ABSTENTION-POLICY.md`](ABSTENTION-POLICY.md): a candidate that has not cleared these gates
cannot authorise a visual prefill.

**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
labels, predictions, and confidences.

## Input

One prediction record per line (JSONL), one file or many:

```json
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
 "truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
```

| Field | Rule |
|---|---|
| `model` | Non-empty. Candidates are grouped by this value, so one file may carry several. |
| `subjectKey` | The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account. |
| `split` | `train`, `validation`, or `test`. |
| `truth.bristolType` | Whole number 1–7. |
| `prediction` | `{bristolType: 1–7, confidence: 0–1}`, or `null` when the candidate offered nothing. |

A record that breaks any row above is refused (exit 2) rather than scored.

## Splits are contributor-held-out and reproducible

The split rule is the ingestion rule from `scripts/ingest_training_photo.py`: the first byte of the
subject key picks the bucket — below `205` is `train`, below `230` is `validation`, otherwise `test`.
Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the
manifest, and a corpus cannot quietly re-label itself.

`assert_contributor_held_out()` refuses:

- a `split` that does not match the one the subject key derives (not reproducible), and
- one subject key appearing in more than one split (contributor leak) — a held-out score where the
  model has already seen that person is not held out.

Several photos from one contributor inside one split are fine; that is the normal case.

## Metrics

The harness is a *selective* classifier evaluation. A suggestion is **accepted** when it exists and
its confidence is at or above the threshold; everything else is an abstention. Two abstention
reasons are counted separately: `no-suggestion` and `low-confidence`.

| Metric | Definition |
|---|---|
| `coverage` | accepted / total in the scored split. |
| `precisionAtCoverage` | correct / accepted. The gate metric. |
| `perClass[label]` | `precision`, `recall`, `f1`, `support`, `predicted` over accepted suggestions. |
| `macroF1` | Mean of `perClass[*].f1` over the classes present in **accepted** truth. A class whose every photo abstained cannot be scored, so it is named in `excludedClasses` instead of being counted as a silent zero. |
| `confusion` | `"truth->predicted": count` over accepted suggestions. |
| `ece` | Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share. |

Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated
by abstaining out of its worst bins.

## Threshold protocol

`select_threshold()` takes the coverage floor and returns the threshold with the highest
precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage.
It reads the **validation split only** and raises otherwise: choosing a threshold on the data the gate
is measured on manufactures whatever precision the report needs.

`compare()` therefore fixes one threshold per candidate on validation, scores that candidate on test,
and never looks back at validation scores.

## Gates

Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.

| Gate | Limit |
|---|---|
| `macroF1` | ≥ 0.80 |
| `precisionAtCoverage` | ≥ 0.90 |
| `ece` | ≤ 0.05 |

A candidate must clear all three on its held-out split. `verdict` is `GO` only when at least one
candidate does; `recommended` is the passing candidate with the highest macro-F1 (then precision).
Otherwise the verdict is `BLOCKED` and no candidate is recommended.

## Run it

```bash
python3 scripts/calibration_report.py \
  --predictions tests/fixtures/calibration/predictions.jsonl \
  --min-coverage 0.5 --json /tmp/calibration-report.json

npm run test:calibration    # the harness contract suite
```

| Exit code | Meaning |
|---|---|
| 0 | `GO` — a candidate cleared every gate. |
| 3 | `BLOCKED` — the release gate refuses every candidate. |
| 2 | Refused — malformed, leaky, or unreproducible input. Nothing was scored or written. |

`--json PATH` writes the full report (per-class rows, confusion matrix, calibration bins, gate detail).
The report is only written after a successful run.

## Fixtures

[`tests/fixtures/calibration/`](../tests/fixtures/calibration/README.md) holds synthetic predictions
that exercise `BLOCKED`, `GO`, and the leaked-corpus refusal. They are invented numbers on fabricated
subject keys: **no model produced them and they say nothing about the real classifier.** Recording a
video or writing a report from them means saying so.