timmy
209e21a951
Quality gates / quality (pull_request) Successful in 13m29s
Confidence was the only thing standing between a model answer and a prefilled journal entry, so a confident reading of a blurry photo could still be offered as fact. Quality is now part of the policy: `poor` abstains however confident the reply looked, with its own fixed copy. The confidence floor, accepted quality levels, and refusal copy move into src/abstention-policy.js so the parser, the corpus, and the browser cannot drift into three explanations of the same refusal. parseVisionResponse takes an optional policy, which is how the tests prove the parser carries no hidden limits of its own. evaluatePrefillGate turns the #29 calibration report into one release decision: GO plus a validation-selected threshold scored on the held-out test split, or prefill is blocked with each reason named. A missing report is a block, never a default allow. scripts/abstention_policy_receipt.mjs exposes that as a receipt (exit 0 / 3 / 2). The committed synthetic predictions currently report BLOCKED and the receipt says so. No candidate exists until #31, and the builder does not hard-gate on the receipt yet; the on-camera quality gate is the evidence this release carries. The corpus gains a case that replays a 0.94-confidence answer about the synthetic motion-blur fixture, and the release demo asserts in the browser that nothing is prefilled before captioning it. Verified: 226 unit tests, calibration harness (19), corpus manifest (8), vision-refusal, photo-first, and provisional acceptance, check:syntax, check:diff, and a full real-browser demo recording whose caption receipt confirms the new scene painted on screen.
118 lines
5.4 KiB
Markdown
118 lines
5.4 KiB
Markdown
# Evaluation harness contract
|
||
|
||
Issue #29 asks for the authoritative comparison and release-gate harness: macro-F1, per-class
|
||
precision/recall, a confusion matrix, calibration error, coverage, and precision-at-coverage, over
|
||
splits that are contributor-held-out and reproducible, comparing candidate models.
|
||
|
||
The implementation is [`scripts/calibration_report.py`](../scripts/calibration_report.py). It reads
|
||
predictions something else produced, and it refuses to compute gates it cannot honestly compute.
|
||
|
||
The GO/BLOCKED verdict this report produces is consumed by the production abstention policy in
|
||
[`docs/ABSTENTION-POLICY.md`](ABSTENTION-POLICY.md): a candidate that has not cleared these gates
|
||
cannot authorise a visual prefill.
|
||
|
||
**This harness does not train, does not diagnose, and never reads an image.** It only sees Bristol
|
||
labels, predictions, and confidences.
|
||
|
||
## Input
|
||
|
||
One prediction record per line (JSONL), one file or many:
|
||
|
||
```json
|
||
{"model":"specialist-candidate","subjectKey":"cd1a2b3c4d5e6f01","split":"validation",
|
||
"truth":{"bristolType":4},"prediction":{"bristolType":4,"confidence":0.97}}
|
||
```
|
||
|
||
| Field | Rule |
|
||
|---|---|
|
||
| `model` | Non-empty. Candidates are grouped by this value, so one file may carry several. |
|
||
| `subjectKey` | The 16-hex pseudonymous key from the corpus manifest. Never a name, email, or account. |
|
||
| `split` | `train`, `validation`, or `test`. |
|
||
| `truth.bristolType` | Whole number 1–7. |
|
||
| `prediction` | `{bristolType: 1–7, confidence: 0–1}`, or `null` when the candidate offered nothing. |
|
||
|
||
A record that breaks any row above is refused (exit 2) rather than scored.
|
||
|
||
## Splits are contributor-held-out and reproducible
|
||
|
||
The split rule is the ingestion rule from `scripts/ingest_training_photo.py`: the first byte of the
|
||
subject key picks the bucket — below `205` is `train`, below `230` is `validation`, otherwise `test`.
|
||
Because the rule reads only the pseudonymous key, a split can be re-derived by anyone holding the
|
||
manifest, and a corpus cannot quietly re-label itself.
|
||
|
||
`assert_contributor_held_out()` refuses:
|
||
|
||
- a `split` that does not match the one the subject key derives (not reproducible), and
|
||
- one subject key appearing in more than one split (contributor leak) — a held-out score where the
|
||
model has already seen that person is not held out.
|
||
|
||
Several photos from one contributor inside one split are fine; that is the normal case.
|
||
|
||
## Metrics
|
||
|
||
The harness is a *selective* classifier evaluation. A suggestion is **accepted** when it exists and
|
||
its confidence is at or above the threshold; everything else is an abstention. Two abstention
|
||
reasons are counted separately: `no-suggestion` and `low-confidence`.
|
||
|
||
| Metric | Definition |
|
||
|---|---|
|
||
| `coverage` | accepted / total in the scored split. |
|
||
| `precisionAtCoverage` | correct / accepted. The gate metric. |
|
||
| `perClass[label]` | `precision`, `recall`, `f1`, `support`, `predicted` over accepted suggestions. |
|
||
| `macroF1` | Mean of `perClass[*].f1` over the classes present in **accepted** truth. A class whose every photo abstained cannot be scored, so it is named in `excludedClasses` instead of being counted as a silent zero. |
|
||
| `confusion` | `"truth->predicted": count` over accepted suggestions. |
|
||
| `ece` | Expected calibration error over accepted suggestions in ten equal-width confidence bins, weighted by each bin's share. |
|
||
|
||
Calibration is measured on the same accepted set as precision, so a candidate cannot look calibrated
|
||
by abstaining out of its worst bins.
|
||
|
||
## Threshold protocol
|
||
|
||
`select_threshold()` takes the coverage floor and returns the threshold with the highest
|
||
precision-at-coverage among the cuts that still clear that floor, breaking ties toward wider coverage.
|
||
It reads the **validation split only** and raises otherwise: choosing a threshold on the data the gate
|
||
is measured on manufactures whatever precision the report needs.
|
||
|
||
`compare()` therefore fixes one threshold per candidate on validation, scores that candidate on test,
|
||
and never looks back at validation scores.
|
||
|
||
## Gates
|
||
|
||
Taken from the ROADMAP release gates. All limits are inclusive — a candidate exactly on the line passes.
|
||
|
||
| Gate | Limit |
|
||
|---|---|
|
||
| `macroF1` | ≥ 0.80 |
|
||
| `precisionAtCoverage` | ≥ 0.90 |
|
||
| `ece` | ≤ 0.05 |
|
||
|
||
A candidate must clear all three on its held-out split. `verdict` is `GO` only when at least one
|
||
candidate does; `recommended` is the passing candidate with the highest macro-F1 (then precision).
|
||
Otherwise the verdict is `BLOCKED` and no candidate is recommended.
|
||
|
||
## Run it
|
||
|
||
```bash
|
||
python3 scripts/calibration_report.py \
|
||
--predictions tests/fixtures/calibration/predictions.jsonl \
|
||
--min-coverage 0.5 --json /tmp/calibration-report.json
|
||
|
||
npm run test:calibration # the harness contract suite
|
||
```
|
||
|
||
| Exit code | Meaning |
|
||
|---|---|
|
||
| 0 | `GO` — a candidate cleared every gate. |
|
||
| 3 | `BLOCKED` — the release gate refuses every candidate. |
|
||
| 2 | Refused — malformed, leaky, or unreproducible input. Nothing was scored or written. |
|
||
|
||
`--json PATH` writes the full report (per-class rows, confusion matrix, calibration bins, gate detail).
|
||
The report is only written after a successful run.
|
||
|
||
## Fixtures
|
||
|
||
[`tests/fixtures/calibration/`](../tests/fixtures/calibration/README.md) holds synthetic predictions
|
||
that exercise `BLOCKED`, `GO`, and the leaked-corpus refusal. They are invented numbers on fabricated
|
||
subject keys: **no model produced them and they say nothing about the real classifier.** Recording a
|
||
video or writing a report from them means saying so.
|