Build contributor-held-out evaluation harness and calibration report #29

New Issue

timmy · 2026-08-19T01:59:01Z

timmy commented

2026-08-19 01:59:01 +00:00

Parent epic: #4

Outcome

Create the authoritative comparison and release-gate harness.

Acceptance criteria

Reports macro-F1, per-class precision/recall, confusion, calibration error, coverage, and precision-at-coverage.
Splits are contributor-held-out and reproducible.
Harness compares VLM bootstrap, frozen encoder, and specialist checkpoints.

Dependencies

#27 — Publish dataset card, labeling guide, and contributor-level split policy

Required evidence

Test or executable receipt attached to the issue/PR
Privacy/safety boundary checked where applicable
Clean checkout and relevant full suite pass

Delivery

Open a focused branch and PR with Closes #ISSUE. Do not include real medical images in issue comments or Git history.

Parent epic: #4 ## Outcome Create the authoritative comparison and release-gate harness. ## Acceptance criteria - [ ] Reports macro-F1, per-class precision/recall, confusion, calibration error, coverage, and precision-at-coverage. - [ ] Splits are contributor-held-out and reproducible. - [ ] Harness compares VLM bootstrap, frozen encoder, and specialist checkpoints. ## Dependencies - #27 — Publish dataset card, labeling guide, and contributor-level split policy ## Required evidence - Test or executable receipt attached to the issue/PR - Privacy/safety boundary checked where applicable - Clean checkout and relevant full suite pass ## Delivery Open a focused branch and PR with `Closes #ISSUE`. Do not include real medical images in issue comments or Git history.

timmy added this to the M2 — Consented Dataset & Specialist Model milestone 2026-08-19 01:59:01 +00:00

timmy added the

labels 2026-08-19 01:59:01 +00:00

timmy referenced this issue

2026-08-19 01:59:01 +00:00

Train frozen-encoder baseline with stool/not-stool and Type 1–7 heads #30

timmy referenced this issue

2026-08-19 01:59:03 +00:00