stackchain/timmy-talking-turd
2
0 Fork 0
Code Issues 21 Pull Requests 7 Actions 4 Packages Projects Releases 41 Wiki Activity

Build contributor-held-out evaluation harness and calibration report #29

New Issue
Closed
opened 2026-08-19 01:59:01 +00:00 by timmy · 0 comments
timmy commented 2026-08-19 01:59:01 +00:00
Owner
Copy link
Copy source

Parent epic: #4

Outcome

Create the authoritative comparison and release-gate harness.

Acceptance criteria

  • Reports macro-F1, per-class precision/recall, confusion, calibration error, coverage, and precision-at-coverage.
  • Splits are contributor-held-out and reproducible.
  • Harness compares VLM bootstrap, frozen encoder, and specialist checkpoints.

Dependencies

  • #27 — Publish dataset card, labeling guide, and contributor-level split policy

Required evidence

  • Test or executable receipt attached to the issue/PR
  • Privacy/safety boundary checked where applicable
  • Clean checkout and relevant full suite pass

Delivery

Open a focused branch and PR with Closes #ISSUE. Do not include real medical images in issue comments or Git history.

Parent epic: #4 ## Outcome Create the authoritative comparison and release-gate harness. ## Acceptance criteria - [ ] Reports macro-F1, per-class precision/recall, confusion, calibration error, coverage, and precision-at-coverage. - [ ] Splits are contributor-held-out and reproducible. - [ ] Harness compares VLM bootstrap, frozen encoder, and specialist checkpoints. ## Dependencies - #27 — Publish dataset card, labeling guide, and contributor-level split policy ## Required evidence - Test or executable receipt attached to the issue/PR - Privacy/safety boundary checked where applicable - Clean checkout and relevant full suite pass ## Delivery Open a focused branch and PR with `Closes #ISSUE`. Do not include real medical images in issue comments or Git history.
timmy added this to the M2 — Consented Dataset & Specialist Model milestone 2026-08-19 01:59:01 +00:00
timmy added the type/testpriority/P0area/modelarea/data labels 2026-08-19 01:59:01 +00:00
rockachopa added the state:review label 2026-10-25 11:06:08 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Labels
Clear labels
area/backend
Application API and storage
 area/data
Dataset and provenance
 area/frontend
PWA and interaction design
 area/model
Vision model and inference
 area/privacy
Privacy and consent
 area/release
Release engineering and distribution
 area/safety
Clinical safety boundary
 blocked
Blocked on an explicit dependency or decision
 human-gate
Requires explicit human approval
 priority/P0
Release-critical
 priority/P1
High priority
 priority/P2
Planned
 state:review
type/docs
Documentation and evidence
 type/epic
Roadmap epic
 type/feature
Product feature
 type/ops
Infrastructure and operations
 type/research
Research or evaluation
 type/security
Security hardening
 type/test
Verification and quality gate
No labels area/data area/model priority/P0 state:review type/test
Milestone
No items
No Milestone M2 — Consented Dataset & Specialist Model
Projects
Clear projects
No projects
Assignees
Clear assignees
timmy (Timmy · the Wizard Mind) zephyrus-sparkborn (Zephyrus Sparkborn)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#29