stackchain/timmy-talking-turd
2
0 Fork 0
Code Issues 21 Pull Requests 7 Actions 4 Packages Projects Releases 41 Wiki Activity

41 Releases 41 Tags

RSS Feed
  • daily-2026-10-26 2533f83af3
    Compare

    Timmy the Talking Turd — daily-2026-10-26 — quality-gated abstention
    Quality gates / quality (push) Failing after 5m13s
    Details
    Pre-Release

    rockachopa released this 2026-10-26 12:57:38 +00:00 | 24 commits to main since this release

    Prerelease for review — tag daily-2026-10-26 on commit 2533f83af3504a8aeeb2a56e1d0df1756e6b6a31.

    What changed

    Quality-gated abstention policy (issue #32, PR #87). Confidence used to be the only thing between a model answer and a prefilled journal entry, so a confident reading of a blurry photo could still be offered as fact. Photo quality is now part of the policy:

    • src/abstention-policy.js — one versioned policy (version: 1, confidence floor 0.55, accepted quality good|fair) plus the four fixed refusal strings, so the parser, the corpus, and the browser cannot drift into three explanations of the same refusal. imageQuality: "poor" abstains with low_quality however confident the reply looked.
    • src/analysis.js — parseVisionResponse(raw, policy) reads its limits and copy from that module; a caller can tighten the policy without editing the parser, which is how the tests prove the parser carries no hidden limits of its own.
    • Release-level gate — evaluatePrefillGate(report) turns the #29 calibration report into one decision: GO plus a recommended candidate whose threshold was selected on validation and scored on the held-out test split, or prefill is blocked with every reason named. A missing report is a block, never a default allow. scripts/abstention_policy_receipt.mjs exposes that as a machine receipt (exit 0 authorised / 3 blocked / 2 unreadable).
    • Corpus — new synthetic case poor-quality-confident: a 0.94-confidence answer about the committed motion-blur fixture, expected to abstain with low_quality. Existing CC0 fixture reused; no camera, no human subject, no new imagery.

    Scope note, stated plainly: the per-photo quality gate is live in the app and is what the recording shows. The release-level receipt is a command-line gate and is not yet wired as a hard gate in scripts/build_release.py — it prints the honest current state, BLOCKED, and today that is correct: there is no candidate model until #31.

    Run against the committed synthetic predictions, the receipt says:

    policy: version 1 · confidence ≥ 0.55 · quality good|fair
calibration: verdict BLOCKED · model none
prefill: BLOCKED — calibration-verdict-not-go, no-recommended-calibration

    Completed ticket / PR

    • Issue #32 — Calibrate confidence and define the production abstention policy (closed by this release). Criterion 2 ("held-out precision reaches the agreed gate or release is blocked") cannot be satisfied yet — there is no real candidate until #31 — so the gate reports BLOCKED instead of passing on synthetic numbers, which is the honest half of that criterion.
    • PR #87 Gate visual prefill on a versioned abstention policy (#32) — merged as 2533f83 after full diff review: 13 files, +428/−14, no secrets, no weights, no real imagery.

    Gates — all 24 passed

    npm test 226/226, tests/calibration-harness.test.py 19/19, tests/corpus-manifest.test.py 8/8, mobile green path, photo-first acceptance, sleek Hermes chat, manual fallback + draft recovery, provisional suggestion state, hard-stool follow-up, vision corpus refusal, abstention quality gate, worker container receipts, worker post-load readiness, longitudinal pattern summary, grounded chat citations, training contribution consent, corpus manifest provenance, calibration harness gates, portable backup restore, dependency audit (0 high), syntax (node/bash/python), secret scan, forbidden artifacts, demo H.264 720×1280 probe, complete MP4 decode, release delta verified painted on camera (78/78 captions painted).

    The new caption is also a builder-required caption now, so a future release cannot drop the quality gate silently.

    Artifacts

    • source archive timmy-talking-turd-2026-10-26-2533f83af350.tar.gz — 297,679 bytes — sha256 7af46dc5840da8751809f6e307a4175e97dd9e0150fe34c78eff4841b637a81e
    • demo video timmy-talking-turd-2026-10-26-2533f83af350-demo.mp4 — 22,232,190 bytes — H.264 720×1280 yuv420p, 350.3 s — sha256 9cd0b7374a7f88e5c1950de7d2d1254c3eac184fb70a24dba3e86220a45e9a6b

    Model / profile

    Recorded against the real app running from the built commit with TIMMY_VISION_ENABLED=false; the demo replays the committed corpus provider responses through the app's real analysis path, which is why a 0.94-confidence "poor quality" reply can be exercised deterministically. No live model inference appears in the video, no real photo is used anywhere, and the visible fixtures are synthetic (CC0, generated by scripts/build_vision_corpus.py).

    Demonstrated on camera

    The whole existing user path (photo log with provisional suggestion, hard-stool follow-up and red-flag escalation, vision-offline fallback, corpus refusals and abstentions, explainable pattern summary with citations, grounded chat with source refs, private photo lifecycle, opt-in training contribution with withdrawal, versioned backup/restore/refusal, calibration card, corpus audit card) plus this release's new scene at ~1:59:

    • the synthetic blur fixture is chosen, consent is ticked, analysis runs;
    • the suggestion sheet says "No confident match." and "The photo is too blurry or poorly lit to suggest a form safely." — the provider answer was confident and is discarded on quality;
    • the only ways forward are "Choose the form yourself" and "Try another photo": nothing is prefilled, no Bristol form appears, and the demo asserts that absence in the browser rather than only claiming it in a caption.

    Known accuracy limitation

    Form, broad colour, and image-quality suggestions remain observable-field assistance, never diagnosis; user confirmation stays mandatory and a red flag still escalates in the app's own fixed words. Bristol accuracy is not clinically validated, the calibration fixtures are synthetic, and the release-level prefill gate reports BLOCKED because no candidate has cleared the #29 gates yet. Open-weight VLM acceptance is proven; clinical accuracy is not.

    The source package excludes model weights, medical-image corpora, raw research pages, generated screenshots, and video production assets.

    Downloads