stackchain/timmy-talking-turd
2
0 Fork 0
Code Issues 21 Pull Requests 7 Actions 4 Packages Projects Releases 41 Wiki Activity

41 Releases 41 Tags

RSS Feed
  • daily-2026-10-25 2a10a20518
    Compare

    Timmy the Talking Turd — daily-2026-10-25
    Quality gates / quality (push) Successful in 12m51s
    Details
    Pre-Release

    rockachopa released this 2026-10-25 12:42:05 +00:00 | 26 commits to main since this release

    Prerelease for review — tag daily-2026-10-25 on commit 2a10a20518603325a85740d0259d4b20d788862b.

    What changed

    Contributor-held-out evaluation harness (issue #29, PR #86). scripts/calibration_report.py is the comparison and release-gate tool the ROADMAP gates were written for:

    • Reports macro-F1, per-class precision/recall, a truth->predicted confusion matrix, expected calibration error over ten equal-width confidence bins, coverage, and precision-at-coverage — abstentions split into no-suggestion and low-confidence.
    • Splits are contributor-held-out and reproducible: a subject key must derive from the existing ingestion rule (first byte of the pseudonymous digest: <205 train, <230 validation, else test), and one contributor appearing in two splits is refused before anything is scored.
    • Compares candidate fixtures under three ids: one threshold per candidate is chosen on validation only, then the candidate is scored on test.
    • Exit codes: 0 = GO, 3 = BLOCKED, 2 = refused (malformed, leaky, or unreproducible input — nothing scored, no report written).
    • Classes whose every photo abstained are named in excludedClasses instead of counting as silent zeros, so abstention cannot inflate or deflate the score unnoticed.

    Gates are the ROADMAP values, inclusive: macroF1 >= 0.80, precision@coverage >= 0.90, ECE <= 0.05.

    Scope note, stated plainly: this is a command-line release gate, not a screen in the app. No app UI changed and this release does not imply a new app feature. The recorded demo runs the real CLI against the committed synthetic fixtures and shows the receipt it wrote.

    A follow-on commit (2a10a20) fixes the demo card layout only — each candidate's name now gets its own row, so specialist-candidate is no longer broken at its hyphen. No app, server, or harness behaviour changes in that commit.

    Gates — all 23 passed

    npm test 219/219, tests/calibration-harness.test.py 19/19, tests/corpus-manifest.test.py 8/8, mobile green path, photo-first acceptance, sleek Hermes chat, manual fallback + draft recovery, provisional suggestion state, hard-stool follow-up, vision corpus refusal, worker container receipts, worker post-load readiness, longitudinal pattern summary, grounded chat citations, training contribution consent, corpus manifest provenance, calibration harness gates, portable backup restore, dependency audit (0 high), syntax (node/bash/python), secret scan, forbidden artifacts, demo H.264 720x1280 probe, complete MP4 decode, release delta verified painted on camera (77/77 captions painted).

    The harness suite is a release gate, not just CI: scripts/build_release.py runs tests/calibration-harness.test.py before anything is built.

    Artifacts

    • source archive timmy-talking-turd-2026-10-25-2a10a2051860.tar.gz — 291,771 bytes — sha256 6ad4329d363df65e37a5bd2b4bb8a8660039ad3e516da9402bef62b4d170a920
    • demo video timmy-talking-turd-2026-10-25-2a10a2051860-demo.mp4 — 22,983,816 bytes — H.264 720x1280 yuv420p, 349.7 s — sha256 524d02edc87db89dbc5a0f09934af6586d0526c3fd1645ba660f152304dcf02a

    Model / profile

    Recorded against the real app running from the built commit with TIMMY_VISION_ENABLED=false; the demo uses the app's deterministic local profile for vision and chat. No live model inference appears in the video and no real photo is used anywhere.

    Demonstrated on camera

    The whole existing user path (photo log with provisional suggestion, hard-stool follow-up and red-flag escalation, vision-offline fallback, corpus refusals and abstentions, explainable pattern summary with citations, grounded chat with source refs, private photo lifecycle, opt-in training contribution with withdrawal, versioned backup/restore/refusal) plus this release's new card:

    • three synthetic candidates, each failing a different gate → BLOCKED, nobody recommended;
    • a fixture engineered to clear every gate → GO, labelled "fixture, not a result";
    • one contributor spread across validation and test → refused, exit 2, "leaks across test, validation: splits must be contributor-held-out".

    Card values read back from the run: frozen-encoder thr 0.57 / F1 1.000 / P@C 1.000 / cov 0.71 / ECE 0.376 (fails ece); specialist-candidate thr 0.00 / F1 0.810 / P@C 0.857 / cov 1.00 / ECE 0.036 (fails precisionAtCoverage); vlm-bootstrap thr 0.58 / F1 0.100 / P@C 0.250 / cov 0.57 / ECE 0.535 (fails macroF1, precisionAtCoverage, ece).

    Known accuracy limitation

    The fixture scores are invented to exercise the arithmetic: no model was measured against these gates, and the harness is not evidence about classifier accuracy. Form, broad colour, and image-quality suggestions remain observable-field assistance, never diagnosis; user confirmation stays mandatory; a red flag escalates in the app's own fixed words. Bristol accuracy is not clinically validated on this corpus.

    The source package excludes model weights, medical-image corpora, raw research pages, generated screenshots, and video production assets.

    Downloads