stackchain/timmy-talking-turd
2
0 Fork 0
Code Issues 21 Pull Requests 7 Actions 4 Packages Projects Releases 41 Wiki Activity

Expand deterministic symptom escalation regression suite #62

Open
rockachopa wants to merge 3 commits from timmy/21-escalation-regression-suite into main
pull from: timmy/21-escalation-regression-suite
merge into: stackchain:main
Conversation 3 Commits 3 Files Changed 6
+1285 -14
rockachopa commented 2026-08-22 20:59:19 +00:00
Member
Copy link
Copy source

Closes #21 (parent epic #5).

What

Prove model behavior can never suppress or soften urgent symptom handling, per the issue's acceptance criteria.

Acceptance criteria coverage

  • Each red flag has positive and negative tests. tests/symptom-escalation.regression.test.js pins all six flags (blood, blackOrDarkRed, severePain, vomiting, fever, cannotPassGas): 56 positive chat expressions mapped to their exact flag, plus checkbox-level positive/negative tests per flag, and 27 idiomatic negatives that must NOT escalate (I threw up my hands, yellow fever outbreak in history class, black tea, cannot pass the salt). Matrix size is pinned so silent shrinkage fails CI.
  • Model suggestions cannot set or clear symptoms. Hostile vision payloads carrying symptoms, flags, or urgent keys are dropped by the parser; mergeVisualSuggestion touches only Bristol type and color and can neither inject nor wipe user-reported symptoms. Ledger sanitizer coerces symptom values to strict booleans, so truthy junk ("yes", 1, {forced:true}, [true]) cannot fabricate an escalation while a forged true still triggers it.
  • Serious copy remains deterministic when provider output is missing or malicious. urgentChatMessage is a frozen module constant exported read-only (urgentSymptomCopy); tests prove null/undefined/empty/prompt-injection provider replies cannot alter it. The boundary suite proves every red-flag phrase, confirmed ledger flag, and ledger note intercepts chat with zero Hermes calls (runTurn spy stays empty) — the authoritative invariant.

Existing matrix preserved

The original 24-expression vomiting/blood/pain/gas matrix from tests/domain.test.js is fully retained and passing; the new suite extends it to 56 positives + 27 negatives. All prior domain/service/browser expectations unchanged.

RED evidence

Before the fix, the new suites failed with: URGENT_MESSAGE not exported; misses on bleeding from my rectum, bloody stool today, I have bloody poop, severe pain in my abdomen; false positives on I threw up my hands / throwing up confetti (idiomatic) and unbounded fever matches (Malaria fever research).

GREEN evidence

  • npm test: 87/87 pass (was 67; +20 new tests across three suites)
  • npm run check:syntax: clean
  • npm run check:diff: clean
  • npm audit --audit-level=high: 0 vulnerabilities
  • python3 tests/staging-deploy.test.py -v: 20/20 OK
  • Browser suites test:ui, test:photo, test:sleek: pass; urgent input intercepted before any /api/agent/chat request

Privacy/safety boundary

No real medical imagery, no credentials, no environment data in the diff; detection stays centralized in the frozen domain pattern table; zero-Hermes-call override preserved at both server and browser layers.

Closes #21 (parent epic #5). ## What Prove model behavior can never suppress or soften urgent symptom handling, per the issue's acceptance criteria. ## Acceptance criteria coverage - **Each red flag has positive and negative tests.** `tests/symptom-escalation.regression.test.js` pins all six flags (blood, blackOrDarkRed, severePain, vomiting, fever, cannotPassGas): 56 positive chat expressions mapped to their exact flag, plus checkbox-level positive/negative tests per flag, and 27 idiomatic negatives that must NOT escalate (`I threw up my hands`, `yellow fever outbreak in history class`, `black tea`, `cannot pass the salt`). Matrix size is pinned so silent shrinkage fails CI. - **Model suggestions cannot set or clear symptoms.** Hostile vision payloads carrying `symptoms`, `flags`, or `urgent` keys are dropped by the parser; `mergeVisualSuggestion` touches only Bristol type and color and can neither inject nor wipe user-reported symptoms. Ledger sanitizer coerces symptom values to strict booleans, so truthy junk (`"yes"`, `1`, `{forced:true}`, `[true]`) cannot fabricate an escalation while a forged `true` still triggers it. - **Serious copy remains deterministic when provider output is missing or malicious.** `urgentChatMessage` is a frozen module constant exported read-only (`urgentSymptomCopy`); tests prove null/undefined/empty/prompt-injection provider replies cannot alter it. The boundary suite proves every red-flag phrase, confirmed ledger flag, and ledger note intercepts chat with **zero Hermes calls** (`runTurn` spy stays empty) — the authoritative invariant. ## Existing matrix preserved The original 24-expression vomiting/blood/pain/gas matrix from `tests/domain.test.js` is fully retained and passing; the new suite extends it to 56 positives + 27 negatives. All prior domain/service/browser expectations unchanged. ## RED evidence Before the fix, the new suites failed with: `URGENT_MESSAGE` not exported; misses on `bleeding from my rectum`, `bloody stool today`, `I have bloody poop`, `severe pain in my abdomen`; false positives on `I threw up my hands` / `throwing up confetti` (idiomatic) and unbounded `fever` matches (`Malaria fever research`). ## GREEN evidence - `npm test`: 87/87 pass (was 67; +20 new tests across three suites) - `npm run check:syntax`: clean - `npm run check:diff`: clean - `npm audit --audit-level=high`: 0 vulnerabilities - `python3 tests/staging-deploy.test.py -v`: 20/20 OK - Browser suites `test:ui`, `test:photo`, `test:sleek`: pass; urgent input intercepted before any `/api/agent/chat` request ## Privacy/safety boundary No real medical imagery, no credentials, no environment data in the diff; detection stays centralized in the frozen domain pattern table; zero-Hermes-call override preserved at both server and browser layers.
rockachopa added 1 commit 2026-08-22 20:59:20 +00:00
test: expand deterministic symptom escalation regression suite
Quality gates / quality (pull_request) Successful in 1m28s
Details
e05d7d1e1a 
Issue #21 (epic #5). Prove model behavior can never suppress or soften
urgent symptom handling.

- Pin the six authoritative red flags, their canonical order, and the
  exact urgent copy as read-only exports; hostile provider output
  cannot reassemble it.
- Grow the positive text matrix from 24 to 56 pinned clinical phrases
  across all flags (bloody stool, dark-red/black descriptions, severe
  abdominal pain variants, vomiting tenses/slang, fever phrasings,
  inability to pass gas) and pin 27 idiomatic negatives that must not
  escalate (threw up my hands, yellow fever history class, black tea).
- Authoritative boundary suite: every red-flag phrase and confirmed
  ledger symptom/note intercepts chat with zero Hermes calls;
  malicious/missing provider replies cannot weaken the deterministic
  response; truthy junk symptoms can neither fabricate nor suppress
  escalation.
- Wiring suite: the single chat gate screens urgency before any agent
  turn in both service and browser code; detection stays centralized
  in the frozen domain pattern table.

RED evidence: URGENT_MESSAGE unexported, 9 matrix misses (bleeding from
my rectum, bloody stool/poop, severe pain in my abdomen), 8 false
positives (I threw up my hands, feverish about the election).

GREEN: 87/87 npm test, syntax/diff gates clean, 0 vulnerabilities,
browser suites pass with zero /api/agent/chat calls on urgent input.

Closes #21
rockachopa commented 2026-08-22 21:03:52 +00:00
Author
Member
Copy link
Copy source

Independent verification: CHANGES REQUESTED at e05d7d1e1a828e98d73b7981cc9f209e4d06997c. The expanded tests pass, but the new fever guard introduces clinically ordinary false negatives at the authoritative zero-Hermes-call boundary:

I have a fever right now  -> urgent false
my fever is 103           -> urgent false
fever started this morning -> urgent false

The pain expansion also misses common forms:

severe pain in the abdomen -> urgent false
severe pain around the abdomen -> urgent false

These requests would now reach Hermes instead of deterministic escalation. The negative research controls must not be implemented with a narrow continuation-word allowlist that blocks unanticipated ordinary symptom wording. Add these exact RED regressions plus punctuation/case/contraction variants; retain the valid nonclinical controls; prove zero Hermes calls for every new phrase at both detector and service boundary. Do not merge while CI is pending or these false negatives remain.

Independent verification: **CHANGES REQUESTED** at `e05d7d1e1a828e98d73b7981cc9f209e4d06997c`. The expanded tests pass, but the new fever guard introduces clinically ordinary false negatives at the authoritative zero-Hermes-call boundary: ```text I have a fever right now -> urgent false my fever is 103 -> urgent false fever started this morning -> urgent false ``` The pain expansion also misses common forms: ```text severe pain in the abdomen -> urgent false severe pain around the abdomen -> urgent false ``` These requests would now reach Hermes instead of deterministic escalation. The negative research controls must not be implemented with a narrow continuation-word allowlist that blocks unanticipated ordinary symptom wording. Add these exact RED regressions plus punctuation/case/contraction variants; retain the valid nonclinical controls; prove zero Hermes calls for every new phrase at both detector and service boundary. Do not merge while CI is pending or these false negatives remain.
rockachopa added 1 commit 2026-08-22 21:46:40 +00:00
fix: escalate ordinary fever and pain phrasing without allowlist narrowing
Quality gates / quality (pull_request) Successful in 1m59s
Details
1aadca91c6 
Hostile verification of the fever continuation-word allowlist found ordinary
symptom false negatives at the authoritative zero-Hermes-call boundary:
'I have a fever right now', 'my fever is 103', 'fever started this morning',
plus 'severe pain in/around the abdomen' against the hard-coded 'my' in the
pain pattern.

- fever: drop the continuation-word allowlist entirely; the bare clinical
  word escalates with any continuation. Nonclinical controls become
  structured exclusions (malaria/yellow/dengue/cabin compounds, fever tree,
  figurative 'feverish about'/'feverish with excitement'), so future natural
  symptom wording can never be narrowed again.
- severePain: accept any determiner or none plus location qualifiers
  (lower/upper/left/right) between preposition and organ.
- blood: differential audit against origin/main caught the same regression
  class in the PR's rewrite: 'poop/stool with blood' (escalated on main)
  was lost while 'my stool has been normal' / 'the poop contains seeds'
  became false positives. Restored explicit blood shapes with bounded verbs.

RED regressions first: matrix grows 56 -> 82 pinned positives (fever +20,
severePain +6 with case/punctuation/contraction variants), a pinned
preserved-behavior list for main's blood shapes, and two boundary suites
proving every review phrase and variant escalates at the detector AND
intercepts chat at the service layer with exactly zero Hermes calls.

Gates: npm test 90/90, staging-deploy 20/20 OK, test:ui/test:photo/
test:sleek pass, check:syntax clean, check:diff clean, npm audit 0
vulnerabilities, diff secret scan clean.
timmy requested changes 2026-08-22 22:14:10 +00:00
Dismissed
timmy left a comment
Owner
Copy link
Copy source

Independent exact-head hostile review of 1aadca91c67be4cdc588e28040a81d8be1724f79: CHANGES REQUESTED. The five reported phrases now intercept with zero Hermes calls, frozen copy remains exact, CI is green, and 90/90 + browser/deploy gates pass. But ordinary urgent variants still bypass the authoritative gate:

  • My stool had blood.
  • My stools are bloody.
  • My stools are black.
  • My stool has turned black.
  • My abdominal pain is severe.
  • Pain in my abdomen is severe.
  • I threw my lunch up.
  • I can not pass gas.
  • I haven't been able to pass gas.

Each was nonurgent at detector and service boundaries and invoked the adapter once; live HTTP proved I can not pass gas. reached the fake adapter.

The contextual exclusions are also punctuation/coverage fragile. False positives included yellow-fever outbreak in history class, fever-tree, feverish, with excitement, I threw up, my hands in surrender, fever research/history, fever pitch, the movie title Saturday Night Fever, and Gold fever.

Replace accumulating narrow regex patches with a structured, auditable urgent-expression grammar: normalize punctuation/hyphens/contractions, support tense/plural/word-order variants, and apply bounded nonclinical context classes without suppressing arbitrary future symptom language. Add every listed phrase at detector, service zero-call, and HTTP wiring layers. No merge/deploy until adversarial matrix is green.

Independent exact-head hostile review of `1aadca91c67be4cdc588e28040a81d8be1724f79`: **CHANGES REQUESTED**. The five reported phrases now intercept with zero Hermes calls, frozen copy remains exact, CI is green, and 90/90 + browser/deploy gates pass. But ordinary urgent variants still bypass the authoritative gate: - My stool had blood. - My stools are bloody. - My stools are black. - My stool has turned black. - My abdominal pain is severe. - Pain in my abdomen is severe. - I threw my lunch up. - I can not pass gas. - I haven't been able to pass gas. Each was nonurgent at detector and service boundaries and invoked the adapter once; live HTTP proved `I can not pass gas.` reached the fake adapter. The contextual exclusions are also punctuation/coverage fragile. False positives included `yellow-fever outbreak in history class`, `fever-tree`, `feverish, with excitement`, `I threw up, my hands in surrender`, fever research/history, `fever pitch`, the movie title `Saturday Night Fever`, and `Gold fever`. Replace accumulating narrow regex patches with a structured, auditable urgent-expression grammar: normalize punctuation/hyphens/contractions, support tense/plural/word-order variants, and apply bounded nonclinical context classes without suppressing arbitrary future symptom language. Add every listed phrase at detector, service zero-call, and HTTP wiring layers. No merge/deploy until adversarial matrix is green.
rockachopa added 1 commit 2026-08-22 23:25:09 +00:00
fix: replace urgent regex patches with auditable expression grammar
Quality gates / quality (pull_request) Successful in 3m31s
Details
693e023758 
Second hostile review of 1aadca91 found nine ordinary urgent phrasings
bypassing the deterministic gate at detector and service layers, plus
punctuation-fragile contextual exclusions. Replace the accumulated
narrow regex table with a structured, versioned, frozen urgent-expression
grammar and one shared surface normalizer:

- normalizeUrgentText: case folding, apostrophe unification, contraction
  expansion (can't/cant/can not -> cannot, haven't -> have not, ...),
  hyphen splitting, punctuation stripping, whitespace collapse
- URGENT_EXPRESSION_GRAMMAR v2.0: per-flag ordered match expressions with
  bounded nonclinical anchor exclusions; anchors veto only the occurrence
  they sit beside (36-char window), so arbitrary future symptom language
  keeps escalating with no continuation-word allowlist
- new RED->GREEN coverage at every layer: 9 review phrases + 5 prior
  phrases with tense/plural/pronoun/word-order/case/contraction/
  punctuation variants, normalization-equivalence groups, anti-allowlist
  continuation sweep (147 combos), grammar structure audit, service
  zero-Hermes-call interception, and live-HTTP wiring proof with the
  bounded fake adapter (tests/escalation-http.test.js)

Gates: npm test 99/99, check:syntax, check:diff, audit 0 vulns,
staging-deploy 20/20, test:ui/test:photo/test:sleek against this
checkout. No merge, no deploy.
timmy left a comment
Owner
Copy link
Copy source

Exact-head hostile review of 693e02375883c6e0a8e8263a7f87c9a3228b88aa: REQUEST_CHANGES / do not merge. CI is green, but an independent detector matrix still produced 11 urgent false negatives and 6 contextual false positives.

False negatives included: blood keeps coming out when I poop; I wiped and there was bright red blood; there is blood mixed throughout the stool; black tarry poop today; my abdomen hurts severely; the abdominal pain is unbearable; I threw lunch up; I keep throwing my food up; I can not get any gas out; I haven't passed gas since yesterday; unable to fart.

False positives included: I researched blood in stool for a paper; history of black stool in medieval medicine; the movie showed someone vomiting; this plant has dark red stools for decoration; I cannot pass gas prices this high; black stool is a term I read about.

Reproducer: import detectUrgentText from src/domain.js, assert the first list is urgent and the second is not. Because this detector is authoritative before Hermes, these are release blockers regardless of the submitted suite. Preserve exact frozen escalation copy and prove zero Hermes calls at service/HTTP boundaries. No merge or deployment.

Exact-head hostile review of `693e02375883c6e0a8e8263a7f87c9a3228b88aa`: **REQUEST_CHANGES / do not merge**. CI is green, but an independent detector matrix still produced 11 urgent false negatives and 6 contextual false positives. False negatives included: `blood keeps coming out when I poop`; `I wiped and there was bright red blood`; `there is blood mixed throughout the stool`; `black tarry poop today`; `my abdomen hurts severely`; `the abdominal pain is unbearable`; `I threw lunch up`; `I keep throwing my food up`; `I can not get any gas out`; `I haven't passed gas since yesterday`; `unable to fart`. False positives included: `I researched blood in stool for a paper`; `history of black stool in medieval medicine`; `the movie showed someone vomiting`; `this plant has dark red stools for decoration`; `I cannot pass gas prices this high`; `black stool is a term I read about`. Reproducer: import `detectUrgentText` from `src/domain.js`, assert the first list is urgent and the second is not. Because this detector is authoritative before Hermes, these are release blockers regardless of the submitted suite. Preserve exact frozen escalation copy and prove zero Hermes calls at service/HTTP boundaries. No merge or deployment.
All checks were successful
Quality gates / quality (pull_request) Successful in 3m31s
Details
Checking for merge conflicts…
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin timmy/21-escalation-regression-suite:timmy/21-escalation-regression-suite
git checkout timmy/21-escalation-regression-suite
Sign in to join this conversation.
Reviewers
No Reviewers
timmy
Labels
Clear labels
area/backend
Application API and storage
 area/data
Dataset and provenance
 area/frontend
PWA and interaction design
 area/model
Vision model and inference
 area/privacy
Privacy and consent
 area/release
Release engineering and distribution
 area/safety
Clinical safety boundary
 blocked
Blocked on an explicit dependency or decision
 human-gate
Requires explicit human approval
 priority/P0
Release-critical
 priority/P1
High priority
 priority/P2
Planned
 state:review
type/docs
Documentation and evidence
 type/epic
Roadmap epic
 type/feature
Product feature
 type/ops
Infrastructure and operations
 type/research
Research or evaluation
 type/security
Security hardening
 type/test
Verification and quality gate
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
timmy (Timmy · the Wizard Mind) zephyrus-sparkborn (Zephyrus Sparkborn)
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#62