fix: escalate the symptom phrasing people actually use, and never clear a reported symptom (#21) #77

2026-10-15T08:15:40Z

rockachopa commented

2026-10-15 08:15:40 +00:00

Closes #21

Outcome

Deterministic escalation now fires for the phrasing people actually use, and no reported
symptom can be silently cleared on the way to the check.

What the expanded suite found

The detector only matched a narrow set of literal phrases. Probing real clinical wording
against main showed 18 of 18 checked phrasings missing, including:

There is blood when I wipe, bleeding from my back passage, blood mixed in my poo
I have tarry stools, my stool looks like tar, my poop is very dark red
my temperature is 103, I have a high temperature
I have severe pain in my belly, the pain in my stomach is severe
I am not passing any gas, I have not passed gas all day

Those turns reached the Hermes provider instead of the fixed medical guidance.

Acceptance criteria

Each red flag has positive and negative tests — per-flag positive tables plus the
near-miss wording that must stay quiet: a paper cut, blood pressure, ink on paper, black
beans, black pants, dark red wine, "the road looks like tar", a 98.6/37 temperature,
feverfew, severe knee or back pain, mild stomach pain, "I passed gas this morning",
"I have not passed my driving test", "the car has no gas".
Model suggestions cannot set or clear symptoms — mergeVisualSuggestion still moves
only form and colour; a schema-valid reply carrying symptoms, urgent, diagnosis, or a
replacement warning cannot reach the user. confidence, observations, and the warning
remain Timmy's own contract.
Serious copy is deterministic when provider output is missing or malicious — every red
flag produces byte-identical copy, each flag phrase is intercepted at the gateway with
safetyOverride, and a provider that is missing, throwing, or insisting "no need to see a
doctor" never changes the reply the user sees.

Also fixed

detectUrgentFlags read only the literal boolean true, so a symptom reported as 1,
"yes", or "true" was read as not reported — a silently cleared symptom, the one
direction this boundary may never fail in. The gateway ledger sanitizer had the same strict
check, which could clear a browser-supplied ledger symptom before the escalation check.
Both now share isReportedSymptom; only an explicit clear (false, 0, "no", blank, absent)
stays quiet.

Evidence

tests/escalation-regression.test.js (16 tests), registered in the default unit gate and
pinned in tests/ci-workflow.test.js.
TDD: the language-coverage tests, the temperature/colour tests, the pain/gas tests, and the
truthy-symptom tests were each observed failing before implementation. The five boundary and
determinism tests (tests 12-16) passed on first write and are reported as hold-out pins,
not as RED cycles.
npm test 173/173 in 9 consecutive full runs; npm run check:syntax, npm run check:diff,
python3 tests/staging-deploy.test.py -v (20 tests), and all ten browser acceptance scripts
(test:ui, test:photo, test:mobile-capture, test:sleek, test:manual-fallback,
test:provisional, test:followup, test:vision-refusal, test:worker-readiness,
test:pattern-summary) pass.
One transient full-suite failure occurred in a single early run while another command ran in
the same turn; the test name was not captured and it did not reproduce in 9 subsequent runs.

Demonstration path (updated, not asserted on paper)

scripts/record_release_demo.mjs now types There is blood when I wipe and I have tarry stools in the real chat, then seeds a fictional ledger whose note is stool was tarry yesterday and asks an innocuous question. Each scene asserts in the browser that exactly one
new Timmy bubble carries Pause and get medical help..., that the copy is the app's own fixed
string, and that hermesCalls never moves. Three new captions are required release gates in
scripts/build_release.py, and tests/release-demo.test.js pins the new scene.

Recorded end to end on this branch: webm produced (8.3 MB) and the caption receipt contains
all 17 required captions, so tomorrow's release job can record this path as-is.

Boundary

Unchanged: no new inference, no diagnosis, no claim from the image, no production service
touched. Escalation only ever gets broader on what the person reports.

Follow-up noted, not done here

A provider can still put medical reassurance into the free-text observations string that the
suggestion card renders (escaped, and beside Timmy's fixed warning). That is a vision-response
hardening item for the corpus/boundary work (#16/#18), not escalation detection, so it is
deliberately out of this change.

Also pre-existing and untouched: fever matches "fever blister". It escalates too eagerly on
main already; over-triggering is the safe direction, and narrowing it is a copy question, not
this issue.

Closes #21 ## Outcome Deterministic escalation now fires for the phrasing people actually use, and no reported symptom can be silently cleared on the way to the check. ## What the expanded suite found The detector only matched a narrow set of literal phrases. Probing real clinical wording against `main` showed **18 of 18 checked phrasings missing**, including: - `There is blood when I wipe`, `bleeding from my back passage`, `blood mixed in my poo` - `I have tarry stools`, `my stool looks like tar`, `my poop is very dark red` - `my temperature is 103`, `I have a high temperature` - `I have severe pain in my belly`, `the pain in my stomach is severe` - `I am not passing any gas`, `I have not passed gas all day` Those turns reached the Hermes provider instead of the fixed medical guidance. ## Acceptance criteria - **Each red flag has positive and negative tests** — per-flag positive tables plus the near-miss wording that must stay quiet: a paper cut, blood pressure, ink on paper, black beans, black pants, dark red wine, "the road looks like tar", a 98.6/37 temperature, feverfew, severe knee or back pain, mild stomach pain, "I passed gas this morning", "I have not passed my driving test", "the car has no gas". - **Model suggestions cannot set or clear symptoms** — `mergeVisualSuggestion` still moves only form and colour; a schema-valid reply carrying `symptoms`, `urgent`, `diagnosis`, or a replacement `warning` cannot reach the user. `confidence`, `observations`, and the warning remain Timmy's own contract. - **Serious copy is deterministic when provider output is missing or malicious** — every red flag produces byte-identical copy, each flag phrase is intercepted at the gateway with `safetyOverride`, and a provider that is missing, throwing, or insisting "no need to see a doctor" never changes the reply the user sees. ## Also fixed `detectUrgentFlags` read only the literal boolean `true`, so a symptom reported as `1`, `"yes"`, or `"true"` was read as *not reported* — a silently cleared symptom, the one direction this boundary may never fail in. The gateway ledger sanitizer had the same strict check, which could clear a browser-supplied ledger symptom before the escalation check. Both now share `isReportedSymptom`; only an explicit clear (false, 0, "no", blank, absent) stays quiet. ## Evidence - `tests/escalation-regression.test.js` (16 tests), registered in the default unit gate and pinned in `tests/ci-workflow.test.js`. - TDD: the language-coverage tests, the temperature/colour tests, the pain/gas tests, and the truthy-symptom tests were each observed failing before implementation. The five boundary and determinism tests (`tests 12-16`) passed on first write and are reported as hold-out pins, not as RED cycles. - `npm test` 173/173 in 9 consecutive full runs; `npm run check:syntax`, `npm run check:diff`, `python3 tests/staging-deploy.test.py -v` (20 tests), and all ten browser acceptance scripts (`test:ui`, `test:photo`, `test:mobile-capture`, `test:sleek`, `test:manual-fallback`, `test:provisional`, `test:followup`, `test:vision-refusal`, `test:worker-readiness`, `test:pattern-summary`) pass. - One transient full-suite failure occurred in a single early run while another command ran in the same turn; the test name was not captured and it did not reproduce in 9 subsequent runs. ## Demonstration path (updated, not asserted on paper) `scripts/record_release_demo.mjs` now types `There is blood when I wipe` and `I have tarry stools` in the real chat, then seeds a fictional ledger whose note is `stool was tarry yesterday` and asks an innocuous question. Each scene asserts in the browser that exactly one new Timmy bubble carries `Pause and get medical help...`, that the copy is the app's own fixed string, and that `hermesCalls` never moves. Three new captions are required release gates in `scripts/build_release.py`, and `tests/release-demo.test.js` pins the new scene. - Recorded end to end on this branch: webm produced (8.3 MB) and the caption receipt contains all 17 required captions, so tomorrow's release job can record this path as-is. ## Boundary Unchanged: no new inference, no diagnosis, no claim from the image, no production service touched. Escalation only ever gets broader on what the person reports. ## Follow-up noted, not done here A provider can still put medical reassurance into the free-text `observations` string that the suggestion card renders (escaped, and beside Timmy's fixed warning). That is a vision-response hardening item for the corpus/boundary work (#16/#18), not escalation detection, so it is deliberately out of this change. Also pre-existing and untouched: `fever` matches "fever blister". It escalates too eagerly on main already; over-triggering is the safe direction, and narrowing it is a copy question, not this issue.

rockachopa added 1 commit 2026-10-15 08:15:41 +00:00

fix: escalate the symptom phrasing people actually use, and never clear a reported symptom (#21 )

Quality gates / quality (pull_request) Successful in 2m38s

Details

b4be7049ea

Deterministic escalation only fired on a narrow set of literal phrases, so real reports
like "there is blood when I wipe" or "I have tarry stools" reached the chat provider
without the fixed medical guidance. Expanding the suite found 18 of 18 checked clinical
phrasings missing.

- domain: cover bleeding reported by wiping/toilet paper/anatomy/mixed-in phrasing,
black-tarry-dark-red stool, reported temperature and high temperature, abdominal-scoped
severe pain in every natural word order, and past-tense/slang inability to pass gas.
Near-miss wording that must stay quiet (paper cut, blood pressure, black beans, dark red
wine, a normal temperature, knee or back pain, passed gas, no gas in the car) is pinned.
- domain: a symptom reported in any truthy shape (1, "yes", "true") now counts as reported.
Reading only the literal boolean `true` silently cleared a symptom, which is the one
direction this boundary may never fail in.
- hermes-agent-service: the gateway ledger sanitizer used the same strict check, so a
browser-supplied ledger entry could have its symptom cleared before the escalation check.
- tests/escalation-regression.test.js: per-flag positive and negative language coverage,
model output can neither set nor clear a symptom/urgency/discomfort/note, and the
escalation copy is byte-identical for every flag and under a missing, throwing, or
malicious provider. Registered in the default unit gate.
- scripts/record_release_demo.mjs: the recording now types the new bleeding and tarry
phrasing, and seeds a fictional ledger whose note says "stool was tarry yesterday", then
asserts in the browser that exactly one new Timmy bubble carries the app's own copy and
that the turn never reaches Hermes. Three new captions are required release gates.

Boundary unchanged: no new inference, no diagnosis, no image claim. Escalation only ever
gets broader on user-reported symptoms.

rockachopa closed this pull request

2026-10-15 12:03:15 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#77