timmy
bc64055196
Brings #82 up to today's `main` (1bbcc10) and resolves the two conflicts as ordered
unions: `package.json` `test`/`check:syntax` and the release builder's syntax allow-list
and evidence keys. Both sides only appended, so nothing was dropped.
Verifying the merge against main's RC attack suite (#40) found two real defects, each
fixed test-first:
- An oversized body destroyed the socket mid-upload, so the caller saw `fetch failed`
instead of the 413 it had already earned, and `security_attack_suite.mjs` could not
complete at all. New test in `tests/analyze-ingress.acceptance.test.js` failed with
`fetch failed` before the fix; `src/http-body.js` now keeps draining without buffering
and refuses on `end`, so the ceiling still holds and the refusal is readable.
- #16's per-caller budget turned the suite's own load phase into 8/8 429s, which would
have published refusals as analysis latency. The load phase now runs against a second
permissive target and the suite refuses to publish a load figure if any sample was
refused. `buildReport` also publishes `evidenceVerified`, so the verdict is
re-derivable from the report — asserted by #40's acceptance test, which only trips once
the verdict is GO.
Evidence on the merged tree: `npm test` 277/277, `test:security-attack` 8/8, the RC attack
suite returns VERDICT: GO with both P0 ingress cases passing (main is BLOCKED on exactly
those two), 14 browser acceptance suites green, and the release demo records end to end
(8:13, 97 captions) including the on-camera refusal scene.
4.5 KiB
RC Security & Abuse Evidence (#40)
The release candidate is attacked before users get to it. This page is the procedure and the rule, not a stored result: a pasted-in "all clear" is exactly the kind of evidence that stops being true after the next commit.
Run it
node scripts/security_attack_suite.mjs --json artifacts/security/rc-attack-report.json
Exit codes are part of the contract:
| Exit | Meaning |
|---|---|
0 |
VERDICT: GO — every case matched its expectation. |
1 |
VERDICT: BLOCKED — at least one P0 case failed, or a required category has no executable evidence. |
2 |
The suite could not complete (target never became ready, vision disabled, no log captured). |
The suite is a child process of nothing: it starts its own copy of server.mjs on a free loopback
port, points it at a counting stub worker, and drives it with synthetic hostile bytes. It never
contacts a real model provider, never uses a real medical image, and never touches a live user's data.
What it does and does not prove
- It proves what this HTTP boundary did with the bytes it was handed, and whether the worker behind it was called. The worker is a stub, so "the provider saw it" is a counted fact, not an inference.
- It does not measure model quality, clinical accuracy, or the safety of the medical/non-diagnostic boundary. Those live in the vision, escalation, and clinical-review issues.
- Latency numbers describe the loopback boundary against a stub. They are not a model benchmark.
Cases
| Case | Class | Severity | Expectation |
|---|---|---|---|
ingress-oversized-body |
oversized | P0 | 413, worker never called |
ingress-polyglot-html |
polyglot | P0 | 400, worker never called |
ingress-garbage-bytes |
truncated | P0 | 400, worker never called |
consent-required |
consent | P0 | 400, worker never called |
model-output-malformed |
malformed | P0 | 503 neutral copy, provider text never echoed |
replay-duplicate-request |
replay | P2 | identical answer, nothing stored server-side |
flood-concurrency |
flood | P1 | burst meets a bound (429/overload), not an open pipe |
log-leak-scan |
log | P0 | captured server log carries no image bytes |
deletion-withdrawal |
deletion | P0 | out of surface — counted only when tests/training-contribution.test.js exists |
The rule
src/release-security-gate.js owns the verdict, so it is testable rather than asserted:
- A case passes only when every expected field matches the observation. A partial match fails and
names the field that broke (
status 400 expected, observed 200). - Any failing P0 case blocks RC approval. P1/P2 failures are recorded as holds.
- A required class with no executable evidence is
unprovenand blocks the release. Coverage may be borrowed from another suite (out-of-surface+ an evidence path) only after the runner verified that file exists — otherwise it blocks. - The report is redacted before it is written, and assembly fails closed if any image bytes survive redaction. Evidence that carries a photo is worse than no evidence.
Current state (read the run, not this line)
This section used to predict BLOCKED until the ingress hardening in
#16 landed. With
that ingress in place the same command returns GO on this tree, because the format is decided from
the bytes rather than the data-URL label: ingress-polyglot-html and ingress-garbage-bytes are now
refused with 400 before the worker is called, and an oversized body is refused with a 413 the
caller can read instead of a dropped connection. Latency for the load phase is measured against a
second target with the request budget opened up, so those samples are analyses rather than 429
refusals. Queue bounding is still #17
and is held here as P1.
Deletion is governed by the contribution ledger, not by this HTTP boundary; the case stays
covered-elsewhere against tests/training-contribution.test.js and blocks if that file disappears.
Evidence hygiene
- The report is written to
artifacts/security/and must never be committed with a real photo in it. - The captured server log is scanned by the same detector (
containsImageBytes) before it is quoted. - Error bodies are recorded as a 120-character snippet; the suite refuses to write a report that still contains a data URI or a long base64 run.