timmy
bc64055196
Quality gates / quality (pull_request) Canceled after 0s
Brings #82 up to today's `main` (1bbcc10) and resolves the two conflicts as ordered
unions: `package.json` `test`/`check:syntax` and the release builder's syntax allow-list
and evidence keys. Both sides only appended, so nothing was dropped.
Verifying the merge against main's RC attack suite (#40) found two real defects, each
fixed test-first:
- An oversized body destroyed the socket mid-upload, so the caller saw `fetch failed`
instead of the 413 it had already earned, and `security_attack_suite.mjs` could not
complete at all. New test in `tests/analyze-ingress.acceptance.test.js` failed with
`fetch failed` before the fix; `src/http-body.js` now keeps draining without buffering
and refuses on `end`, so the ceiling still holds and the refusal is readable.
- #16's per-caller budget turned the suite's own load phase into 8/8 429s, which would
have published refusals as analysis latency. The load phase now runs against a second
permissive target and the suite refuses to publish a load figure if any sample was
refused. `buildReport` also publishes `evidenceVerified`, so the verdict is
re-derivable from the report — asserted by #40's acceptance test, which only trips once
the verdict is GO.
Evidence on the merged tree: `npm test` 277/277, `test:security-attack` 8/8, the RC attack
suite returns VERDICT: GO with both P0 ingress cases passing (main is BLOCKED on exactly
those two), 14 browser acceptance suites green, and the release demo records end to end
(8:13, 97 captions) including the on-camera refusal scene.
81 lines
4.5 KiB
Markdown
81 lines
4.5 KiB
Markdown
# RC Security & Abuse Evidence (#40)
|
|
|
|
The release candidate is attacked before users get to it. This page is the procedure and the rule,
|
|
not a stored result: a pasted-in "all clear" is exactly the kind of evidence that stops being true
|
|
after the next commit.
|
|
|
|
## Run it
|
|
|
|
```bash
|
|
node scripts/security_attack_suite.mjs --json artifacts/security/rc-attack-report.json
|
|
```
|
|
|
|
Exit codes are part of the contract:
|
|
|
|
| Exit | Meaning |
|
|
|---:|---|
|
|
| `0` | `VERDICT: GO` — every case matched its expectation. |
|
|
| `1` | `VERDICT: BLOCKED` — at least one P0 case failed, or a required category has no executable evidence. |
|
|
| `2` | The suite could not complete (target never became ready, vision disabled, no log captured). |
|
|
|
|
The suite is a child process of nothing: it starts its own copy of `server.mjs` on a free loopback
|
|
port, points it at a **counting stub worker**, and drives it with synthetic hostile bytes. It never
|
|
contacts a real model provider, never uses a real medical image, and never touches a live user's data.
|
|
|
|
## What it does and does not prove
|
|
|
|
- It proves what this HTTP boundary did with the bytes it was handed, and whether the worker behind
|
|
it was called. The worker is a stub, so "the provider saw it" is a counted fact, not an inference.
|
|
- It does **not** measure model quality, clinical accuracy, or the safety of the medical/non-diagnostic
|
|
boundary. Those live in the vision, escalation, and clinical-review issues.
|
|
- Latency numbers describe the loopback boundary against a stub. They are not a model benchmark.
|
|
|
|
## Cases
|
|
|
|
| Case | Class | Severity | Expectation |
|
|
|---|---|---:|---|
|
|
| `ingress-oversized-body` | oversized | P0 | `413`, worker never called |
|
|
| `ingress-polyglot-html` | polyglot | P0 | `400`, worker never called |
|
|
| `ingress-garbage-bytes` | truncated | P0 | `400`, worker never called |
|
|
| `consent-required` | consent | P0 | `400`, worker never called |
|
|
| `model-output-malformed` | malformed | P0 | `503` neutral copy, provider text never echoed |
|
|
| `replay-duplicate-request` | replay | P2 | identical answer, nothing stored server-side |
|
|
| `flood-concurrency` | flood | P1 | burst meets a bound (`429`/overload), not an open pipe |
|
|
| `log-leak-scan` | log | P0 | captured server log carries no image bytes |
|
|
| `deletion-withdrawal` | deletion | P0 | out of surface — counted only when `tests/training-contribution.test.js` exists |
|
|
|
|
## The rule
|
|
|
|
`src/release-security-gate.js` owns the verdict, so it is testable rather than asserted:
|
|
|
|
- A case passes only when **every** expected field matches the observation. A partial match fails and
|
|
names the field that broke (`status 400 expected, observed 200`).
|
|
- **Any** failing P0 case blocks RC approval. P1/P2 failures are recorded as holds.
|
|
- A required class with no executable evidence is `unproven` and blocks the release. Coverage may be
|
|
borrowed from another suite (`out-of-surface` + an evidence path) only after the runner verified
|
|
that file exists — otherwise it blocks.
|
|
- The report is redacted before it is written, and assembly fails closed if any image bytes survive
|
|
redaction. Evidence that carries a photo is worse than no evidence.
|
|
|
|
## Current state (read the run, not this line)
|
|
|
|
This section used to predict **BLOCKED** until the ingress hardening in
|
|
[#16](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/16) landed. With
|
|
that ingress in place the same command returns **GO** on this tree, because the format is decided from
|
|
the bytes rather than the data-URL label: `ingress-polyglot-html` and `ingress-garbage-bytes` are now
|
|
refused with `400` before the worker is called, and an oversized body is refused with a `413` the
|
|
caller can read instead of a dropped connection. Latency for the load phase is measured against a
|
|
second target with the request budget opened up, so those samples are analyses rather than `429`
|
|
refusals. Queue bounding is still [#17](https://forge.alexanderwhitestone.com/git/stackchain/timmy-talking-turd/issues/17)
|
|
and is held here as P1.
|
|
|
|
Deletion is governed by the contribution ledger, not by this HTTP boundary; the case stays
|
|
`covered-elsewhere` against `tests/training-contribution.test.js` and blocks if that file disappears.
|
|
|
|
## Evidence hygiene
|
|
|
|
- The report is written to `artifacts/security/` and must never be committed with a real photo in it.
|
|
- The captured server log is scanned by the same detector (`containsImageBytes`) before it is quoted.
|
|
- Error bodies are recorded as a 120-character snippet; the suite refuses to write a report that
|
|
still contains a data URI or a long base64 run.
|