feat: pin the worker container and gate readiness on the loaded model (#15) #75

2026-10-13T08:20:45Z

rockachopa commented

2026-10-13 08:20:45 +00:00

Refs #15 — deliberately not Closes: the target-GPU half of criterion 4 (cold/warm latency and VRAM recorded on the target GPU) is instrumented but not measured. This run had no GPU host and buying compute is out of bounds, so no receipt is bundled. Criterion 3 is enforced by contract tests rather than a running container: the image targets a CUDA host, and this host has no CUDA to build or run it on. Said plainly here so nobody reads the green CI as more than it is.

What changed

A worker that answers /v1/models has not necessarily loaded its weights. Real evidence from the pinned build (llama.cpp 6d054983, SmolVLM2-2.2B-Instruct-Q4_K_M + mmproj):

$ curl /v1/models                     -> HTTP 503   (listener open, weights still loading)
$ bash scripts/worker_ready.sh --port 18080
  Worker reachable on http://127.0.0.1:18080/v1 but SmolVLM2-2.2B-Instruct is not loaded yet.
  exit 3
$ bash scripts/worker_ready.sh --port 18080 --wait 600
  Worker ready: SmolVLM2-2.2B-Instruct is loaded
  exit 0                                (model loaded at 9.25s per the server log)

The app called that first state ready and offered consent + Analyze visible form + color, which could only fail after the upload. Readiness now requires the loaded pinned model, and that state lands on manual Bristol entry in one tap.

User-visible (src/vision-readiness.js, app.js, styles.css): new loading phase — "Vision worker up · the model is not loaded yet · manual logging is available" — with its own amber treatment, no consent card, no analyze control, and the manual path still one action. analyze enablement now follows visionReadiness(...).ready instead of providerReady.

Container recipe (deploy/worker/): digest-pinned CUDA build/runtime bases (nvidia/cuda:12.4.1-*@sha256:…), pinned llama.cpp commit, weights never baked in but mounted read-only at /models and sha256-verified by the entrypoint before the server starts, numeric non-root USER, no credential of any kind, and a run recipe that refuses any publish host outside loopback (--read-only --cap-drop=ALL --security-opt no-new-privileges). deploy/worker/receipts.json is the machine-readable receipt, drift-guarded against scripts/bootstrap_selfhost_smolvlm.sh.

Post-load gate (scripts/worker_ready.sh): one implementation shared by the host bootstrap (bootstrap_selfhost_smolvlm.sh ready), the image HEALTHCHECK, and the benchmark — exit 2 unreachable, 3 reachable-but-not-loaded, 0 ready.

Target-GPU benchmark (scripts/record_worker_benchmark.mjs): records cold start, warm latency percentiles, and VRAM against the target device, and refuses to write anything it did not measure. Exercised against the real running worker with no GPU:

$ node scripts/record_worker_benchmark.mjs --out /tmp/worker-benchmark.json --port 18080
  No GPU could be probed with nvidia-smi: spawn nvidia-smi ENOENT. A benchmark receipt needs a real
  device, so nothing was written.
  exit 1   (no file created)

Acceptance criteria

Image pins llama.cpp commit and model hashes — the Dockerfile, receipts.json, and the host bootstrap must agree or tests/worker-container.test.js fails; the pins were confirmed against the real weight files (sha256sum matches the recorded values).
Health/readiness only pass after model load — gate exits 3 while loading, 0 once loaded (real worker above); the image HEALTHCHECK calls the same gate with --start-period=600s; the app contract fails closed on an unconfirmed model.
No public bind, default credentials, or bundled user images — asserted on the recipe: loopback-only publish (a non-loopback host is refused), no --privileged, read-only mounts, no credential defaults, .dockerignore excludes weights/user images/releases/secrets. Runtime behaviour of the unbuilt CUDA image is not verified here.
Cold/warm latency and VRAM recorded on the target GPU — harness added, refuses to fabricate, not measured.

Demonstration path

scripts/record_release_demo.mjs runs a new asserted scene (issue #15): the worker answers while the model loads — the recording proves no "Self-hosted model ready" claim, no #analyze-photo, no .consent-card, the distinct amber not-loaded status, and one-tap manual entry; then the model loads and the AI path returns. The recorded captions were replayed for real (42 captions rendered, TIMMY_DEMO_CAPTIONS_OUT receipt), and the caption is added to DEMO_REQUIRED_CAPTIONS so the release build fails if the scene disappears.

Tests

Unit: 146 pass (npm test), including the new tests/worker-container.test.js, tests/worker-ready.test.js, tests/worker-benchmark.test.js.
Browser acceptance: all nine suites pass (test:ui, test:photo, test:mobile-capture, test:sleek, test:manual-fallback, test:provisional, test:followup, test:vision-refusal, and the new test:worker-readiness).
npm run check:syntax, npm run check:diff, python3 tests/staging-deploy.test.py pass; secret scan over the diff is clean.

Contract widening worth a reviewer's eye

visionReadiness previously treated providerReady: true as ready. It now requires modelSeen: true, and a missing modelSeen fails closed. server.mjs always sends it, so the self-hosted app path is unaffected (verified against the real worker, which returns both data and models). Three test fixtures that mocked a ready provider without modelSeen were completed to match the real contract (mobile-capture, sleek-chat), and the pre-existing phase assertion in tests/vision-readiness.test.js was updated to the widened contract.

**Refs #15 — deliberately not `Closes`:** the target-GPU half of criterion 4 (cold/warm latency and VRAM recorded on the target GPU) is instrumented but **not measured**. This run had no GPU host and buying compute is out of bounds, so no receipt is bundled. Criterion 3 is enforced by contract tests rather than a running container: the image targets a CUDA host, and this host has no CUDA to build or run it on. Said plainly here so nobody reads the green CI as more than it is. ## What changed A worker that answers `/v1/models` has not necessarily loaded its weights. Real evidence from the pinned build (`llama.cpp 6d054983`, `SmolVLM2-2.2B-Instruct-Q4_K_M` + mmproj): ``` $ curl /v1/models -> HTTP 503 (listener open, weights still loading) $ bash scripts/worker_ready.sh --port 18080 Worker reachable on http://127.0.0.1:18080/v1 but SmolVLM2-2.2B-Instruct is not loaded yet. exit 3 $ bash scripts/worker_ready.sh --port 18080 --wait 600 Worker ready: SmolVLM2-2.2B-Instruct is loaded exit 0 (model loaded at 9.25s per the server log) ``` The app called that first state **ready** and offered consent + *Analyze visible form + color*, which could only fail after the upload. Readiness now requires the loaded pinned model, and that state lands on manual Bristol entry in one tap. **User-visible** (`src/vision-readiness.js`, `app.js`, `styles.css`): new `loading` phase — "Vision worker up · the model is not loaded yet · manual logging is available" — with its own amber treatment, no consent card, no analyze control, and the manual path still one action. `analyze` enablement now follows `visionReadiness(...).ready` instead of `providerReady`. **Container recipe** (`deploy/worker/`): digest-pinned CUDA build/runtime bases (`nvidia/cuda:12.4.1-*@sha256:…`), pinned llama.cpp commit, weights never baked in but mounted read-only at `/models` and sha256-verified by the entrypoint before the server starts, numeric non-root `USER`, no credential of any kind, and a run recipe that refuses any publish host outside loopback (`--read-only --cap-drop=ALL --security-opt no-new-privileges`). `deploy/worker/receipts.json` is the machine-readable receipt, drift-guarded against `scripts/bootstrap_selfhost_smolvlm.sh`. **Post-load gate** (`scripts/worker_ready.sh`): one implementation shared by the host bootstrap (`bootstrap_selfhost_smolvlm.sh ready`), the image `HEALTHCHECK`, and the benchmark — exit 2 unreachable, 3 reachable-but-not-loaded, 0 ready. **Target-GPU benchmark** (`scripts/record_worker_benchmark.mjs`): records cold start, warm latency percentiles, and VRAM against the target device, and refuses to write anything it did not measure. Exercised against the real running worker with no GPU: ``` $ node scripts/record_worker_benchmark.mjs --out /tmp/worker-benchmark.json --port 18080 No GPU could be probed with nvidia-smi: spawn nvidia-smi ENOENT. A benchmark receipt needs a real device, so nothing was written. exit 1 (no file created) ``` ## Acceptance criteria - [x] Image pins llama.cpp commit and model hashes — the Dockerfile, `receipts.json`, and the host bootstrap must agree or `tests/worker-container.test.js` fails; the pins were confirmed against the real weight files (`sha256sum` matches the recorded values). - [x] Health/readiness only pass after model load — gate exits 3 while loading, 0 once loaded (real worker above); the image `HEALTHCHECK` calls the same gate with `--start-period=600s`; the app contract fails closed on an unconfirmed model. - [x] No public bind, default credentials, or bundled user images — asserted on the recipe: loopback-only publish (a non-loopback host is refused), no `--privileged`, read-only mounts, no credential defaults, `.dockerignore` excludes weights/user images/releases/secrets. Runtime behaviour of the unbuilt CUDA image is **not** verified here. - [ ] Cold/warm latency and VRAM recorded on the target GPU — harness added, refuses to fabricate, **not measured**. ## Demonstration path `scripts/record_release_demo.mjs` runs a new asserted scene (issue #15): the worker answers while the model loads — the recording proves no "Self-hosted model ready" claim, no `#analyze-photo`, no `.consent-card`, the distinct amber not-loaded status, and one-tap manual entry; then the model loads and the AI path returns. The recorded captions were replayed for real (42 captions rendered, `TIMMY_DEMO_CAPTIONS_OUT` receipt), and the caption is added to `DEMO_REQUIRED_CAPTIONS` so the release build fails if the scene disappears. ## Tests - Unit: **146 pass** (`npm test`), including the new `tests/worker-container.test.js`, `tests/worker-ready.test.js`, `tests/worker-benchmark.test.js`. - Browser acceptance: all nine suites pass (`test:ui`, `test:photo`, `test:mobile-capture`, `test:sleek`, `test:manual-fallback`, `test:provisional`, `test:followup`, `test:vision-refusal`, and the new `test:worker-readiness`). - `npm run check:syntax`, `npm run check:diff`, `python3 tests/staging-deploy.test.py` pass; secret scan over the diff is clean. ## Contract widening worth a reviewer's eye `visionReadiness` previously treated `providerReady: true` as ready. It now requires `modelSeen: true`, and a **missing** `modelSeen` fails closed. `server.mjs` always sends it, so the self-hosted app path is unaffected (verified against the real worker, which returns both `data` and `models`). Three test fixtures that mocked a ready provider without `modelSeen` were completed to match the real contract (`mobile-capture`, `sleek-chat`), and the pre-existing phase assertion in `tests/vision-readiness.test.js` was updated to the widened contract.

rockachopa added 1 commit 2026-10-13 08:20:46 +00:00

feat: pin the self-hosted worker container and gate readiness on the loaded model (#15 )

Quality gates / quality (pull_request) Successful in 3m29s

Details

46f961668f

A worker that answers /v1/models has not necessarily loaded its weights, so the app
was offering an AI control that could only fail after the upload. Readiness now
passes only once the pinned model is loaded, and the not-loaded state reaches manual
Bristol entry in one tap. Verified against the real pinned llama.cpp build: it serves
503 while loading and the gate reports the distinct not-loaded state.

Adds the reproducible container recipe (digest-pinned bases, pinned llama.cpp commit,
hash-verified read-only weights, unprivileged, loopback-only publish), one shared
post-load readiness gate used by the host bootstrap and the image healthcheck, and a
target-GPU benchmark that refuses to write a receipt it did not measure.

Refs #15

rockachopa referenced this pull request

2026-10-13 08:21:13 +00:00

Containerize the llama.cpp GPU worker with pinned model receipts #15

rockachopa closed this pull request

2026-10-13 12:05:31 +00:00

Pull request closed

Please reopen this pull request to perform a merge.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#75