Operator backup, restore, and production runbook (#39) #90

2026-10-29T11:03:15Z

rockachopa commented

2026-10-29 11:03:15 +00:00

Outcome

scripts/operator_backup.py plus a production runbook that an operator can actually follow, and a recorded demonstration of both.

What it does

Backup archives only records. manifest.jsonl (provenance + withdrawal tombstones) and the derived images it points at. Model weights, transient inference images, secrets, logs, symlinks, and oversize files are refused by name, and every skipped path is listed in the manifest with its reason — an omission is an auditable decision, not silence.
Every stored member carries a SHA-256, and the archive is reproducible (sorted members, zeroed mtime/uid/gid, gzip mtime 0), so two backups of unchanged records are byte-identical.
Restore fails closed. It verifies every digest, rejects absolute/traversal/link/unlisted members, refuses a non-empty target, and only then writes. Afterwards it recounts manifest.jsonl and refuses if the counts differ from the backup, so a lost tombstone cannot restore quietly.
docs/PRODUCTION-RUNBOOK.md documents the real topology, the network boundaries (the app's own HOST default is 0.0.0.0; the deployment sets HOST=127.0.0.1 from the mode-600 env template; the worker refuses a non-loopback publish), the four secrets and their rotation, retention, and both rollback paths.

Tests

tests/backup-restore.test.py — 11 cases: exclusions with reasons, symlink refusal, digest mismatch, unlisted member, traversal, absolute path, link member, non-empty target, record counts, tombstone round trip, byte-identical reproducibility. RED before scripts/operator_backup.py existed, GREEN after.
tests/production-runbook.test.py — 9 cases: required sections, and every documented python3 scripts/... command is parsed and its flags checked against the real CLI help, so the runbook cannot drift from the tools. Also fails if server.mjs starts persisting data while the runbook claims it keeps no journal, and pins the bind-address claims to server.mjs, deploy/timmy-staging.env.example, and deploy/worker/run_worker_container.sh.
Full suite: npm test 230/230, test:ui, test:photo, test:mobile-capture, test:corpus, test:review, test:dataset-card, test:calibration, test:backup, test:runbook, check:syntax, check:diff — all pass.

Demonstration

scripts/record_release_demo.mjs gains an OPERATOR BACKUP card that runs the real backup, verify, and restore against a synthetic corpus built by the real ingest and revoke commands, with real decoy files (weight shard, transient upload, env file, log). Every value on the card is read back from those runs, and the four captions are in DEMO_REQUIRED_CAPTIONS, so the release builder fails if they never paint.

Recorded end-to-end on this branch against a real browser: 91/91 captions pixel-verified, exit 0.

Also: the demo's target origin is now TIMMY_DEMO_URL with the documented http://127.0.0.1:4173 default (asserted in tests/release-demo.test.js), so a local run can be recorded while the live 4173 listener serves another tree instead of killing it.

Scope note

This PR does not close #39. Acceptance criterion 3 is "rollback procedure is exercised on staging": the rollback code path is exercised by tests/staging-deploy.test.py against a rootless fixture, but the live drill on the private staging host is a human gate under #53 and is not run automatically by this loop. The runbook says exactly that.

No medical/non-diagnostic boundary was moved, no infrastructure was bought, no inference or weights are included, and no production service was touched.

## Outcome `scripts/operator_backup.py` plus a production runbook that an operator can actually follow, and a recorded demonstration of both. ## What it does - **Backup archives only records.** `manifest.jsonl` (provenance + withdrawal tombstones) and the derived images it points at. Model weights, transient inference images, secrets, logs, symlinks, and oversize files are refused **by name**, and every skipped path is listed in the manifest with its reason — an omission is an auditable decision, not silence. - **Every stored member carries a SHA-256**, and the archive is reproducible (sorted members, zeroed mtime/uid/gid, `gzip` mtime 0), so two backups of unchanged records are byte-identical. - **Restore fails closed.** It verifies every digest, rejects absolute/traversal/link/unlisted members, refuses a non-empty target, and only then writes. Afterwards it recounts `manifest.jsonl` and refuses if the counts differ from the backup, so a lost tombstone cannot restore quietly. - **`docs/PRODUCTION-RUNBOOK.md`** documents the real topology, the network boundaries (the app's own `HOST` default is `0.0.0.0`; the deployment sets `HOST=127.0.0.1` from the mode-600 env template; the worker refuses a non-loopback publish), the four secrets and their rotation, retention, and both rollback paths. ## Tests - `tests/backup-restore.test.py` — 11 cases: exclusions with reasons, symlink refusal, digest mismatch, unlisted member, traversal, absolute path, link member, non-empty target, record counts, tombstone round trip, byte-identical reproducibility. RED before `scripts/operator_backup.py` existed, GREEN after. - `tests/production-runbook.test.py` — 9 cases: required sections, and **every documented `python3 scripts/...` command is parsed and its flags checked against the real CLI help**, so the runbook cannot drift from the tools. Also fails if `server.mjs` starts persisting data while the runbook claims it keeps no journal, and pins the bind-address claims to `server.mjs`, `deploy/timmy-staging.env.example`, and `deploy/worker/run_worker_container.sh`. - Full suite: `npm test` 230/230, `test:ui`, `test:photo`, `test:mobile-capture`, `test:corpus`, `test:review`, `test:dataset-card`, `test:calibration`, `test:backup`, `test:runbook`, `check:syntax`, `check:diff` — all pass. ## Demonstration `scripts/record_release_demo.mjs` gains an `OPERATOR BACKUP` card that runs the real `backup`, `verify`, and `restore` against a synthetic corpus built by the real ingest and revoke commands, with real decoy files (weight shard, transient upload, env file, log). Every value on the card is read back from those runs, and the four captions are in `DEMO_REQUIRED_CAPTIONS`, so the release builder fails if they never paint. Recorded end-to-end on this branch against a real browser: 91/91 captions pixel-verified, exit 0. Also: the demo's target origin is now `TIMMY_DEMO_URL` with the documented `http://127.0.0.1:4173` default (asserted in `tests/release-demo.test.js`), so a local run can be recorded while the live `4173` listener serves another tree instead of killing it. ## Scope note This PR **does not close #39**. Acceptance criterion 3 is "rollback procedure is exercised on staging": the rollback code path is exercised by `tests/staging-deploy.test.py` against a rootless fixture, but the live drill on the private staging host is a human gate under #53 and is not run automatically by this loop. The runbook says exactly that. No medical/non-diagnostic boundary was moved, no infrastructure was bought, no inference or weights are included, and no production service was touched.

rockachopa added 1 commit 2026-10-29 11:03:15 +00:00

feat(ops): operator backup, restore, and production runbook (#39 )

Quality gates / quality (pull_request) Canceled after 0s

Details

48f0f56835

The app keeps the journal in the person's browser, so the server has no diary to lose. What an
operator can lose is the corpus: the append-only manifest.jsonl with its provenance records and its
withdrawal tombstones. scripts/operator_backup.py archives only records and names every path it
refuses (model weights, transient inference images, secrets, logs, symlinks, oversize) with the
reason, writes one SHA-256 per stored member, and produces a reproducible archive.

restore re-verifies every digest, member name, and record count before it writes a byte, and refuses
absolute, traversal, link, unlisted, and non-empty-target cases, so a tampered archive or a lost
tombstone fails loudly. docs/PRODUCTION-RUNBOOK.md covers topology, network boundaries (including
the app's real 0.0.0.0 default and the staging HOST=127.0.0.1 template), the four secrets, retention,
and both rollback paths; its documented commands are parsed against the real CLIs by its test.

The release demo runs the real backup, verify, and restore against a synthetic corpus built by the
real ingest and revoke commands and shows what was stored, what was refused, and what came back.

rockachopa referenced this pull request

2026-10-29 11:03:25 +00:00

Define production topology, secrets, backup, and rollback runbook #39

timmy merged commit 65a0258bba into main

2026-10-29 12:05:43 +00:00

timmy deleted branch timmy/39-production-runbook-backup

2026-10-29 12:05:45 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: stackchain/timmy-talking-turd#90