timmy-talking-turd/docs/PRODUCTION-RUNBOOK.md

Production topology, secrets, backup, and rollback

This runbook answers one question for the operator: how do the parts of Timmy run, what may they reach, what must never be lost, and how does each part come back? It covers the pieces that exist in this tree today — the Node app, the private vision worker, the private staging directory, and the consented corpus. It does not describe a training fleet, a cloud account, or a hosted database, because this build has none.

Everything below is run by automation or by one operator on the host. Nothing here is performed by a participant, and no step needs a public endpoint.

Topology

Process	Entry point	Holds	Restart unit
App and API	node server.mjs on HOST=127.0.0.1 PORT=4173	Nothing durable. Serves the PWA, proxies /api/vision/*, and gates /api/agent/* behind an access code.	timmy-staging.service (staging), node server.mjs (local)
Vision worker	deploy/worker/run_worker_container.sh	The pinned GGUF weights, mounted read-only from the host.	scripts/run_selfhost_smolvlm.sh start, container restart policy
Staging releases	/opt/timmy-staging/releases/<40-character-commit> with a current symlink	Immutable release trees plus the mode-600 environment file.	timmy-staging.service
Consented corpus	/srv/timmy-corpus	manifest.jsonl (append-only provenance and tombstones) and images/ derived from consented contributions.	none — file data
Caddy	host Caddy	Terminates TLS and routes /git/ to Gitea and /timmy-staging/ to the app.	host Caddy

The app is stateless for a reason: there is no server-side copy of the journal. A person's logs and photos live in their own browser storage, and they hold the portable JSON backup. The server holds no diary, so a lost server loses no one's health record — only the operator records listed above can be lost, and only those are backed up here.

Health and readiness are separate, and the probe reports readiness only:

GET /api/healthz  ->  {"ok":true,"release":"<tag>","commit":"<commit>","visionEnabled":<bool>,"agentEnabled":<bool>}

visionEnabled and agentEnabled say whether the feature is configured, never which key or endpoint is configured. The worker answers /v1/models before its pinned model is resident, so readiness means "served a request from the loaded model", not "the port is open" — scripts/worker_ready.sh enforces that distinction and the app withholds the AI control until it passes.

Network boundaries

• The app reads HOST and its own default is 0.0.0.0, which is a local-development convenience and not the deployment. The deployment sets HOST=127.0.0.1 in the mode-600 environment file (deploy/timmy-staging.env.example is the template) and is reachable from outside solely through the Caddy /timmy-staging/ subpath. A bare node server.mjs on a shared host listens on every interface — treat that as a laptop command, never as a production start.
• The Gitea /git/ catch-all is declared before the staging route and must stay reachable across every Caddy reload.
• The vision worker binds 127.0.0.1:8080 and is never published: deploy/worker/run_worker_container.sh refuses a non-loopback publish host instead of warning about it, and no route in server.mjs forwards arbitrary paths to the worker — the only vision endpoints are /api/vision-status and the analysis request.
• Outbound from the host is limited to the pinned model download during bootstrap, which is refused unless both SHA-256 receipts match. After that the worker needs no egress at all.
• Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a request body. A backup that contained them would be a second copy of someone's photo, so they are excluded by the tool rather than by the operator remembering.
• No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential.

Secrets

Five secrets exist, and every one of them is injected at runtime:

Secret	Source	Scope
TIMMY_DATASET_SALT	mode-600 environment file on the host	Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually.
Vision provider key or worker token	mode-600 environment file on the host	Only the app process reads it; the browser never receives it.
Agent access code	mode-600 environment file on the host	Unlocks /api/agent/unlock; the cookie it mints is HttpOnly and same-origin.
TIMMY_METRICS_TOKEN	mode-600 environment file on the host	Optional. Bearer token for the operator-only /api/metrics snapshot. When it is unset the route returns 404, so an unconfigured deployment exposes no counters at all.
Gitea access token	local operator shell or secret store	Issue and PR automation only. It is never written into a release tree, an archive, or a log.

Rules:

• Never commit a secret, and never pass one on a command line that lands in shell history. Use the environment file (chmod 600, owned by the service user) or the systemd EnvironmentFile=.
• Placeholders in this document are angle-bracketed or environment references, e.g. TIMMY_DATASET_SALT=<random 32+ characters, never committed>. A literal value here is a bug.
• scripts/build_release.py scans the cloned tree for private-key blocks and assigned credentials and fails the release if it finds one, so a secret that reaches a commit is caught before it reaches a release.
• Rotation is: issue the replacement on the host, restart the unit, confirm /api/healthz and one real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a copy that would need rewriting.
• The backup tool refuses .env, *.env, *.key, *.pem, id_rsa, tokens.json, and credentials.json by name, so a backup can be copied off-host without carrying the keys that operate the host.

Backup

The operator records that cannot be rebuilt are the corpus manifest.jsonl and the derived images it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a correctness bug, not a storage detail.

scripts/operator_backup.py archives only records, and writes one manifest naming every byte it stored and every path it refused:

• Stored: manifest.jsonl, derived images, and any receipt files under the data root.
• Excluded, with the reason recorded: model weights (model-weights — re-downloadable, pinned by hash), transient inference directories and uploads (transient-inference — a second copy of someone's photo), secret files (secret), logs (log), symlinks (symlink), and anything over --max-bytes (oversize).
• Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed timestamps and ownership, gzip with a fixed mtime. Two backups of unchanged records are byte-identical, so "did anything change?" is answerable by digest.

python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json

The command writes <archive>.manifest.json unless --manifest names another path, and prints a receipt with the stored file count, archive digest, and record counts (samples, tombstones, activeSamples). It refuses a source that is not a directory and refuses to overwrite an existing archive, so a nightly run cannot erase the previous night.

Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its manifest off-host. Also keep a human-readable record of the counts: verify compares the archive against the manifest, and a restore compares the restored counts against both.

What this backup deliberately does not cover: the browser-held journal (there is no server copy; the person's own portable export is the backup), model weights, and anything in the release tree, which is rebuilt from a commit by scripts/build_release.py.

Restore

A restore never writes over live data. It verifies first, and it writes into a directory that must not exist or must be empty:

python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check

In order, before the first byte is written:

1. the manifest is loaded and its schemaVersion checked;
2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal path, or a name the manifest never listed;
3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore;
4. a member the manifest lists but the archive lacks stops the restore.

After the extract it recounts manifest.jsonl in the target and refuses if the counts differ from the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the restored tree against the live one, and promotes it by moving the directories — the tool does not pretend to know which directory is authoritative.

Restoring the person's own logs is a different operation and stays where it belongs: the app's import control accepts the JSON file they exported, upgrades an older version and says which version it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced.

Rollback

Two rollbacks exist, and only one of them is a live staging action.

Release rollback (staging). Staging keeps one immutable directory per commit and a current symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies the previous release if the new one does not come up:

python3 scripts/deploy_staging.py status
python3 scripts/deploy_staging.py rollback --commit <FULL_COMMIT>

The rollback path is exercised by tests/staging-deploy.test.py, which drives the real scripts/deploy_staging.py code against a rootless fixture (--root), including the case where the named commit has no immutable release directory, where the restart fails, and where the health probe never passes. That is the same code path the host runs; the difference is only the root.

The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the participant-facing page again — is a human gate, tracked under #53. This loop does not run that drill automatically, because a rolled-back staging host is being used for release testing at the time.

Data rollback (operator records). The equivalent for the corpus is the restore above: verify, restore into a new directory, compare counts, then move it into place. A restore is never attempted in place, and a failed verify leaves the live directory untouched.