timmy-talking-turd/docs/PRODUCTION-RUNBOOK.md

# Production topology, secrets, backup, and rollback

This runbook answers one question for the operator: how do the parts of Timmy run, what may they
reach, what must never be lost, and how does each part come back? It covers the pieces that exist in
this tree today — the Node app, the private vision worker, the private staging directory, and the
consented corpus. It does not describe a training fleet, a cloud account, or a hosted database,
because this build has none.

Everything below is run by automation or by one operator on the host. Nothing here is performed by a
participant, and no step needs a public endpoint.

## Topology

| Process | Entry point | Holds | Restart unit |
|---|---|---|---|
| App and API | `node server.mjs` on `HOST=127.0.0.1 PORT=4173` | Nothing durable. Serves the PWA, proxies `/api/vision/*`, and gates `/api/agent/*` behind an access code. | `timmy-staging.service` (staging), `node server.mjs` (local) |
| Vision worker | `deploy/worker/run_worker_container.sh` | The pinned GGUF weights, mounted read-only from the host. | `scripts/run_selfhost_smolvlm.sh start`, container restart policy |
| Staging releases | `/opt/timmy-staging/releases/<40-character-commit>` with a `current` symlink | Immutable release trees plus the mode-600 environment file. | `timmy-staging.service` |
| Consented corpus | `/srv/timmy-corpus` | `manifest.jsonl` (append-only provenance and tombstones) and `images/` derived from consented contributions. | none — file data |
| Caddy | host Caddy | Terminates TLS and routes `/git/` to Gitea and `/timmy-staging/` to the app. | host Caddy |

The app is stateless for a reason: there is **no server-side copy of the journal**. A person's logs
and photos live in their own browser storage, and they hold the portable JSON backup. The server
holds no diary, so a lost server loses no one's health record — only the operator records listed
above can be lost, and only those are backed up here.

Health and readiness are separate, and the probe reports readiness only:

```
GET /api/healthz  ->  {"ok":true,"release":"<tag>","commit":"<commit>","visionEnabled":<bool>,"agentEnabled":<bool>}
```

`visionEnabled` and `agentEnabled` say whether the feature is configured, never which key or endpoint
is configured. The worker answers `/v1/models` before its pinned model is resident, so readiness
means "served a request from the loaded model", not "the port is open" — `scripts/worker_ready.sh`
enforces that distinction and the app withholds the AI control until it passes.

## Network boundaries

- The app reads `HOST` and its own default is `0.0.0.0`, which is a local-development convenience and
  not the deployment. The deployment sets `HOST=127.0.0.1` in the mode-600 environment file
  (`deploy/timmy-staging.env.example` is the template) and is reachable from outside solely through
  the Caddy `/timmy-staging/` subpath. A bare `node server.mjs` on a shared host listens on every
  interface — treat that as a laptop command, never as a production start.
- The Gitea `/git/` catch-all is declared before the staging route and must stay reachable across
  every Caddy reload.
- The vision worker binds `127.0.0.1:8080` and is never published: `deploy/worker/run_worker_container.sh`
  refuses a non-loopback publish host instead of warning about it, and no route in `server.mjs`
  forwards arbitrary paths to the worker — the only vision endpoints are `/api/vision-status` and the
  analysis request.
- Outbound from the host is limited to the pinned model download during bootstrap, which is refused
  unless both SHA-256 receipts match. After that the worker needs no egress at all.
- Inference bodies and image bytes are never logged, and no reverse-proxy access log retains a
  request body. A backup that contained them would be a second copy of someone's photo, so they are
  excluded by the tool rather than by the operator remembering.
- No participant-facing endpoint requires an inbound port, a DNS record, or a cloud credential.

## Secrets

Five secrets exist, and every one of them is injected at runtime:

| Secret | Source | Scope |
|---|---|---|
| `TIMMY_DATASET_SALT` | mode-600 environment file on the host | Corpus subject keys and split assignment. Rotating it re-splits future ingests, so it is not rotated casually. |
| Vision provider key or worker token | mode-600 environment file on the host | Only the app process reads it; the browser never receives it. |
| Agent access code | mode-600 environment file on the host | Unlocks `/api/agent/unlock`; the cookie it mints is `HttpOnly` and same-origin. |
| `TIMMY_METRICS_TOKEN` | mode-600 environment file on the host | Optional. Bearer token for the operator-only `/api/metrics` snapshot. When it is unset the route returns 404, so an unconfigured deployment exposes no counters at all. |
| Gitea access token | local operator shell or secret store | Issue and PR automation only. It is never written into a release tree, an archive, or a log. |

Rules:

- Never commit a secret, and never pass one on a command line that lands in shell history. Use the
  environment file (`chmod 600`, owned by the service user) or the systemd `EnvironmentFile=`.
- Placeholders in this document are angle-bracketed or environment references, e.g.
  `TIMMY_DATASET_SALT=<random 32+ characters, never committed>`. A literal value here is a bug.
- `scripts/build_release.py` scans the cloned tree for private-key blocks and assigned credentials
  and fails the release if it finds one, so a secret that reaches a commit is caught before it
  reaches a release.
- Rotation is: issue the replacement on the host, restart the unit, confirm `/api/healthz` and one
  real inference request, then revoke the old value in the provider's own UI. Nothing in Git holds a
  copy that would need rewriting.
- The backup tool refuses `.env`, `*.env`, `*.key`, `*.pem`, `id_rsa`, `tokens.json`, and
  `credentials.json` by name, so a backup can be copied off-host without carrying the keys that
  operate the host.

## Backup

The operator records that cannot be rebuilt are the corpus `manifest.jsonl` and the derived images
it points at. A tombstone is the only proof that a participant withdrew, so a backup that drops
tombstone lines silently re-authorises training on withdrawn data. The tool treats that as a
correctness bug, not a storage detail.

`scripts/operator_backup.py` archives **only** records, and writes one manifest naming every byte it
stored and every path it refused:

- Stored: `manifest.jsonl`, derived images, and any receipt files under the data root.
- Excluded, with the reason recorded: **model weights** (`model-weights` — re-downloadable, pinned by
  hash), **transient inference** directories and uploads (`transient-inference` — a second copy of
  someone's photo), **secret** files (`secret`), logs (`log`), symlinks (`symlink`), and anything over
  `--max-bytes` (`oversize`).
- Every stored member carries a SHA-256, and the archive is reproducible: sorted members, zeroed
  timestamps and ownership, `gzip` with a fixed mtime. Two backups of unchanged records are
  byte-identical, so "did anything change?" is answerable by digest.

```bash
python3 scripts/operator_backup.py backup --source /srv/timmy-corpus --archive /srv/backups/timmy-nightly.tar.gz --label nightly
python3 scripts/operator_backup.py verify --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json
```

The command writes `<archive>.manifest.json` unless `--manifest` names another path, and prints a
receipt with the stored file count, archive digest, and record counts (`samples`, `tombstones`,
`activeSamples`). It refuses a source that is not a directory and refuses to overwrite an existing
archive, so a nightly run cannot erase the previous night.

Retention: keep fourteen nightlies and the first backup of each month, and copy each archive plus its
manifest off-host. Also keep a human-readable record of the counts: `verify` compares the archive
against the manifest, and a restore compares the restored counts against both.

What this backup deliberately does not cover: the browser-held journal (there is no server copy; the
person's own portable export is the backup), model weights, and anything in the release tree, which
is rebuilt from a commit by `scripts/build_release.py`.

## Restore

A restore never writes over live data. It verifies first, and it writes into a directory that must
not exist or must be empty:

```bash
python3 scripts/operator_backup.py restore --archive /srv/backups/timmy-nightly.tar.gz --manifest /srv/backups/timmy-nightly.tar.gz.manifest.json --target /srv/timmy-corpus.restore-check
```

In order, before the first byte is written:

1. the manifest is loaded and its `schemaVersion` checked;
2. every archive member is refused if it is a link, a non-regular file, an absolute path, a traversal
   path, or a name the manifest never listed;
3. every member's SHA-256 is recomputed and compared, and a mismatch stops the restore;
4. a member the manifest lists but the archive lacks stops the restore.

After the extract it recounts `manifest.jsonl` in the target and refuses if the counts differ from
the backup, so a restore that silently lost a tombstone fails loudly. Then the operator compares the
restored tree against the live one, and promotes it by moving the directories — the tool does not
pretend to know which directory is authoritative.

Restoring the person's own logs is a different operation and stays where it belongs: the app's
import control accepts the JSON file they exported, upgrades an older version and says which version
it read, and refuses a file from a newer Timmy without touching the ledger it would have replaced.

## Rollback

Two rollbacks exist, and only one of them is a live staging action.

**Release rollback (staging).** Staging keeps one immutable directory per commit and a `current`
symlink, so a rollback is a repoint, a restart, and a bounded health check that restores and verifies
the previous release if the new one does not come up:

```bash
python3 scripts/deploy_staging.py status
python3 scripts/deploy_staging.py rollback --commit <FULL_COMMIT>
```

The rollback path is exercised by `tests/staging-deploy.test.py`, which drives the real
`scripts/deploy_staging.py` code against a rootless fixture (`--root`), including the case where the
named commit has no immutable release directory, where the restart fails, and where the health probe
never passes. That is the same code path the host runs; the difference is only the root.

The live drill on the private staging host — promote a build, serve it, roll it back, and confirm the
participant-facing page again — is a **human gate**, tracked under #53. This loop does not run that
drill automatically, because a rolled-back staging host is being used for release testing at the time.

**Data rollback (operator records).** The equivalent for the corpus is the restore above: verify,
restore into a new directory, compare counts, then move it into place. A restore is never attempted in
place, and a failed verify leaves the live directory untouched.