stackchain/timmy-talking-turd
1
0
Fork 0
Code Issues 41 Pull Requests 6 Actions Packages Projects Releases 15 Wiki Activity
b8532f587d
timmy-talking-turd/docs/STAGING-RUNBOOK.md
Timmy f8f01c4b2e
All checks were successful
Quality gates / quality (pull_request) Successful in 1m28s
Details
feat: add hardened atomic staging deployment
2026-08-21 14:49:28 +00:00

7.7 KiB
Raw Blame History

Timmy private staging runbook

This runbook is a reviewed host template, not approval to change a live host. Apply it only in a separately approved provisioning change. Staging is Phase 1: Hermes Agent and vision are off.

Prerequisites

  • Linux host with Node.js 22, systemd, and Caddy.
  • Existing HTTPS origin and Gitea route inventory.
  • A reviewed 40-character commit, source archive, and SHA-256 receipt from the release manifest.
  • Root only for one-time account/unit/config installation; promotions use the narrowest available sudo policy.
  • A fresh Caddy password hash delivered outside Git. Never put the password or hash in shell history, release notes, screenshots, or this repository.

Confirm that 127.0.0.1:4174 is unused. Do not copy .env, credentials, model weights, raw media, or a mutable Git checkout into a release.

DNS and URL

No DNS change is required for the approved subpage deployment. The private URL is https://forge.alexanderwhitestone.com/timmy-staging/. Caddy remains the only public listener; Node binds only to 127.0.0.1:4174. Verify /git/ before and after any separately approved Caddy reload.

Install

These commands are reference commands for an approved maintenance window; they have not been run by this change:

NODE_SOURCE="$(readlink -f "$(command -v node)")"
"$NODE_SOURCE" --version  # must report v22.x before installation
sudo install -D -o root -g root -m 755 "$NODE_SOURCE" /usr/local/lib/timmy-staging/node
/usr/local/lib/timmy-staging/node --version
sudo useradd --system --home-dir /nonexistent --no-create-home --shell /usr/sbin/nologin timmy-staging
sudo install -d -o root -g root -m 755 /opt/timmy-staging/releases
sudo install -d -o timmy-staging -g timmy-staging -m 700 /var/lib/timmy-staging
sudo install -o root -g root -m 644 deploy/timmy-staging.service /etc/systemd/system/timmy-staging.service
sudo install -o root -g root -m 600 deploy/timmy-staging.env.example /etc/timmy-staging.env
sudo chmod 600 /etc/timmy-staging.env
sudo systemctl daemon-reload

Edit only TIMMY_RELEASE_TAG and TIMMY_RELEASE_COMMIT for the selected artifact. The unit deliberately applies TIMMY_AGENT_ENABLED=false and TIMMY_VISION_ENABLED=false on the ExecStart command after loading the environment file, so values in that file cannot enable either subsystem. Leave the loopback host, port, and base path unchanged. The environment file must stay root-owned mode 600 and absent from release archives. The copied Node 22 executable is root-owned at /usr/local/lib/timmy-staging/node, outside every home directory, so ProtectHome=true remains enforceable; replace it only through a separately reviewed Node upgrade.

Generate Caddy basic-auth material interactively (for example, caddy hash-password) and inject it as TIMMY_STAGING_PASSWORD_HASH; do not commit the output. Merge the staging handle before the existing catchall, validate a temporary complete config, then use caddy reload rather than restarting unrelated services.

Promote an immutable release

First inspect without mutation:

python3 scripts/deploy_staging.py --dry-run promote \
  --tag daily-YYYY-MM-DD.N --archive /secure/inbox/timmy.tar.gz \
  --sha256 64_LOWERCASE_HEX_CHARACTERS --commit 40_LOWERCASE_HEX_CHARACTERS

Then run the same command without --dry-run. Promotion opens only a non-symlink regular source, copies it while hashing into a mode-400 file in a private temporary directory, and inspects and extracts only that verified copy before cleaning it. It rejects unsafe members and forbidden artifacts, extracts once to /opt/timmy-staging/releases/<commit>, and never overwrites that directory. It atomically swaps current, restarts only timmy-staging.service, checks bounded health, and invokes npm run test:staging-smoke as an argv array with shell=False. Restart, health, or smoke failure automatically restores and verifies the prior symlink. If there is no prior release, the tool removes current and stops the service instead of restarting it against a missing path; the failed release remains unreferenced as inert immutable evidence for operator review.

The operator must update /etc/timmy-staging.env release identity to the same reviewed tag and commit before promotion. A narrow wrapper/sudo policy may set the smoke environment without granting arbitrary command execution:

export TIMMY_STAGING_URL=https://forge.alexanderwhitestone.com/timmy-staging/
export TIMMY_STAGING_USER=staging
# Read TIMMY_STAGING_PASSWORD from an approved secret channel; never paste it here.
sudo -E python3 scripts/deploy_staging.py promote --tag "$TAG" --archive "$ARCHIVE" --sha256 "$SHA256" --commit "$COMMIT"

Smoke test

The promotion runs the synthetic 390×844 Playwright suite. It checks edge auth, build identity, one dominant photo action, manual save, urgent-language interception, Journal visibility, export, no horizontal overflow, browser errors, and suspicious secret-bearing responses. It must never use a real stool photo or medical record.

For an explicit rerun:

TIMMY_STAGING_URL=https://forge.alexanderwhitestone.com/timmy-staging/ \
TIMMY_EXPECT_RELEASE_TAG="$TAG" TIMMY_EXPECT_RELEASE_COMMIT="$COMMIT" \
TIMMY_STAGING_USER=staging TIMMY_STAGING_PASSWORD="$(secret-reader)" \
npm run test:staging-smoke

Also verify curl --fail http://127.0.0.1:4174/timmy-staging/api/healthz, authenticated public health, unauthenticated HTTP 401, /git/, and that no wildcard/public listener owns port 4174.

Status and logs

python3 scripts/deploy_staging.py status
sudo systemctl status timmy-staging.service
sudo journalctl -u timmy-staging.service --since today --no-pager
# Only after an approved environment-file change:
sudo systemctl restart timmy-staging.service

Health and logs may show bounded release identity, but must not show environment values, cookies, passwords, tokens, session data, filesystem secrets, or raw user content.

Rollback

List reviewed immutable commit directories, choose the known-good receipt, dry-run, then roll back:

python3 scripts/deploy_staging.py --dry-run rollback --commit 40_LOWERCASE_HEX_CHARACTERS
sudo python3 scripts/deploy_staging.py rollback --commit 40_LOWERCASE_HEX_CHARACTERS

Rollback atomically repoints current, restarts only Timmy, and verifies commit-specific loopback health. If rollback verification fails, the tool restores the release that was current when rollback began. Never repoint the symlink manually while the service is running.

Backup

Browser journal data is local to each browser and is not server state. Before host maintenance, back up only operator-owned receipts and any explicitly required /var/lib/timmy-staging state with root-only permissions. Release archives should be recoverable from their verified release source; do not back up /etc/timmy-staging.env into a general artifact store. Test restore procedures without live credentials.

Template validation

node --test tests/staging-config.test.js
systemd-analyze verify deploy/timmy-staging.service
HASH="$(caddy hash-password --plaintext validation-only-password)"
TIMMY_STAGING_PASSWORD_HASH="$HASH" caddy validate --config deploy/Caddyfile.staging.example --adapter caddyfile
python3 -m py_compile scripts/deploy_staging.py

Use only a disposable validation hash. Validation does not authorize installation or reload.

Remove staging

In a separately approved maintenance window: disable and stop only timmy-staging.service; remove only the reviewed Caddy staging handle and validate/reload Caddy; verify /git/; remove the unit and run systemctl daemon-reload; archive required receipts; then remove /opt/timmy-staging and /var/lib/timmy-staging. Delete the dedicated locked user last. Do not delete shared Caddy, Gitea, TLS, or browser data.