feat: add hardened atomic staging deployment
Quality gates / quality (pull_request) Successful in 1m28s
Quality gates / quality (pull_request) Successful in 1m28s
This commit is contained in:
1 parent
fc763fb758
commit
f8f01c4b2e
15 files changed
+1248
-6
No files matched your search
@@ -0,0 +1,118 @@
|
||||
# Timmy private staging runbook
|
||||
|
||||
This runbook is a reviewed host template, not approval to change a live host. Apply it only in a separately approved provisioning change. Staging is Phase 1: Hermes Agent and vision are off.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Linux host with Node.js 22, systemd, and Caddy.
|
||||
- Existing HTTPS origin and Gitea route inventory.
|
||||
- A reviewed 40-character commit, source archive, and SHA-256 receipt from the release manifest.
|
||||
- Root only for one-time account/unit/config installation; promotions use the narrowest available sudo policy.
|
||||
- A fresh Caddy password hash delivered outside Git. Never put the password or hash in shell history, release notes, screenshots, or this repository.
|
||||
|
||||
Confirm that `127.0.0.1:4174` is unused. Do not copy `.env`, credentials, model weights, raw media, or a mutable Git checkout into a release.
|
||||
|
||||
## DNS and URL
|
||||
|
||||
No DNS change is required for the approved subpage deployment. The private URL is `https://forge.alexanderwhitestone.com/timmy-staging/`. Caddy remains the only public listener; Node binds only to `127.0.0.1:4174`. Verify `/git/` before and after any separately approved Caddy reload.
|
||||
|
||||
## Install
|
||||
|
||||
These commands are reference commands for an approved maintenance window; they have not been run by this change:
|
||||
|
||||
```bash
|
||||
NODE_SOURCE="$(readlink -f "$(command -v node)")"
|
||||
"$NODE_SOURCE" --version # must report v22.x before installation
|
||||
sudo install -D -o root -g root -m 755 "$NODE_SOURCE" /usr/local/lib/timmy-staging/node
|
||||
/usr/local/lib/timmy-staging/node --version
|
||||
sudo useradd --system --home-dir /nonexistent --no-create-home --shell /usr/sbin/nologin timmy-staging
|
||||
sudo install -d -o root -g root -m 755 /opt/timmy-staging/releases
|
||||
sudo install -d -o timmy-staging -g timmy-staging -m 700 /var/lib/timmy-staging
|
||||
sudo install -o root -g root -m 644 deploy/timmy-staging.service /etc/systemd/system/timmy-staging.service
|
||||
sudo install -o root -g root -m 600 deploy/timmy-staging.env.example /etc/timmy-staging.env
|
||||
sudo chmod 600 /etc/timmy-staging.env
|
||||
sudo systemctl daemon-reload
|
||||
```
|
||||
|
||||
Edit only `TIMMY_RELEASE_TAG` and `TIMMY_RELEASE_COMMIT` for the selected artifact. The unit deliberately applies `TIMMY_AGENT_ENABLED=false` and `TIMMY_VISION_ENABLED=false` on the `ExecStart` command after loading the environment file, so values in that file cannot enable either subsystem. Leave the loopback host, port, and base path unchanged. The environment file must stay root-owned mode 600 and absent from release archives. The copied Node 22 executable is root-owned at `/usr/local/lib/timmy-staging/node`, outside every home directory, so `ProtectHome=true` remains enforceable; replace it only through a separately reviewed Node upgrade.
|
||||
|
||||
Generate Caddy basic-auth material interactively (for example, `caddy hash-password`) and inject it as `TIMMY_STAGING_PASSWORD_HASH`; do not commit the output. Merge the staging handle before the existing catchall, validate a temporary complete config, then use `caddy reload` rather than restarting unrelated services.
|
||||
|
||||
## Promote an immutable release
|
||||
|
||||
First inspect without mutation:
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_staging.py --dry-run promote \
|
||||
--tag daily-YYYY-MM-DD.N --archive /secure/inbox/timmy.tar.gz \
|
||||
--sha256 64_LOWERCASE_HEX_CHARACTERS --commit 40_LOWERCASE_HEX_CHARACTERS
|
||||
```
|
||||
|
||||
Then run the same command without `--dry-run`. Promotion opens only a non-symlink regular source, copies it while hashing into a mode-400 file in a private temporary directory, and inspects and extracts only that verified copy before cleaning it. It rejects unsafe members and forbidden artifacts, extracts once to `/opt/timmy-staging/releases/<commit>`, and never overwrites that directory. It atomically swaps `current`, restarts only `timmy-staging.service`, checks bounded health, and invokes `npm run test:staging-smoke` as an argv array with `shell=False`. Restart, health, or smoke failure automatically restores and verifies the prior symlink. If there is no prior release, the tool removes `current` and stops the service instead of restarting it against a missing path; the failed release remains unreferenced as inert immutable evidence for operator review.
|
||||
|
||||
The operator must update `/etc/timmy-staging.env` release identity to the same reviewed tag and commit before promotion. A narrow wrapper/sudo policy may set the smoke environment without granting arbitrary command execution:
|
||||
|
||||
```bash
|
||||
export TIMMY_STAGING_URL=https://forge.alexanderwhitestone.com/timmy-staging/
|
||||
export TIMMY_STAGING_USER=staging
|
||||
# Read TIMMY_STAGING_PASSWORD from an approved secret channel; never paste it here.
|
||||
sudo -E python3 scripts/deploy_staging.py promote --tag "$TAG" --archive "$ARCHIVE" --sha256 "$SHA256" --commit "$COMMIT"
|
||||
```
|
||||
|
||||
## Smoke test
|
||||
|
||||
The promotion runs the synthetic 390×844 Playwright suite. It checks edge auth, build identity, one dominant photo action, manual save, urgent-language interception, Journal visibility, export, no horizontal overflow, browser errors, and suspicious secret-bearing responses. It must never use a real stool photo or medical record.
|
||||
|
||||
For an explicit rerun:
|
||||
|
||||
```bash
|
||||
TIMMY_STAGING_URL=https://forge.alexanderwhitestone.com/timmy-staging/ \
|
||||
TIMMY_EXPECT_RELEASE_TAG="$TAG" TIMMY_EXPECT_RELEASE_COMMIT="$COMMIT" \
|
||||
TIMMY_STAGING_USER=staging TIMMY_STAGING_PASSWORD="$(secret-reader)" \
|
||||
npm run test:staging-smoke
|
||||
```
|
||||
|
||||
Also verify `curl --fail http://127.0.0.1:4174/timmy-staging/api/healthz`, authenticated public health, unauthenticated HTTP 401, `/git/`, and that no wildcard/public listener owns port 4174.
|
||||
|
||||
## Status and logs
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_staging.py status
|
||||
sudo systemctl status timmy-staging.service
|
||||
sudo journalctl -u timmy-staging.service --since today --no-pager
|
||||
# Only after an approved environment-file change:
|
||||
sudo systemctl restart timmy-staging.service
|
||||
```
|
||||
|
||||
Health and logs may show bounded release identity, but must not show environment values, cookies, passwords, tokens, session data, filesystem secrets, or raw user content.
|
||||
|
||||
## Rollback
|
||||
|
||||
List reviewed immutable commit directories, choose the known-good receipt, dry-run, then roll back:
|
||||
|
||||
```bash
|
||||
python3 scripts/deploy_staging.py --dry-run rollback --commit 40_LOWERCASE_HEX_CHARACTERS
|
||||
sudo python3 scripts/deploy_staging.py rollback --commit 40_LOWERCASE_HEX_CHARACTERS
|
||||
```
|
||||
|
||||
Rollback atomically repoints `current`, restarts only Timmy, and verifies commit-specific loopback health. If rollback verification fails, the tool restores the release that was current when rollback began. Never repoint the symlink manually while the service is running.
|
||||
|
||||
## Backup
|
||||
|
||||
Browser journal data is local to each browser and is not server state. Before host maintenance, back up only operator-owned receipts and any explicitly required `/var/lib/timmy-staging` state with root-only permissions. Release archives should be recoverable from their verified release source; do not back up `/etc/timmy-staging.env` into a general artifact store. Test restore procedures without live credentials.
|
||||
|
||||
## Template validation
|
||||
|
||||
```bash
|
||||
node --test tests/staging-config.test.js
|
||||
systemd-analyze verify deploy/timmy-staging.service
|
||||
HASH="$(caddy hash-password --plaintext validation-only-password)"
|
||||
TIMMY_STAGING_PASSWORD_HASH="$HASH" caddy validate --config deploy/Caddyfile.staging.example --adapter caddyfile
|
||||
python3 -m py_compile scripts/deploy_staging.py
|
||||
```
|
||||
|
||||
Use only a disposable validation hash. Validation does not authorize installation or reload.
|
||||
|
||||
## Remove staging
|
||||
|
||||
In a separately approved maintenance window: disable and stop only `timmy-staging.service`; remove only the reviewed Caddy staging handle and validate/reload Caddy; verify `/git/`; remove the unit and run `systemctl daemon-reload`; archive required receipts; then remove `/opt/timmy-staging` and `/var/lib/timmy-staging`. Delete the dedicated locked user last. Do not delete shared Caddy, Gitea, TLS, or browser data.
|
||||
Reference in new issue
Block a user