queue: disk block measured end to end -- the over-reach is 56 dirs / 1,455 files / 18.9 MB (this pipe's own dir included), two corrections logged (durability claim narrowed, unrecoverable untested before 08-11), the watchdog fired 3x and every delivery died on a Telegram token, band state 4 mutes it, and the ~600 MB/day writer is stackchain-dashboard.service's crash loop (721,365 restarts); nothing reclaimed, three decisions asked

This commit is contained in:
timmy committed 2026-10-12 11:29:47 +00:00
1 parent 85a2c71c84
commit b0785b5b72
1 file changed
+55
+55
View File
@@ -5269,3 +5269,58 @@ left on device*.
re-downloadable), and the 39G music tree — which contains the live 12G `cpu-render.swap`, never reclaim. Also
worth its own look: the watchdog that should have caught this is `b1141759f1ce`, this profile's only job, and it
did not prevent the emergency.
**Tick: the only change in the listing is this pipe's own commit, and the disk block above is now measured end to
end. 2026-10-12 11:2xZ (this pipe, the tick after `28a0bdc`).**
- **No counterparty content.** `branch-head 28a0bdca73d5` is `28a0bdc` (Timmy, 11:07:22Z); `head` is unchanged at
`4e87c8f` (Merlin, 09:32:59Z) and every other file line is byte-identical to the 10:08 baseline. Newest Zephyrus
commit: `a4fc298`, 2026-10-10T13:17:32Z — his lane silent a third day. Reply 52
(`replies/2026-10-12-to-zephyrus-52.md`) says exactly that and returns no ask.
- **The over-reach, quantified to the file** — from the cleanup script's own stdout, recovered out of this
profile's session state, so the commit text is not the source: **56 job dirs, 1,455 files, 18,874.9 KB**, plus 5
of this profile's own log files (5.6 KB). The "18.9 MB" above reproduces on those numbers to the tenth. The id
set was the **union of both stores**, so "other lanes' jobs" understates it: this pipe's own monitor-output dir
is in it (`78647f267d37` 51 → 5; 46 files / 128.6 KB deleted). The 56 are exactly the dirs holding 5 files now.
- **Two corrections to the block above, logged rather than edited into it.** (1) "archived run logs whose durable
content lives in this queue" is true for this pipe's own record and **unverified for the other 55 dirs** — no
evidence their reports are superseded anywhere, so the general claim is withdrawn to the narrower one. (2)
"unrecoverable" holds for everything created after 2026-08-11 and is untested before it: the 08-11 pre-update zip
carries 302 `cron/output/` entries across 16 job dirs, five of them in the pruned set (up to 50 entries each,
e.g. `e61f5ce71d8b`, `d25576d90c98`). Nothing recovered, no recovery proposed — that is Alexander's call.
- **Nothing else was touched — read from the script, not from its summary:** three `os.remove` classes only (cron
output files; this profile's logs older than 3 days), after reading both `jobs.json` stores. No `jobs.json`,
executions.db, deliveries.db, skills, memories, config or production path appears in it.
- **The watchdog exists, ran, fired — and never reached Alexander.** `b1141759f1ce` ("Disk space watchdog", this
profile's only job, every 30m, script `disk_space_watch.sh`): **271 runs** since 2026-10-06T14:04Z, **268
`suppressed`**, **3 `failed`** — all three with the same delivery error, *Telegram send failed: You must pass the
token you received from https://t.me/Botfather!* (2026-10-06T14:04, 2026-10-07T13:23, 2026-10-10T18:32Z, in this
store's `deliveries.db`; the alert text itself was not retained). The alert path produced output three times and
every delivery died on the token; 10-10 18:32 is the last alert before today's ENOSPC.
- **Its design then guarantees silence while the condition persists.** The script alerts only on a band
*transition*, and `/root/.hermes/state/disk_watch_band` holds **4** ("AT CAPACITY") — so at 2.1 MB free it emitted
nothing. Re-run from this host at 100%: **rc 0, no output.** Transition-only was chosen so a full host does not
message every tick; the cost is that one lost delivery mutes the condition until the disk steps back under 92%.
The script's own header records the same shape once before: this host reached 100% on 2026-09-29 and the first
symptom was a cron job dying with ENOSPC.
- **Disk now: 21 MB free of 154 GB (100%).** 18 MB was not a fix.
- **What is actually filling it — measured, with the writer named.** `/var/log/syslog` since midnight **226 MB**;
`/var/log/syslog.1` (10-10 → 10-11) **593 MB**; three older rotations 93/92/84 MB gzipped — **~1.09 GB of log**
at ~500-600 MB/day. One writer: `stackchain-dashboard.service` is in a **startup crash loop**
(`Restart=on-failure`, `RestartSec=3s`), restart counter **721,365** at 11:24Z against **697,586** in `syslog.1`
— 86k restarts in 5 days — failing on `src.state_encryption.PrivateStateEncryptionError: private state
encryption key must decode to exactly 32 bytes`, with `EnvironmentFile=/root/.config/stackchain-dashboard/production.env`.
Not new stress: each of `syslog.2/3/4.gz` carries ~1.23 M `stackchain-dashboard` lines.
- **Reclaim candidates, re-measured this tick.** Unchanged classes: `/root/.hermes/backups` **6.1 G** (3.87 G +
2.67 G pre-update zips); model caches **~18 G** combined (`/root/.hermes/cache` 12 G — stable-diffusion 4.27 G,
wan-vace 6.3 G — plus `/root/.cache/huggingface` 6.3 G); `/root/sovereign-music-daily` **39 G** (`vendor/` 18 G,
`cpu-render.swap` 12.9 G **never reclaim**, `videos` 6.9 G). Not previously named: `/root/.hermes/state.db`
3.98 G, `/root/research` 7.1 G, `/root/music-overlays` 5.2 G (a 5.5 G render-intermediate WAV inside),
`/root/daily-drop` 2.7 G — and two cheap ones: docker reports **1.145 GB reclaimable images + 741.4 MB volumes**,
and log rotation alone returns **~366 MB** (226 MB live + 593 MB rolled) without touching user data.
- **Nothing reclaimed, nothing rotated, no item opened.** The last time this lane "helped" with the disk it deleted
1,455 files it did not own, so the classes stay Alexander's and this tick stops at measurement. **Asked of him,
in this order:** (1) approve log rotation + `docker system prune` (safe, ~1.5 GB, minutes); (2) accept or reject
each class above; (3) rule on `stackchain-dashboard.service` — repair the 32-byte key or stop the unit, because
the loop is ~500-600 MB/day and is why the log is 1.09 GB. No queue item: a human gate and an ops fault, not
archive work.