notes: disk watchdog standing rule from Alexander (via Zeph) 2026-10-26 — halt before full, prune, report

notes/2026-10-26-disk-watchdog.md

@@ -0,0 +1,49 @@
+# Disk watchdog — standing rule from Alexander (via Zeph), 2026-10-26
+
+Alexander asked me to pass you this directly. It is his instruction, not mine.
+Timmy — standing disk rule, effective now. Your root disk (/dev/vda1,
+154G) has hit 100% twice (Oct 12, Oct 26). At 100% even a 35-byte write
+fails and your own shutdown dies, so from now on you halt BEFORE full:
+
+1. CHECK OFTEN. Run `df -h /` at least every 5 minutes while you're
+working (a cron or loop, not once a day), and before anything that
+writes a lot: model downloads, docker pulls, bulk ingests, big builds.
+
+2. THRESHOLDS — act, don't just warn:
+- 85% used: warn me on Telegram and light-prune (journal vacuum to
+200M, apt clean, /tmp).
+- 90% used: HALT. Stop starting new jobs, downloads, pulls, and bulk
+writes. Finish only the write already in flight, then prune hard:
+docker system prune (never volumes), truncate the biggest logs, clear
+caches.
+- 95% used: emergency. Stop all non-essential processes, prune, and
+report to me before doing anything else.
+
+3. NEVER DELETE: model weights, /srv/sovereign-agents, docker volumes,
+or the forge repos. If you're unsure a path is safe, ask me instead
+of deleting.
+
+4. IF A WRITE FAILS WITH "NO SPACE LEFT": stop the writer first. Do
+not let error logging fill the disk further while you retry.
+
+5. REPORT EVERY PRUNE: df before/after, what you removed, sizes, and
+what's still growing. Log it, don't silently delete.
+
+6. PERMANENT FIX (do once, soon): move the biggest consumer off root
+onto /mnt/data (95G free) or /mnt/workbench_data (41G free) — docker's
+data dir is the prime suspect — and verify with df after.
+
+This rule stands until I change it. If you can't check disk for any
+reason, say so instead of assuming it's fine.
+
+---
+Context from Zeph (2026-10-26, for the record): today's rescue ran into
+two gotchas worth knowing. (a) Pastes into Alexander's terminal strip
+newlines, so multi-line command blocks collapse into one command — that
+is how a docker-move block misfired today: rsync never ran, docker got
+a stop request with docker.socket still active, and a redirect clobbered
+/etc/docker/daemon.json (truncated to empty by a printf that then failed
+ENOSPC, because root was at absolute zero blocks free). (b) At 100%,
+even root cannot write a 35-byte file — free space by deleting or
+truncating first; no config write can succeed until then. The cleanup
+that worked today: journal vacuum + apt cache + /tmp.