2.4 KiB
Disk watchdog — standing rule from Alexander (via Zeph), 2026-10-26
Alexander asked me to pass you this directly. It is his instruction, not mine. Timmy — standing disk rule, effective now. Your root disk (/dev/vda1, 154G) has hit 100% twice (Oct 12, Oct 26). At 100% even a 35-byte write fails and your own shutdown dies, so from now on you halt BEFORE full:
-
CHECK OFTEN. Run
df -h /at least every 5 minutes while you're working (a cron or loop, not once a day), and before anything that writes a lot: model downloads, docker pulls, bulk ingests, big builds. -
THRESHOLDS — act, don't just warn:
- 85% used: warn me on Telegram and light-prune (journal vacuum to 200M, apt clean, /tmp).
- 90% used: HALT. Stop starting new jobs, downloads, pulls, and bulk writes. Finish only the write already in flight, then prune hard: docker system prune (never volumes), truncate the biggest logs, clear caches.
- 95% used: emergency. Stop all non-essential processes, prune, and report to me before doing anything else.
-
NEVER DELETE: model weights, /srv/sovereign-agents, docker volumes, or the forge repos. If you're unsure a path is safe, ask me instead of deleting.
-
IF A WRITE FAILS WITH "NO SPACE LEFT": stop the writer first. Do not let error logging fill the disk further while you retry.
-
REPORT EVERY PRUNE: df before/after, what you removed, sizes, and what's still growing. Log it, don't silently delete.
-
PERMANENT FIX (do once, soon): move the biggest consumer off root onto /mnt/data (95G free) or /mnt/workbench_data (41G free) — docker's data dir is the prime suspect — and verify with df after.
This rule stands until I change it. If you can't check disk for any reason, say so instead of assuming it's fine.
Context from Zeph (2026-10-26, for the record): today's rescue ran into two gotchas worth knowing. (a) Pastes into Alexander's terminal strip newlines, so multi-line command blocks collapse into one command — that is how a docker-move block misfired today: rsync never ran, docker got a stop request with docker.socket still active, and a redirect clobbered /etc/docker/daemon.json (truncated to empty by a printf that then failed ENOSPC, because root was at absolute zero blocks free). (b) At 100%, even root cannot write a 35-byte file — free space by deleting or truncating first; no config write can succeed until then. The cleanup that worked today: journal vacuum + apt cache + /tmp.