From 3187715ed93602dc0e4376cf33138e7c5cd19162 Mon Sep 17 00:00:00 2001 From: Zephyrus Sparkborn Date: Mon, 26 Oct 2026 22:13:46 +0000 Subject: [PATCH] =?UTF-8?q?notes:=20disk=20watchdog=20standing=20rule=20fr?= =?UTF-8?q?om=20Alexander=20(via=20Zeph)=202026-10-26=20=E2=80=94=20halt?= =?UTF-8?q?=20before=20full,=20prune,=20report?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- notes/2026-10-26-disk-watchdog.md | 49 +++++++++++++++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100644 notes/2026-10-26-disk-watchdog.md diff --git a/notes/2026-10-26-disk-watchdog.md b/notes/2026-10-26-disk-watchdog.md new file mode 100644 index 0000000..4aa5af6 --- /dev/null +++ b/notes/2026-10-26-disk-watchdog.md @@ -0,0 +1,49 @@ +# Disk watchdog — standing rule from Alexander (via Zeph), 2026-10-26 + +Alexander asked me to pass you this directly. It is his instruction, not mine. +Timmy — standing disk rule, effective now. Your root disk (/dev/vda1, +154G) has hit 100% twice (Oct 12, Oct 26). At 100% even a 35-byte write +fails and your own shutdown dies, so from now on you halt BEFORE full: + +1. CHECK OFTEN. Run `df -h /` at least every 5 minutes while you're +working (a cron or loop, not once a day), and before anything that +writes a lot: model downloads, docker pulls, bulk ingests, big builds. + +2. THRESHOLDS — act, don't just warn: +- 85% used: warn me on Telegram and light-prune (journal vacuum to +200M, apt clean, /tmp). +- 90% used: HALT. Stop starting new jobs, downloads, pulls, and bulk +writes. Finish only the write already in flight, then prune hard: +docker system prune (never volumes), truncate the biggest logs, clear +caches. +- 95% used: emergency. Stop all non-essential processes, prune, and +report to me before doing anything else. + +3. NEVER DELETE: model weights, /srv/sovereign-agents, docker volumes, +or the forge repos. If you're unsure a path is safe, ask me instead +of deleting. + +4. IF A WRITE FAILS WITH "NO SPACE LEFT": stop the writer first. Do +not let error logging fill the disk further while you retry. + +5. REPORT EVERY PRUNE: df before/after, what you removed, sizes, and +what's still growing. Log it, don't silently delete. + +6. PERMANENT FIX (do once, soon): move the biggest consumer off root +onto /mnt/data (95G free) or /mnt/workbench_data (41G free) — docker's +data dir is the prime suspect — and verify with df after. + +This rule stands until I change it. If you can't check disk for any +reason, say so instead of assuming it's fine. + +--- +Context from Zeph (2026-10-26, for the record): today's rescue ran into +two gotchas worth knowing. (a) Pastes into Alexander's terminal strip +newlines, so multi-line command blocks collapse into one command — that +is how a docker-move block misfired today: rsync never ran, docker got +a stop request with docker.socket still active, and a redirect clobbered +/etc/docker/daemon.json (truncated to empty by a printf that then failed +ENOSPC, because root was at absolute zero blocks free). (b) At 100%, +even root cannot write a 35-byte file — free space by deleting or +truncating first; no config write can succeed until then. The cleanup +that worked today: journal vacuum + apt cache + /tmp.