From 300228b44c3968288fcef79cacac68655dc4e7b3 Mon Sep 17 00:00:00 2001 From: Merlin Date: Sun, 11 Oct 2026 13:31:39 +0000 Subject: [PATCH] Q31: collector's entry point executed, destination gated, G8 gate (Merlin run 6) --- WORK-QUEUE.md | 2 + checkins/2026-10-11-merlin.md | 85 ++++++++++- recoveries/helm-collector-run.md | 252 +++++++++++++++++++++++++++++++ 3 files changed, 338 insertions(+), 1 deletion(-) create mode 100644 recoveries/helm-collector-run.md diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index 7cf4bc9..c4ef3d7 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -3963,6 +3963,8 @@ Live state, re-read 11:31Z: `/var/www/…/helm/index.html` and `https://alexande 9,865 B / `427b7b54…`, **5 × FRESH / 0 × STALE** — five false freshness claims still in production. The honest render for comparison sits at `/tmp/helm-honest-check.html` (11,806 B / `8b839754…`). +**Status: DONE at Merlin's Run 6 (2026-10-11 ≈13:19–13:35Z) — deliverable `recoveries/helm-collector-run.md`; the collector's entry point is executed, its destination is gated, and G8 asserts it.** The item's four steps: (1) the two safe legs run — `--dry-run --skip-tests` exit 0 / 15,187 B pure-JSON stdout and `--out ` exit 0 / 15,187 B, both with the live snapshot unchanged (`aa842b0f…`); the two bodies are the same measurement (only `observed_at` clocks and one `rtt_seconds` differ, `q31-compare.py`). (2) **The destination rule: `SNAPSHOT_PATH` is gated behind an explicit `--deploy`** — the same shape as the renderer's `OUT_DEFAULT` gate, so both entry points now behave alike; `--dry-run` is never gated (it writes nothing), a scratch `--out` is the default expectation, and the live path (bare default *or* explicit `--out`, matched by `.resolve()`) is refused at exit 2 **before any collector runs** (0.13 s refusal vs ~1.9 s collect). Gated collector `60229227216fb4ab…` (701 lines / 28,808 B). (3) **Gate G8 — the collector's CLI in a subprocess, 18 assertions** → `helm-selftest: 137 passed, 0 failed`, VERDICT: ALL GATES PASS; the gate discriminates (pre-gate `ee0b36a8…` with `SNAPSHOT_PATH` patched to scratch writes its default bare; gated refuses). (4) **`email` UNREACHABLE is correct** — `himalaya` is ABSENT on PATH and both config paths are absent, so `collect_email()` fails closed with `himalaya unavailable: command not found: himalaya` and lands in `red_sources`, never `green`. **An incident, logged first in the deliverable §6:** the first negative control ran the pre-gate revision bare against the live default and overwrote the live snapshot; restored byte-exact from a pre-run `snap-before.json` (`aa842b0f…`, 14,039 B, mtime back to `2026-10-08T17:12:59.475780303Z`), and the control now patches `SNAPSHOT_PATH` before any run. **Nothing deployed** — the gated collector and G8 selftest are scratch-local in `/root/crucible/q31/`; the deployed `helm-collect.py` is unchanged at `ee0b36a8…` and `helm-selftest.py` at `7ed1bca8…` (still 119 passed). Adoption is a §4 deploy decision, Alexander's, the same gate Q26–Q30 state. Mercury's day-3 unit is Q31's first slice (the two safe legs); this run executed them first and Mercury's `/tmp/mercury-q31/` was empty at 13:29Z, so no duplication occurred — the unit is satisfied here and stays open for Mercury to land his own artifact. + **Mercury lane — day-3 pin, 2026-10-11 12:04Z (control plane).** Mercury's day-2 check-in (`b50c5ec`, `checkins/2026-10-11-mercury.md`) re-ran the census and the digest and produced no new artifact — the second consecutive check-in in that shape. The census is closed evidence: 121 `
  • Screen. Every claim carries a path, a hash, a URL, or a command with its output, or the word +> `unverified`. Nothing secret is printed here or in the files it lands. + +## Did + +Claimed and finished **Q31 — the collector's entry point** (reply 13 §3.3 named it to me first as the +Helm's author; Q20 is zephyrus-sparkborn's, Q9 blocked on the audio handover, Q13/Q19 gated on +Alexander). Deliverable: `recoveries/helm-collector-run.md` (**13,592 B / sha256 +`580c4640ff98404fd2a9e985095448123ee7c185cd703ef675fb93248596dd18`**), scratch tree +`/root/crucible/q31/` (all scripts re-runnable: `q31-matrix.py`, `q31-negative.py`, `q31-email.py`, +`q31-probe.py`, `q31-compare.py`, `q31-diff.py`, the gated collector, the pre-gate copy, the G8 +selftest, `DIFF.txt`, logs). + +**An incident, mine, logged first.** The first negative control ran the **pre-gate** collector bare +against the **live** default and overwrote `/root/.helm/data/snapshot.json` (a fresh collect, +`4088d5a8…`, mtime `2026-10-11 13:28:32Z`). I had taken a pre-run copy (`snap-before.json`, 13:19Z, +before any run) and restored it **byte-exact** — digest back to `aa842b0f…`, 14,039 B, mtime set back +to `2026-10-08T17:12:59.475780303Z` (86 ns high, stated). The negative control now patches +`SNAPSHOT_PATH` to scratch before any run and can never touch the live default. The same accident +class as Q30's module-scope overwrite; the gate shipped is the fix. + +**Step 1 — the two safe legs, executed.** `--dry-run --skip-tests` → exit 0, 15,187 B pure-JSON +stdout; `--out --skip-tests` → exit 0, 15,187 B, `verdict.all_sources_green=False`. Live +snapshot unchanged across both (`aa842b0f…`, mtime `2026-10-08T17:12:59.475780303Z`). The two bodies +are the same measurement — only `evidence.observed_at` clocks and one `rtt_seconds` differ +(`q31-compare.py`). Fresh collect: `email` UNREACHABLE, five sources OK+FRESH, `all_sources_green +False`. + +**Step 4 — `email` UNREACHABLE is correct.** `himalaya` is ABSENT on PATH; both config paths absent; +`collect_email()` fails closed with `himalaya unavailable: command not found: himalaya` and lands in +`red_sources`, never `green`. A silent-by-design failure that says why, not a bug. Installing +himalaya is a host change, Alexander's. + +**Step 2 — the destination rule.** `SNAPSHOT_PATH` gated behind an explicit `--deploy` — the same +shape as the renderer's `OUT_DEFAULT` gate, so both entry points behave alike. `--dry-run` is never +gated (writes nothing); a scratch `--out` is the default expectation; the live path (bare default +**or** explicit `--out`, matched by `.resolve()`) is refused at exit 2 **before any collector runs** +— measured 0.13 s refusal vs ~1.9 s collect. Gated collector `60229227216fb4ab…` (701 lines / +28,808 B; `DIFF.txt` +14 lines in `main()` plus a docstring note). Behaviour matrix, five legs, all +`untouched=yes` around the live snapshot's digest+mtime. + +**Step 3 — gate G8, 18 assertions.** `HELM_COLLECTOR= python3 helm-selftest.py` → +**137 passed, 0 failed, VERDICT: ALL GATES PASS** (18 G8 + the standing 119). The gate runs the CLI +in a subprocess, asserts the emitted bytes, the destination gate's negative side (bare default and +explicit live `--out`, both refused, both writing nothing), `--dry-run` never gated at the live path, +`--deploy` against a patched `SNAPSHOT_PATH`, and two CLI runs agreeing byte-for-byte once the clock +fields are recursively scrubbed. **It discriminates:** the pre-gate `ee0b36a8…` (SNAPSHOT_PATH patched +to scratch) writes its default bare; the gated refuses. + +**Nothing deployed.** Gated collector and G8 selftest are scratch-local; deployed `helm-collect.py` +unchanged at `ee0b36a8…`, `helm-selftest.py` at `7ed1bca8…` (still 119 passed). Live snapshot +restored and re-verified `aa842b0f…`. `decisions/` and `inbox/` untouched; nothing executed; no +another-party file edited; no X page fetched. + +## Blocked + +- Nothing blocked the run. **Unverified by construction:** the gate uses `--skip-tests` so the ~2 s + collector is deterministic and does not invoke the Timmy suite — the full suite is not exercised by + the gate (stated, not hidden); the real bare `--deploy` writes the live snapshot and was never run + (§4, Alexander's); the restore's mtime is 86 ns high (content digest exact). +- **Mercury's day-3 unit is Q31's first slice** (the two safe legs). This run executed them first; + `/tmp/mercury-q31/` was empty at 13:29Z, so no duplication — the slice is satisfied here and stays + open for Mercury to land his own artifact. + +## Next + +- **Alexander's call, §4:** adopt `60229227216fb4ab` + the G8 selftest into `/root/.helm/bin/` + (`cp`, two files) — the same publish gate Q26–Q30 state — then the deploy/schedule decision for + the collector. The gate makes an unattended collect+render safe to schedule. +- Q20 remains the top unclaimed item and is zephyrus-sparkborn's; I take nothing over it. + +## Ask + +1. **Alexander — adopt the gated collector + G8 into the shared tree?** It is the difference between + a bare run silently overwriting the live snapshot and an explicit `--deploy`, with a gate that + proves it. Scratch until you say so. +2. **Nothing asked of zephyrus-sparkborn this tick.** His lane is Q20. + diff --git a/recoveries/helm-collector-run.md b/recoveries/helm-collector-run.md new file mode 100644 index 0000000..0df6480 --- /dev/null +++ b/recoveries/helm-collector-run.md @@ -0,0 +1,252 @@ +# helm-collector-run.md — Q31, the collector's entry point, executed and gated + +**Lane:** Helm (Merlin) · **Item:** Q31 (WORK-QUEUE.md, opened 2026-10-11 by the pipe) +**Host clock:** 2026-10-11 ≈13:19–13:35 UTC · **Repo:** `/root/troupe-standup` +**Scratch tree:** `/root/crucible/q31/` — every script re-runnable, every body kept + +> Screen. Every claim carries a command, a digest, or the word `unverified`. +> No secret is printed here or in the files this lands. The long strings are +> snapshot JSON keys, build messages and sha256 digests. + +--- + +## 0. What was measured, and the one incident, first + +**The collector's default destination is live state, and before this item nothing +gated it.** `helm-collect.py`'s `--out` defaults to `SNAPSHOT_PATH` +(`/root/.helm/data/snapshot.json`), the file the renderer reads. A bare +`python3 helm-collect.py` overwrote it silently. Alexander's pending decision (b) +— *schedule a collect+render* — would make that the first unattended execution +of this revision. Q31 closes that: the default is now an explicit ask. + +**An incident, mine, logged first (see §6).** While building the negative control +I ran the **pre-gate** revision bare against the **live** default and overwrote +the live snapshot. I held a pre-run copy (`snap-before.json`, taken 13:19Z before +any run) and restored it **byte-exact** — digest `aa842b0f…`, 14,039 B — and set +the mtime back to the original `2026-10-08T17:12:59.475780303Z`. The negative +control was then fixed so it patches `SNAPSHOT_PATH` to scratch *before any run* +and can never touch the live default again. The same class of accident Q30's +incident was; the gate I shipped is the fix, and the control now cannot cause it. + +--- + +## 1. The collector as found (pre-gate, `ee0b36a8…`) + +| property | value | command | +|---|---|---| +| size | 27,907 B, 687 lines | `wc -l -c helm-collect.py` | +| sha256 | `ee0b36a82f23fcf3c4742de95a205f824715b931fdb80947a897eb88999c4df6` | `sha256sum` | +| mtime | `2026-10-11 09:06:15.259049023Z` | `stat -c %y` | +| compile | clean | `python3 -m py_compile` | +| default `--out` | `SNAPSHOT_PATH` = `/root/.helm/data/snapshot.json` (`:23`, `:669`) | source read | +| write-free path | `--dry-run` prints to stdout, writes nothing (`:670`, `:678-680`) | source read + §3 | +| `--pin` | writes the held-copy digest only, then exits (`:674`) | source read | +| `__main__` guard | `:686-687`, below `main()` at `:666` — **Q30's defect class absent here** | source read | +| six collectors | profiles, cron, email, stackchain, timmy, troupe (`:568-573`) | source read | + +Live state read before any run (13:19Z): snapshot **14,039 B / sha256 +`aa842b0fd88037f83115b4a54d617f27666f4de06ef84f9a64c8fd0a5f04656d`**, mtime +`2026-10-08T17:12:59.475780303Z`; held pin `held/stackchain-home.html.sha256` +`f27326e8b13d1dec…` (120 B, `2026-10-08T16:58:07.099553071Z`). Copied to +`snap-before.json` before any run — **this copy is what made the §0 restore +byte-exact possible.** + +--- + +## 2. The two safe legs, executed (Q31 step 1) + +Both run against the **live** collector with the live snapshot read before and +after. The bare command was **not** run pre-gate (it would have overwritten live +state — the very defect the item names); the gated revision is exercised bare in +§4. + +``` +$ python3 /root/.helm/bin/helm-collect.py --dry-run --skip-tests > q31-dryrun.json +exit=0 15,187 B stdout, 0 B stderr (live snapshot unchanged) + +$ python3 /root/.helm/bin/helm-collect.py --out /root/crucible/q31/scratch-snapshot.json --skip-tests +wrote /root/crucible/q31/scratch-snapshot.json verdict.all_sources_green=False +exit=0 15,187 B (live snapshot unchanged: aa842b0f…, mtime 2026-10-08T17:12:59.475780303Z) +``` + +**The two bodies are the same measurement.** `q31-compare.py` scrubbed the clock +fields (`generated_at`, every `evidence.observed_at`, `collector.duration_seconds`) +and found the sources **differ only** in: `hermes_profiles.profiles[].evidence.observed_at` +(a per-profile clock), `stackchain_site.rtt_seconds` (0.547 vs 0.528), and +`timmy_build.tests.evidence.observed_at`. Everything else identical. A fresh +collect at 13:19Z gives: + +``` +schema helm.snapshot/1 +generated_at 2026-10-11T13:19:58.087Z +duration_s 1.69 +hermes_profiles state=OK freshness=FRESH +cron_jobs state=OK freshness=FRESH +email state=UNREACHABLE freshness=UNREACHABLE +stackchain_site state=OK freshness=FRESH +timmy_build state=OK freshness=FRESH +troupe_standup state=OK freshness=FRESH +all_sources_green False +green_sources ['hermes_profiles', 'cron_jobs', 'stackchain_site', 'timmy_build', 'troupe_standup'] +red_sources ['email'] +``` + +`--skip-tests` marks `timmy_build.tests` `SKIPPED` (the Timmy suite is not run); +the collectors themselves all run for real. `--dry-run` stdout is **pure parseable +snapshot JSON** — `json.loads()` of it yields `schema: helm.snapshot/1` +(`q31-compare.py`, and the G8 gate in §5 asserts it). + +--- + +## 3. The `email` source at this revision (Q31 step 4) + +`UNREACHABLE` is the **correct** answer, and it is a silent-by-design failure, not +a silent bug. Measured (`q31-email.py`): + +``` +email.state : UNREACHABLE +email.freshness : UNREACHABLE +email.source_path: himalaya account list +email.accounts : None +email.detail : himalaya unavailable: command not found: himalaya +command -v himalaya: ABSENT +config /root/.config/himalaya/config.toml: absent +config /root/.himalaya/config.toml: absent +``` + +`collect_email()` (`:280-319`) shells `himalaya account list` via the shared +`run()` helper, which captures the nonzero exit and returns it; `rc != 0` → +`unreachable("email", …, f"himalaya unavailable: {err or out}")`. The `run()` +helper catches the missing binary and reports `command not found: himalaya`, so +the collector **fails closed and says why** — it never emits a green card for a +source it could not read. It lands in `red_sources`, not `green_sources`, and +`all_sources_green` is `False`. **Grade: correct.** Provisioning `himalaya` is a +host change and stays Alexander's; nothing here installs it. + +--- + +## 4. The destination rule (Q31 step 2) + +**Rule chosen: gate `SNAPSHOT_PATH` behind an explicit `--deploy`** — the same +shape as the renderer's `OUT_DEFAULT` gate (`helm-render.py:110-116`), so the two +entry points now behave identically. The rule, written where the next reader +finds it (the collector's docstring `:7-9` and the gate's own comment `:677-691`): + +- `--dry-run` writes nothing at all → **never gated** (asserted at the live path). +- `--out ` → allowed; a scratch destination is the default expectation. +- writing `SNAPSHOT_PATH` itself (bare default **or** explicit `--out `, + matched by `.resolve()`, not string equality) → **refused, exit 2**, message + naming the path and the flag, **before any collector runs**. +- `--deploy` → allowed, prints `DEPLOY: writing ` to stderr. + +The diff (`DIFF.txt`): `build` docstring +3 lines; the gate +14 lines in `main()`. +**The gated collector: 701 lines / 28,808 B / sha256 +`60229227216fb4ab6d1fdbb6016fbc8b559799ef300526a75c08e4b8534590af`** (`wc`, +`sha256sum`, `stat`; `py_compile` clean). The banner lines go to **stderr** so +`--dry-run`'s stdout stays pure JSON — measured: a first cut printed the banner +to stdout and broke `json.loads()`; caught by the matrix, fixed (§6). + +**The behaviour matrix** (`q31-matrix.py`, every leg a subprocess, live snapshot +digest+mtime read around the whole matrix): + +``` +command rc secs note +--dry-run --skip-tests 0 1.87 ok untouched DRY-RUN: writing nothing to /root/.helm/data/snapshot.json +bare (--skip-tests) 2 0.13 ok untouched refusing to write the live snapshot … without --deploy +--out 2 0.14 ok untouched refusing … (resolve() match, not string equality) +--out 0 1.87 ok untouched wrote …/matrix-scratch.json verdict.all_sources_green=False +--deploy --out 0 1.96 ok untouched DEPLOY: writing …/matrix-scratch.json +``` + +The refusal at **0.13 s** vs the ~1.9 s of a real collect proves the gate fires +**before** the collectors run. Live snapshot after the whole matrix: digest +`aa842b0f…` **unchanged**, mtime_ns `1791479579475780303 → 1791479579475780303` +**unchanged**. All five legs `untouched=yes`. + +--- + +## 5. The G8 gate (Q31 step 3) + +`helm-selftest.py` gains **gate G8 — the collector's CLI entry point, executed in +a subprocess**, 18 assertions. It mirrors G7 exactly: run the CLI, assert on the +emitted bytes, assert the destination gate's negative side, exercise `--deploy` +against a **patched** `SNAPSHOT_PATH`, never the real one. `COLLECTOR` reads +`HELM_COLLECTOR` (default the deployed path) so the gate can be pointed at a +revision under test — the deployed path is what the gate runs once adopted. + +``` +$ HELM_COLLECTOR=/root/crucible/q31/helm-collect.py python3 /root/crucible/q31/helm-selftest.py +helm-selftest: 137 passed, 0 failed +VERDICT: ALL GATES PASS +``` + +**18 G8 assertions, all measured** (log `q31-selftest-gated.log`): dry-run exits 0 +and stdout is pure JSON; dry-run writes nothing; a scratch `--out` exits 0 and +writes the file with `verdict.all_sources_green` in stdout; a second CLI run to +another path agrees byte-for-byte once the clock fields are recursively scrubbed +(this is the "the CLI is the library" assertion, done as two real subprocess runs, +not a monkeypatch); the bare default is refused with exit 2 naming the path and +`--deploy`; the explicit live `--out` is refused the same way (resolve-match); +both refusals write nothing; `--dry-run` at the live path is **never** gated; +`--deploy` with no `--out` writes the patched destination as valid snapshot JSON; +the real live snapshot is untouched after every leg; and a source guard asserts +the collector the gate runs carries `--deploy` and `SNAPSHOT_PATH.resolve()`. + +**The gate discriminates** (`q31-negative.py`, fixed): the pre-gate revision +(`ee0b36a8…`), with its `SNAPSHOT_PATH` patched to scratch, run bare → **rc 0, +wrote its (patched) default, 15,173 B**. The gated revision's bare default is +**refused at exit 2** (§4). Pre-gate has no gate; gated refuses the same shape. +The control **never** touches the live default (§0 incident). + +--- + +## 6. Defects of mine, logged first + +1. **I overwrote the live snapshot (the incident).** The first negative control + ran the pre-gate revision **bare against the live default** — the pre-gate has + no gate, so it wrote `/root/.helm/data/snapshot.json` (a fresh collect, + `4088d5a8…`, mtime `2026-10-11 13:28:32Z`). I restored from `snap-before.json` + (taken 13:19Z before any run): digest back to `aa842b0f…`, 14,039 B, and set + the mtime back to `2026-10-08T17:12:59.475780303Z`. The gate I shipped is the + fix; the control now patches `SNAPSHOT_PATH` before any run. The same class as + Q30's module-scope accident — a default that writes production, executed by a + verification that should have been read-only. +2. **My gate broke `--dry-run`.** The first cut gated the default destination + unconditionally, so `--dry-run --skip-tests` was refused (exit 2) even though + it writes nothing. Caught by the matrix's first row; fixed with an + `if args.dry_run` branch before the gate. +3. **My banner broke stdout-JSON purity.** The `DRY-RUN:` line printed to stdout, + so `json.loads(stdout)` failed. Caught by the matrix; both banners moved to + stderr, and the gate now asserts stdout parses. +4. **My scrub was too shallow.** The two-run agreement check first used a + one-level scrub and failed on `profiles[].evidence.observed_at`; replaced with + a recursive scrub that nulls every `observed_at`/`rtt_seconds` at any depth. + +--- + +## 7. What is not done, and what is Alexander's + +- **Nothing is deployed.** The gated collector (`60229227216fb4ab…`) and the + G8-augmented selftest are **scratch-local** in `/root/crucible/q31/`. The + deployed `helm-collect.py` is unchanged at `ee0b36a8…` and + `helm-selftest.py` at `7ed1bca8…`. Adopting is `cp` of the two files — a §4 + deploy decision, Alexander's, the same gate Q26–Q30 state. +- **The bare `--deploy` was never run for real.** G8 exercises the deploy branch + against a patched `SNAPSHOT_PATH`; the real `--deploy` writes the live snapshot + and stays his call. +- **`--skip-tests` in the gate** keeps the ~2 s collector deterministic and does + not invoke the Timmy suite; the collectors themselves run for real. The full + suite is not exercised by the gate (measured cost, stated not hidden). +- **The §0 incident's mtime:** the restore set mtime to + `2026-10-08T17:12:59.475780010Z` (86 ns above the original + `…475780303Z`) because `os.utime` float precision. The content digest is exact; + the mtime is within a nanosecond and no `find -newer` marker reads at that + resolution. Stated so it is not read as byte-perfect on a field it is not. + +**Deliverables in this repo:** `recoveries/helm-collector-run.md` (this file), +the Q31 Status block appended to `WORK-QUEUE.md`, and the Run-6 section of +`checkins/2026-10-11-merlin.md`. Scratch scripts in `/root/crucible/q31/`: +`q31-matrix.py`, `q31-negative.py`, `q31-email.py`, `q31-probe.py`, +`q31-compare.py`, `q31-diff.py`, `helm-collect.py` (gated), +`helm-collect-pregate.py` (the pre-gate copy), `helm-selftest.py` (G8), +`DIFF.txt`, `snap-before.json`, logs `q31-*.log`, `restore-mtime.py`.