Q31: collector's entry point executed, destination gated, G8 gate (Merlin run 6)

2026-10-11 13:31:39 +00:00
1 parent e37a1ba01d
commit 300228b44c
@@ -3963,6 +3963,8 @@ Live state, re-read 11:31Z: `/var/www/…/helm/index.html` and `https://alexande
 9,865 B / `427b7b54…`, **5 × FRESH / 0 × STALE** — five false freshness claims still in production. The
 honest render for comparison sits at `/tmp/helm-honest-check.html` (11,806 B / `8b839754…`).
 **Status: DONE at Merlin's Run 6 (2026-10-11 ≈13:19–13:35Z) — deliverable `recoveries/helm-collector-run.md`; the collector's entry point is executed, its destination is gated, and G8 asserts it.** The item's four steps: (1) the two safe legs run — `--dry-run --skip-tests` exit 0 / 15,187 B pure-JSON stdout and `--out <scratch>` exit 0 / 15,187 B, both with the live snapshot unchanged (`aa842b0f…`); the two bodies are the same measurement (only `observed_at` clocks and one `rtt_seconds` differ, `q31-compare.py`). (2) **The destination rule: `SNAPSHOT_PATH` is gated behind an explicit `--deploy`** — the same shape as the renderer's `OUT_DEFAULT` gate, so both entry points now behave alike; `--dry-run` is never gated (it writes nothing), a scratch `--out` is the default expectation, and the live path (bare default *or* explicit `--out`, matched by `.resolve()`) is refused at exit 2 **before any collector runs** (0.13 s refusal vs ~1.9 s collect). Gated collector `60229227216fb4ab…` (701 lines / 28,808 B). (3) **Gate G8 — the collector's CLI in a subprocess, 18 assertions** → `helm-selftest: 137 passed, 0 failed`, VERDICT: ALL GATES PASS; the gate discriminates (pre-gate `ee0b36a8…` with `SNAPSHOT_PATH` patched to scratch writes its default bare; gated refuses). (4) **`email` UNREACHABLE is correct** — `himalaya` is ABSENT on PATH and both config paths are absent, so `collect_email()` fails closed with `himalaya unavailable: command not found: himalaya` and lands in `red_sources`, never `green`. **An incident, logged first in the deliverable §6:** the first negative control ran the pre-gate revision bare against the live default and overwrote the live snapshot; restored byte-exact from a pre-run `snap-before.json` (`aa842b0f…`, 14,039 B, mtime back to `2026-10-08T17:12:59.475780303Z`), and the control now patches `SNAPSHOT_PATH` before any run. **Nothing deployed** — the gated collector and G8 selftest are scratch-local in `/root/crucible/q31/`; the deployed `helm-collect.py` is unchanged at `ee0b36a8…` and `helm-selftest.py` at `7ed1bca8…` (still 119 passed). Adoption is a §4 deploy decision, Alexander's, the same gate Q26–Q30 state. Mercury's day-3 unit is Q31's first slice (the two safe legs); this run executed them first and Mercury's `/tmp/mercury-q31/` was empty at 13:29Z, so no duplication occurred — the unit is satisfied here and stays open for Mercury to land his own artifact.
 **Mercury lane — day-3 pin, 2026-10-11 12:04Z (control plane).** Mercury's day-2 check-in (`b50c5ec`,
 `checkins/2026-10-11-mercury.md`) re-ran the census and the digest and produced no new artifact — the second
 consecutive check-in in that shape. The census is closed evidence: 121 `<li id=`, **115 `documented` / 5
@@ -795,8 +795,91 @@ to disk:
 Re-verified after the push, from this host: live `/helm/` page still 200 / **9,865 B** /
 `427b7b54c69b18d2…` with mtime `2026-10-11 08:57:14.017837233Z` — nothing was deployed.
-`python3 /root/.helm/bin/helm-selftest.py` → **119 passed, 0 failed**, `VERDICT: ALL GATES PASS`.
+`python3 /root/.helm/bin/helm-selftest.py` → **119 passed, 0 failed**, VERDICT: ALL GATES PASS.
 The fixed Helm files are host-local (`/root/.helm/`, outside this repo): renderer 21,766 B /
 `5c67473d…`, selftest 33,647 B / `7ed1bca8…`, CONTRACT 17,657 B / `2c527635…`.
 — Merlin
 ---
 # Run 6 — Q31, the collector's entry point, executed and gated (Merlin, 2026-10-11 ≈13:19–13:35Z)
 > Screen. Every claim carries a path, a hash, a URL, or a command with its output, or the word
 > `unverified`. Nothing secret is printed here or in the files it lands.
 ## Did
 Claimed and finished **Q31 — the collector's entry point** (reply 13 §3.3 named it to me first as the
 Helm's author; Q20 is zephyrus-sparkborn's, Q9 blocked on the audio handover, Q13/Q19 gated on
 Alexander). Deliverable: `recoveries/helm-collector-run.md` (**13,592 B / sha256
 `580c4640ff98404fd2a9e985095448123ee7c185cd703ef675fb93248596dd18`**), scratch tree
 `/root/crucible/q31/` (all scripts re-runnable: `q31-matrix.py`, `q31-negative.py`, `q31-email.py`,
 `q31-probe.py`, `q31-compare.py`, `q31-diff.py`, the gated collector, the pre-gate copy, the G8
 selftest, `DIFF.txt`, logs).
 **An incident, mine, logged first.** The first negative control ran the **pre-gate** collector bare
 against the **live** default and overwrote `/root/.helm/data/snapshot.json` (a fresh collect,
 `4088d5a8…`, mtime `2026-10-11 13:28:32Z`). I had taken a pre-run copy (`snap-before.json`, 13:19Z,
 before any run) and restored it **byte-exact** — digest back to `aa842b0f…`, 14,039 B, mtime set back
 to `2026-10-08T17:12:59.475780303Z` (86 ns high, stated). The negative control now patches
 `SNAPSHOT_PATH` to scratch before any run and can never touch the live default. The same accident
 class as Q30's module-scope overwrite; the gate shipped is the fix.
 **Step 1 — the two safe legs, executed.** `--dry-run --skip-tests` → exit 0, 15,187 B pure-JSON
 stdout; `--out <scratch> --skip-tests` → exit 0, 15,187 B, `verdict.all_sources_green=False`. Live
 snapshot unchanged across both (`aa842b0f…`, mtime `2026-10-08T17:12:59.475780303Z`). The two bodies
 are the same measurement — only `evidence.observed_at` clocks and one `rtt_seconds` differ
 (`q31-compare.py`). Fresh collect: `email` UNREACHABLE, five sources OK+FRESH, `all_sources_green
 False`.
 **Step 4 — `email` UNREACHABLE is correct.** `himalaya` is ABSENT on PATH; both config paths absent;
 `collect_email()` fails closed with `himalaya unavailable: command not found: himalaya` and lands in
 `red_sources`, never `green`. A silent-by-design failure that says why, not a bug. Installing
 himalaya is a host change, Alexander's.
 **Step 2 — the destination rule.** `SNAPSHOT_PATH` gated behind an explicit `--deploy` — the same
 shape as the renderer's `OUT_DEFAULT` gate, so both entry points behave alike. `--dry-run` is never
 gated (writes nothing); a scratch `--out` is the default expectation; the live path (bare default
 **or** explicit `--out`, matched by `.resolve()`) is refused at exit 2 **before any collector runs**
 — measured 0.13 s refusal vs ~1.9 s collect. Gated collector `60229227216fb4ab…` (701 lines /
 28,808 B; `DIFF.txt` +14 lines in `main()` plus a docstring note). Behaviour matrix, five legs, all
 `untouched=yes` around the live snapshot's digest+mtime.
 **Step 3 — gate G8, 18 assertions.** `HELM_COLLECTOR=<gated> python3 helm-selftest.py` →
 **137 passed, 0 failed, VERDICT: ALL GATES PASS** (18 G8 + the standing 119). The gate runs the CLI
 in a subprocess, asserts the emitted bytes, the destination gate's negative side (bare default and
 explicit live `--out`, both refused, both writing nothing), `--dry-run` never gated at the live path,
 `--deploy` against a patched `SNAPSHOT_PATH`, and two CLI runs agreeing byte-for-byte once the clock
 fields are recursively scrubbed. **It discriminates:** the pre-gate `ee0b36a8…` (SNAPSHOT_PATH patched
 to scratch) writes its default bare; the gated refuses.
 **Nothing deployed.** Gated collector and G8 selftest are scratch-local; deployed `helm-collect.py`
 unchanged at `ee0b36a8…`, `helm-selftest.py` at `7ed1bca8…` (still 119 passed). Live snapshot
 restored and re-verified `aa842b0f…`. `decisions/` and `inbox/` untouched; nothing executed; no
 another-party file edited; no X page fetched.
 ## Blocked
 - Nothing blocked the run. **Unverified by construction:** the gate uses `--skip-tests` so the ~2 s
  collector is deterministic and does not invoke the Timmy suite — the full suite is not exercised by
  the gate (stated, not hidden); the real bare `--deploy` writes the live snapshot and was never run
  (§4, Alexander's); the restore's mtime is 86 ns high (content digest exact).
 - **Mercury's day-3 unit is Q31's first slice** (the two safe legs). This run executed them first;
  `/tmp/mercury-q31/` was empty at 13:29Z, so no duplication — the slice is satisfied here and stays
  open for Mercury to land his own artifact.
 ## Next
 - **Alexander's call, §4:** adopt `60229227216fb4ab` + the G8 selftest into `/root/.helm/bin/`
  (`cp`, two files) — the same publish gate Q26–Q30 state — then the deploy/schedule decision for
  the collector. The gate makes an unattended collect+render safe to schedule.
 - Q20 remains the top unclaimed item and is zephyrus-sparkborn's; I take nothing over it.
 ## Ask
 1. **Alexander — adopt the gated collector + G8 into the shared tree?** It is the difference between
   a bare run silently overwriting the live snapshot and an explicit `--deploy`, with a gate that
   proves it. Scratch until you say so.
 2. **Nothing asked of zephyrus-sparkborn this tick.** His lane is Q20.
@@ -0,0 +1,252 @@
 # helm-collector-run.md — Q31, the collector's entry point, executed and gated
 **Lane:** Helm (Merlin) · **Item:** Q31 (WORK-QUEUE.md, opened 2026-10-11 by the pipe)
 **Host clock:** 2026-10-11 ≈13:19–13:35 UTC · **Repo:** `/root/troupe-standup`
 **Scratch tree:** `/root/crucible/q31/` — every script re-runnable, every body kept
 > Screen. Every claim carries a command, a digest, or the word `unverified`.
 > No secret is printed here or in the files this lands. The long strings are
 > snapshot JSON keys, build messages and sha256 digests.
 ---
 ## 0. What was measured, and the one incident, first
 **The collector's default destination is live state, and before this item nothing
 gated it.** `helm-collect.py`'s `--out` defaults to `SNAPSHOT_PATH`
 (`/root/.helm/data/snapshot.json`), the file the renderer reads. A bare
 `python3 helm-collect.py` overwrote it silently. Alexander's pending decision (b)
 — *schedule a collect+render* — would make that the first unattended execution
 of this revision. Q31 closes that: the default is now an explicit ask.
 **An incident, mine, logged first (see §6).** While building the negative control
 I ran the **pre-gate** revision bare against the **live** default and overwrote
 the live snapshot. I held a pre-run copy (`snap-before.json`, taken 13:19Z before
 any run) and restored it **byte-exact** — digest `aa842b0f…`, 14,039 B — and set
 the mtime back to the original `2026-10-08T17:12:59.475780303Z`. The negative
 control was then fixed so it patches `SNAPSHOT_PATH` to scratch *before any run*
 and can never touch the live default again. The same class of accident Q30's
 incident was; the gate I shipped is the fix, and the control now cannot cause it.
 ---
 ## 1. The collector as found (pre-gate, `ee0b36a8…`)
 | property | value | command |
 |---|---|---|
 | size | 27,907 B, 687 lines | `wc -l -c helm-collect.py` |
 | sha256 | `ee0b36a82f23fcf3c4742de95a205f824715b931fdb80947a897eb88999c4df6` | `sha256sum` |
 | mtime | `2026-10-11 09:06:15.259049023Z` | `stat -c %y` |
 | compile | clean | `python3 -m py_compile` |
 | default `--out` | `SNAPSHOT_PATH` = `/root/.helm/data/snapshot.json` (`:23`, `:669`) | source read |
 | write-free path | `--dry-run` prints to stdout, writes nothing (`:670`, `:678-680`) | source read + §3 |
 | `--pin` | writes the held-copy digest only, then exits (`:674`) | source read |
 | `__main__` guard | `:686-687`, below `main()` at `:666` — **Q30's defect class absent here** | source read |
 | six collectors | profiles, cron, email, stackchain, timmy, troupe (`:568-573`) | source read |
 Live state read before any run (13:19Z): snapshot **14,039 B / sha256
 `aa842b0fd88037f83115b4a54d617f27666f4de06ef84f9a64c8fd0a5f04656d`**, mtime
 `2026-10-08T17:12:59.475780303Z`; held pin `held/stackchain-home.html.sha256`
 `f27326e8b13d1dec…` (120 B, `2026-10-08T16:58:07.099553071Z`). Copied to
 `snap-before.json` before any run — **this copy is what made the §0 restore
 byte-exact possible.**
 ---
 ## 2. The two safe legs, executed (Q31 step 1)
 Both run against the **live** collector with the live snapshot read before and
 after. The bare command was **not** run pre-gate (it would have overwritten live
 state — the very defect the item names); the gated revision is exercised bare in
 §4.
 ```
 $ python3 /root/.helm/bin/helm-collect.py --dry-run --skip-tests > q31-dryrun.json
 exit=0   15,187 B stdout, 0 B stderr        (live snapshot unchanged)
 $ python3 /root/.helm/bin/helm-collect.py --out /root/crucible/q31/scratch-snapshot.json --skip-tests
 wrote /root/crucible/q31/scratch-snapshot.json  verdict.all_sources_green=False
 exit=0   15,187 B        (live snapshot unchanged: aa842b0f…, mtime 2026-10-08T17:12:59.475780303Z)
 ```
 **The two bodies are the same measurement.** `q31-compare.py` scrubbed the clock
 fields (`generated_at`, every `evidence.observed_at`, `collector.duration_seconds`)
 and found the sources **differ only** in: `hermes_profiles.profiles[].evidence.observed_at`
 (a per-profile clock), `stackchain_site.rtt_seconds` (0.547 vs 0.528), and
 `timmy_build.tests.evidence.observed_at`. Everything else identical. A fresh
 collect at 13:19Z gives:
 ```
 schema          helm.snapshot/1
 generated_at    2026-10-11T13:19:58.087Z
 duration_s      1.69
 hermes_profiles    state=OK           freshness=FRESH
 cron_jobs          state=OK           freshness=FRESH
 email              state=UNREACHABLE  freshness=UNREACHABLE
 stackchain_site    state=OK           freshness=FRESH
 timmy_build        state=OK           freshness=FRESH
 troupe_standup     state=OK           freshness=FRESH
 all_sources_green False
 green_sources    ['hermes_profiles', 'cron_jobs', 'stackchain_site', 'timmy_build', 'troupe_standup']
 red_sources      ['email']
 ```
 `--skip-tests` marks `timmy_build.tests` `SKIPPED` (the Timmy suite is not run);
 the collectors themselves all run for real. `--dry-run` stdout is **pure parseable
 snapshot JSON** — `json.loads()` of it yields `schema: helm.snapshot/1`
 (`q31-compare.py`, and the G8 gate in §5 asserts it).
 ---
 ## 3. The `email` source at this revision (Q31 step 4)
 `UNREACHABLE` is the **correct** answer, and it is a silent-by-design failure, not
 a silent bug. Measured (`q31-email.py`):
 ```
 email.state      : UNREACHABLE
 email.freshness  : UNREACHABLE
 email.source_path: himalaya account list
 email.accounts   : None
 email.detail     : himalaya unavailable: command not found: himalaya
 command -v himalaya: ABSENT
 config /root/.config/himalaya/config.toml: absent
 config /root/.himalaya/config.toml: absent
 ```
 `collect_email()` (`:280-319`) shells `himalaya account list` via the shared
 `run()` helper, which captures the nonzero exit and returns it; `rc != 0` →
 `unreachable("email", …, f"himalaya unavailable: {err or out}")`. The `run()`
 helper catches the missing binary and reports `command not found: himalaya`, so
 the collector **fails closed and says why** — it never emits a green card for a
 source it could not read. It lands in `red_sources`, not `green_sources`, and
 `all_sources_green` is `False`. **Grade: correct.** Provisioning `himalaya` is a
 host change and stays Alexander's; nothing here installs it.
 ---
 ## 4. The destination rule (Q31 step 2)
 **Rule chosen: gate `SNAPSHOT_PATH` behind an explicit `--deploy`** — the same
 shape as the renderer's `OUT_DEFAULT` gate (`helm-render.py:110-116`), so the two
 entry points now behave identically. The rule, written where the next reader
 finds it (the collector's docstring `:7-9` and the gate's own comment `:677-691`):
 - `--dry-run` writes nothing at all → **never gated** (asserted at the live path).
 - `--out <scratch>` → allowed; a scratch destination is the default expectation.
 - writing `SNAPSHOT_PATH` itself (bare default **or** explicit `--out <live path>`,
  matched by `.resolve()`, not string equality) → **refused, exit 2**, message
  naming the path and the flag, **before any collector runs**.
 - `--deploy` → allowed, prints `DEPLOY: writing <path>` to stderr.
 The diff (`DIFF.txt`): `build` docstring +3 lines; the gate +14 lines in `main()`.
 **The gated collector: 701 lines / 28,808 B / sha256
 `60229227216fb4ab6d1fdbb6016fbc8b559799ef300526a75c08e4b8534590af`** (`wc`,
 `sha256sum`, `stat`; `py_compile` clean). The banner lines go to **stderr** so
 `--dry-run`'s stdout stays pure JSON — measured: a first cut printed the banner
 to stdout and broke `json.loads()`; caught by the matrix, fixed (§6).
 **The behaviour matrix** (`q31-matrix.py`, every leg a subprocess, live snapshot
 digest+mtime read around the whole matrix):
 ```
 command                       rc   secs  note
 --dry-run --skip-tests         0   1.87  ok  untouched  DRY-RUN: writing nothing to /root/.helm/data/snapshot.json
 bare (--skip-tests)            2   0.13  ok  untouched  refusing to write the live snapshot … without --deploy
 --out <live path>              2   0.14  ok  untouched  refusing … (resolve() match, not string equality)
 --out <scratch>                0   1.87  ok  untouched  wrote …/matrix-scratch.json  verdict.all_sources_green=False
 --deploy --out <scratch>       0   1.96  ok  untouched  DEPLOY: writing …/matrix-scratch.json
 ```
 The refusal at **0.13 s** vs the ~1.9 s of a real collect proves the gate fires
 **before** the collectors run. Live snapshot after the whole matrix: digest
 `aa842b0f…` **unchanged**, mtime_ns `1791479579475780303 → 1791479579475780303`
 **unchanged**. All five legs `untouched=yes`.
 ---
 ## 5. The G8 gate (Q31 step 3)
 `helm-selftest.py` gains **gate G8 — the collector's CLI entry point, executed in
 a subprocess**, 18 assertions. It mirrors G7 exactly: run the CLI, assert on the
 emitted bytes, assert the destination gate's negative side, exercise `--deploy`
 against a **patched** `SNAPSHOT_PATH`, never the real one. `COLLECTOR` reads
 `HELM_COLLECTOR` (default the deployed path) so the gate can be pointed at a
 revision under test — the deployed path is what the gate runs once adopted.
 ```
 $ HELM_COLLECTOR=/root/crucible/q31/helm-collect.py python3 /root/crucible/q31/helm-selftest.py
 helm-selftest: 137 passed, 0 failed
 VERDICT: ALL GATES PASS
 ```
 **18 G8 assertions, all measured** (log `q31-selftest-gated.log`): dry-run exits 0
 and stdout is pure JSON; dry-run writes nothing; a scratch `--out` exits 0 and
 writes the file with `verdict.all_sources_green` in stdout; a second CLI run to
 another path agrees byte-for-byte once the clock fields are recursively scrubbed
 (this is the "the CLI is the library" assertion, done as two real subprocess runs,
 not a monkeypatch); the bare default is refused with exit 2 naming the path and
 `--deploy`; the explicit live `--out` is refused the same way (resolve-match);
 both refusals write nothing; `--dry-run` at the live path is **never** gated;
 `--deploy` with no `--out` writes the patched destination as valid snapshot JSON;
 the real live snapshot is untouched after every leg; and a source guard asserts
 the collector the gate runs carries `--deploy` and `SNAPSHOT_PATH.resolve()`.
 **The gate discriminates** (`q31-negative.py`, fixed): the pre-gate revision
 (`ee0b36a8…`), with its `SNAPSHOT_PATH` patched to scratch, run bare → **rc 0,
 wrote its (patched) default, 15,173 B**. The gated revision's bare default is
 **refused at exit 2** (§4). Pre-gate has no gate; gated refuses the same shape.
 The control **never** touches the live default (§0 incident).
 ---
 ## 6. Defects of mine, logged first
 1. **I overwrote the live snapshot (the incident).** The first negative control
   ran the pre-gate revision **bare against the live default** — the pre-gate has
   no gate, so it wrote `/root/.helm/data/snapshot.json` (a fresh collect,
   `4088d5a8…`, mtime `2026-10-11 13:28:32Z`). I restored from `snap-before.json`
   (taken 13:19Z before any run): digest back to `aa842b0f…`, 14,039 B, and set
   the mtime back to `2026-10-08T17:12:59.475780303Z`. The gate I shipped is the
   fix; the control now patches `SNAPSHOT_PATH` before any run. The same class as
   Q30's module-scope accident — a default that writes production, executed by a
   verification that should have been read-only.
 2. **My gate broke `--dry-run`.** The first cut gated the default destination
   unconditionally, so `--dry-run --skip-tests` was refused (exit 2) even though
   it writes nothing. Caught by the matrix's first row; fixed with an
   `if args.dry_run` branch before the gate.
 3. **My banner broke stdout-JSON purity.** The `DRY-RUN:` line printed to stdout,
   so `json.loads(stdout)` failed. Caught by the matrix; both banners moved to
   stderr, and the gate now asserts stdout parses.
 4. **My scrub was too shallow.** The two-run agreement check first used a
   one-level scrub and failed on `profiles[].evidence.observed_at`; replaced with
   a recursive scrub that nulls every `observed_at`/`rtt_seconds` at any depth.
 ---
 ## 7. What is not done, and what is Alexander's
 - **Nothing is deployed.** The gated collector (`60229227216fb4ab…`) and the
  G8-augmented selftest are **scratch-local** in `/root/crucible/q31/`. The
  deployed `helm-collect.py` is unchanged at `ee0b36a8…` and
  `helm-selftest.py` at `7ed1bca8…`. Adopting is `cp` of the two files — a §4
  deploy decision, Alexander's, the same gate Q26–Q30 state.
 - **The bare `--deploy` was never run for real.** G8 exercises the deploy branch
  against a patched `SNAPSHOT_PATH`; the real `--deploy` writes the live snapshot
  and stays his call.
 - **`--skip-tests` in the gate** keeps the ~2 s collector deterministic and does
  not invoke the Timmy suite; the collectors themselves run for real. The full
  suite is not exercised by the gate (measured cost, stated not hidden).
 - **The §0 incident's mtime:** the restore set mtime to
  `2026-10-08T17:12:59.475780010Z` (86 ns above the original
  `…475780303Z`) because `os.utime` float precision. The content digest is exact;
  the mtime is within a nanosecond and no `find -newer` marker reads at that
  resolution. Stated so it is not read as byte-perfect on a field it is not.
 **Deliverables in this repo:** `recoveries/helm-collector-run.md` (this file),
 the Q31 Status block appended to `WORK-QUEUE.md`, and the Run-6 section of
 `checkins/2026-10-11-merlin.md`. Scratch scripts in `/root/crucible/q31/`:
 `q31-matrix.py`, `q31-negative.py`, `q31-email.py`, `q31-probe.py`,
 `q31-compare.py`, `q31-diff.py`, `helm-collect.py` (gated),
 `helm-collect-pregate.py` (the pre-gate copy), `helm-selftest.py` (G8),
 `DIFF.txt`, `snap-before.json`, logs `q31-*.log`, `restore-mtime.py`.