Files
troupe-standup/recoveries/helm-g8-scrub.md
T

9.3 KiB
Raw Blame History

Q32 — the G8 two-run agreement check is deterministic

Screen. Every claim below carries a path, a digest, or a command with its output, measured this run (2026-10-11 ≈19:5x–20:2x UTC) from this host on the scratch tree /root/crucible/q31/. No secret is printed here or in the files this lands.

The defect

G8 asserted "two CLI runs agree byte-for-byte once the clock fields are scrubbed", comparing two real subprocess collects. The scrub covered observed_at, rtt_seconds, the top-level generated_at and collector.duration_seconds — and nothing else. helm-collect.py:545 stamps age_seconds = round(age, 1) on every source block from the run-end clock, so any run pair straddling a 0.1 s boundary in a source's age disagreed on that leaf alone. A gate whose red is a coin flip cannot be cited as evidence that an artifact is safe, and the Q31 adoption ask rests on exactly that citation.

Root cause, and it is wider than age_seconds

Before patching, a paired-run probe ran with an age_seconds-only patch: 7 of 8 pairs still red, on two further volatile fields the scrub also missed:

  • profiles[].last_active_at (helm-collect.py:208) — newest mtime across sessions/, cron/output/, logs/. Changes whenever a live session writes a file, i.e. constantly on this host.
  • profiles[].evidence.source_path (helm-collect.py:200, :212) — not a citation but a pointer: the path of that newest file. Volatile by construction wherever a live mtime reading exists. Every other evidence.source_path in the snapshot is a stable citation.
  • members[].last_checkin_at (helm-collect.py:493) — check-in file mtime, the same class (the checkins/ directory is written by other agents). It did not differ in this run's 8 pairs; it is the same class and is covered.

The fix — selftest only, the collector is untouched

  • SCRUB_KEYS = {observed_at, rtt_seconds, age_seconds, last_active_at, last_checkin_at}, declared. A gate check verifies every key appears as a literal in the collector source (with generated_at and duration_seconds handled explicitly at the top level), and a second check that the declared set is exactly the clock literals minus the two top-level keys.
  • The pointer rule: evidence.source_path is nulled inside any dict carrying last_active_at or last_checkin_at. A gate check proves the carve-out is measured, not asserted: 12 stable evidence.source_path citations survive the scrub (the 24 newest-file pointers are nulled).
  • Non-vacuity per key: each check name prints the fresh collect's live (non-None) value count, and the pre-Q32 scrub runs alongside as a control proving the old set kept every value the new one nulls. (email.age_seconds is legitimately None — UNREACHABLE source — so 5/6 is the honest count.)
  • Diff: helm-selftest-q32-before.py (sha256 1d58b823809aa76e…) → helm-selftest.py (sha256 9ce6a4e971525fdd4e49150fbeb49e584b7eb9cad293164f060a6f2c47739562), DIFF-Q32.txt, 115 lines added / 5 removed.

Results, measured

Paired runs — q32-paired.py, 8 pairs, two real subprocess collects per pair, both bodies scrubbed with the gate's own _scrub (extracted from helm-selftest.py by AST and exec'd, so the paired script cannot drift from the gate), compared leaf by leaf:

  • 8 pairs, 0 differing leaf paths under the gate's scrub.
  • Control, same 8 pairs under the pre-Q32 scrub: 12 differing leaf paths, 4 of 8 pairs red (sources.cron_jobs.age_seconds, sources.hermes_profiles.age_seconds, profiles[N].last_active_at, profiles[N].evidence.source_path on actively-written profiles). Log: q32-paired.txt.

Gate invocations — HELM_COLLECTOR=/root/crucible/q31/helm-collect.py python3 helm-selftest.py, 6 consecutive runs, each exit 0, helm-selftest: 150 passed, 0 failed, VERDICT: ALL GATES PASS (137 standing + 13 new Q32 assertions; G8 is now 31 checks). One log's digest per run:

run exit passed / failed log sha256 (first 16)
1 0 150 / 0 bfcc35cd92484acb
2 0 150 / 0 0a6272e3f6d88a96
3 0 150 / 0 bcf24e51049e661e
4 0 150 / 0 b9740dbd33f9f494
5 0 150 / 0 f69d56ef176b2eaf
6 0 150 / 0 4bf31912f396161d

The digests differ run to run by construction: the collector's stdout carries the digest of the file it wrote, and the written body carries the unscrubbed clock fields. Each run's digest is the evidence for that run.

New assertions, with their fresh-collect counts from run 1's log (q32-gate-run-1.log):

PASS  G8 two CLI runs agree byte-for-byte once the clock fields are scrubbed
PASS  G8 every clock-derived key the collector emits appears in its source
PASS  G8 the declared scrub set is exactly the clock literals minus the two top-level keys
PASS  G8 the scrub normalises age_seconds (fresh collect: 5/6 non-None)
PASS  G8 the pre-Q32 scrub kept age_seconds (the defect, as a control)
PASS  G8 the scrub normalises last_active_at (fresh collect: 20/20 non-None)
PASS  G8 the pre-Q32 scrub kept last_active_at (the defect, as a control)
PASS  G8 the scrub normalises last_checkin_at (fresh collect: 4/4 non-None)
PASS  G8 the pre-Q32 scrub kept last_checkin_at (the defect, as a control)
PASS  G8 the scrub normalises observed_at (fresh collect: 36/36 non-None)
PASS  G8 the scrub normalises rtt_seconds (fresh collect: 1/1 non-None)
PASS  G8 the newest-file pointer evidence.source_path is scrubbed (fresh collect: 24/24 non-None)
PASS  G8 the pre-Q32 scrub kept the newest-file pointer (the defect, as a control)
PASS  G8 stable evidence.source_path citations survive the scrub (12 paths)

Nothing touched

  • The gated collector /root/crucible/q31/helm-collect.py is unchanged at 8a477376f72c578b0633e07741b44c717482fe62b29a3cc3f8abb7a6696ea76c, verified before and after every run this tick. Q32 changes the selftest only.
  • The deployed /root/.helm/bin/helm-collect.py is unchanged at ee0b36a82f23fcf3… and /root/.helm/bin/helm-selftest.py at 7ed1bca8f9da624f… (the deployed gate is still the 119-pass one; nothing was deployed).
  • The live snapshot /root/.helm/data/snapshot.json is 14,039 B / sha256 aa842b0fd88037f83115b4a54d617f27666f4de06ef84f9a64c8fd0a5f04656d / mtime_ns 1791479579475780303 — byte- and mtime-identical before and after the paired runs and all six gate invocations (delta 0).
  • The live ledger index https://alexanderwhitestone.com/stackchain/blocks/ is unchanged: 200 / 34,488 B / sha256 4f764ef4dfb9c08b2654ff0783550d20ad1bb7a31cb0c32070974a5fe451557e.
  • blocks.json, decisions/ and inbox/ were not touched; nothing was executed; no another-party file was edited; no secret is printed in this record or in the files it lands.

Limits, stated not hidden

  • The enumeration is declared and source-verified, not proven complete. The gate fails closed on any new volatile field (a pair goes red and the leaf path is named in the failure), but nothing proves a priori that the seven clock literals are all the clock-derived keys the collector will ever emit.

  • The pointer rule is a scrub convention; the design smell stands. For a profile or a member, evidence.source_path is a mutable newest-file pointer where every other evidence.source_path is a stable citation. The clean fix is collector-side — point source_path at the profile directory (or the member's file) and name the newest-file path its own field. The collector is the artifact under adoption and Q32 deliberately does not change it.

  • The gate runs with --skip-tests (the standing Q31 caveat): the ~2 s collector is deterministic and does not invoke the Timmy suite.

  • Correction C5 (appended 2026-10-11, from the control plane's Q32 record-precision pass): the first bullet's clause "the leaf path is named in the failure" is wrong as measured. The gate fails closed — a patched collector carrying a new top-level clock field goes red at 149/1 — but the failure carries the static message two runs of the same collector disagree beyond the clocks, and the leaf name appears 0 times in that log. What names the leaf is the paired-run comparison (q32-paired.py prints the differing leaf paths), not the gate's failure message. The clause should read: fails closed; the leaf is named by the pair comparison, not by the gate. Measured by the control plane 2026-10-11 and reproduced from this lane's own paired output the same day.

  • Q33 closes the pointer-rule bullet above, collector-side (2026-10-11, scratch, adoption pending). The collector now emits the newest-file path as its own named field — evidence.latest_active_path on profiles, evidence.newest_checkin_file on members — and evidence.source_path is a stable citation everywhere (the profile directory; the check-ins directory). The Q32 pointer carve-out is deleted from the gate; the pointer is a declared scrub key. Full record: recoveries/helm-pointer-split.md.

Adoption

Q32 changes only the selftest. The Q31 adoption ask becomes: cp the collector 8a477376… (unchanged) plus the selftest 9ce6a4e9… (superseding 1d58b823…) into /root/.helm/bin/ — the gate that ships is the deterministic one. Deploying collect+render, and scheduling it, remain Alexander's under §4.

— Merlin, 2026-10-11