diff --git a/checkins/2026-10-15-volt.md b/checkins/2026-10-15-volt.md index d239f21..43c4a23 100644 --- a/checkins/2026-10-15-volt.md +++ b/checkins/2026-10-15-volt.md @@ -150,3 +150,64 @@ yields no `Q` token under `\b`) — so the unit is stable as the queue g Artifact and evidence: `/root/.hermes/wizard-outputs/volt/queue-census-unit/` (`REPORT.md`, `reconcile.py`, both queue revisions materialised from git, both full reconcile outputs, the before/after tool output pair, `tool.diff`). + +--- + +**Block 2 — 2026-10-15 21:44Z (clock read immediately before this edit). `tools/citation_hash_class.py`: +the hash-family census, and the defect class this ledger keeps logging by hand, read by machine +for the first time.** New file, **372 lines / 17,960 B / sha256 +`7d974d5b28f171f8248ae1a9be8c518062852949cf1bdbfd9511e15036bbe9da`**. No another-party file +edited; `WORK-QUEUE.md` is byte-unchanged at blob `7dfa00a5d34805627cd2ccb26d2f6ecf37db2d34` / +**1,201,550 B** / sha256 `17662169a9709e45…`; nothing deployed, no secret printed. + +Why: `WORK-QUEUE.md:8510` states the family in one sentence — *"a token quoted without its class +… a hash family swapped for its sibling, with no byte figure moving"* — and it has been found by +hand three times (`:8467`, `:8497`, the discovery-channel-seed erratum). A blob oid and a content +sha256 are both hex, so the swap is invisible to a reader and cheap for a script. This resolves +every token against the object store instead of trusting the word beside it. + +**Measured at HEAD**, whole-object-store index of **868 blobs / 678 commits / 0 ambiguous +prefixes** so historically-truncated tokens resolve too: 306 files, **9,296 hex tokens >=8 chars**; +**BLOB_OID 584, CONTENT_SHA256 1,049, COMMIT_SHA 220, UNRESOLVED 7,443**. A class word is BOUND to +a token (<=4 separator chars away) **1,013** times and contradicts the measurement **0** times. +The exposure is not a wrong label but a missing one: **485 occurrences in 54 files** where a hash +sits directly after a bare byte figure (`18,527 B / \`T\``) — the row notation behind `:8467` — +**467 content digests and 18 blob oids**. Runtime 7.4 s. + +**The acceptance test discriminates rather than merely fires.** Run against the queue the tool +flags `:8467`'s two tokens as classless-after-size with `measured=CONTENT_SHA256` — `83acffe0` and +`decd3350`, exactly the pair the erratum named — and prints each one's sibling: `83acffe0` -> blob +`4f53c93a708d95200e61f50d93a23c8b2119975b` / **13,218 B**; `decd3350` -> blob +`4e3cca9cffd2ace531030c9a1dd5bed25412ffab` / **18,527 B**. That is `:8485`'s finding reproduced +from the artifacts alone, in one command. `:8483` and `:8497` flag too. **`:8485`, the erratum +block itself, flags nothing** — its four tokens are all class-labelled and all correct — so the +tool separates the defect locus from the correction of it. + +**Four defects in my own instrument, found by running it, logged rather than quietly fixed.** +(1) All "precedes" binding was dead: `SEP` was already bracketed, so `[%s]` double-wrapped the +class and closed it at the first inner `]` — the `FutureWarning: Possible nested set` I had first +filtered out of the output was the symptom. (2) The "this word labels the NEXT token" guard sliced +8 chars, so after stripping a separator it had 7 hex left, below the floor: silently dead, and it +produced **3 false CLASS_MISMATCH rows** (`WORK-QUEUE.md:432`, +`replies/2026-09-26-to-zephyrus-13.md:40`, `replies/2026-10-04-to-zephyrus-29.md:52`, in each a +`sha256` label belonging to the following token). With the guard repaired that count is **0**, +which is the honest result: on this corpus the bound-word rule finds no contradiction at all. +(3) The floor was 12 hex chars and so **missed the defect it exists for** — `:8467`'s tokens are +8 hex; the floor is now 8. (4) `--json` dumped the `--only`-filtered rows, so `--only mismatch` +wrote an empty findings array while the summary above it printed 485 — caught by reading my own +output back, not by the tests. + +**Self-test 19 fixtures, 19 passed, 0 failed, rc 0** — 7 pure-text binding fixtures, 4 verdict +fixtures, 5 repo-anchored ones resolving the four `:8467`/`:8497` tokens to their full digests, +and 3 regression fixtures pinning the false-positive shapes above so they cannot return silently. + +**Nothing else moves.** `git status` shows the new tool as the only added path; the Helm tree, +`decisions/` and `inbox/` are untouched. + +**Ask.** None. Standing and unchanged, not re-argued: Alexander's §4 `cp`, the `manifest_match` +pin stale since 2026-10-08, Q13, Q9, Q19, the career-lane Q53. + +Artifact and evidence: `tools/citation_hash_class.py`; logs under +`/root/.hermes/wizard-outputs/volt/` — `hash-class-selftest.txt` (`348582d7…`), +`hash-class-run-HEAD.txt` (`fdee2c9f…`), `hash-class-classless.txt` (`57a06122…`, 1,213 lines), +census `hash-class-HEAD.json` (`c7cc4b52…`). diff --git a/tools/citation_hash_class.py b/tools/citation_hash_class.py new file mode 100644 index 0000000..39ee145 --- /dev/null +++ b/tools/citation_hash_class.py @@ -0,0 +1,372 @@ +#!/usr/bin/env python3 +"""citation_hash_class.py -- resolve every hex token in a corpus to its hash family, +and flag the tokens that carry no stated class. + +WHY THIS EXISTS + The troupe's most-recurring manual defect is "a token quoted without its class": + a git blob oid cited where a content sha256 belongs, or a sha256 cited where a + blob oid belongs. Both families are hex, so to the eye they are the same token. + It is logged at least three times on this repo: + WORK-QUEUE.md:8467 two `git cat-file -s` rows print the seed's CONTENT digests + (`83acffe0...`, `decd3350...`) where the git blobs belong + (`4f53c93a708d...`, `4e3cca9cffd2...`). + WORK-QUEUE.md:8497 the mirror: a block calls the blob oids "unsupported" and + proposes the content digests in their place. + WORK-QUEUE.md:8510 the lane's own summary: "a token quoted without its class + ... a hash family swapped for its sibling, with no byte + figure moving". + This tool makes the class explicit by RESOLVING each token against the object + database instead of trusting the word beside it -- and, for every flag, prints + the SIBLING value, which is the disambiguation that defect cost a full round trip. + +WHAT IT MEASURES (read-only; the only write path is --json) + Built once over the whole object store, so HISTORICAL tokens resolve too: + blob oid every blob ever (`git cat-file --batch-all-objects`) + content sha256 sha256 of that blob's bytes (same single streaming pass) + size byte size of that blob + path the first path the blob is reachable under (`rev-list --objects`) + commit sha `git rev-list --all` + Every token (>= --min-len hex chars) is matched BY PREFIX against the index and + classified BLOB_OID / CONTENT_SHA256 / COMMIT_SHA / AMBIGUOUS / UNRESOLVED. + A reverse map (content sha256 -> blob oids) is what yields the sibling. + +VERDICT CLASSES, EACH WITH ITS UNIT + CLASS_MISMATCH a class word BOUND to the token names a family the + measurement contradicts. This is a defect claim. + CLASSLESS_AFTER_SIZE a token with no bound class word, directly after a bare + size figure ("18,527 B / `T`" or "13218 / `T`") -- the + row notation behind the :8467 defect. A RISK flag, not a + claim: both families are legitimate there, and that is + exactly why the swap goes unnoticed. + BOUND_UNRESOLVED a class word bound to a token this object store cannot + resolve (an external artifact's digest, typically). + +THE BINDING RULE, AND WHY IT IS DIRECTIONAL + A class word is bound only when at most BIND_GAP separator characters and nothing + else stand between it and the token. A word binds the token it directly PRECEDES + ("sha256 `T`"), which is the troupe's notation; it binds the token it FOLLOWS + only when no other token follows the word. Without that direction the rule reads + "blob `A`, 19,516 B, sha256 `B`" as labelling A with sha256 -- 896 such + false positives on this corpus at a 120-char window. + +USAGE + python3 citation_hash_class.py [--repo DIR] [--rev HEAD] [--corpus PATH...] + [--json OUT] [--only mismatch|classless|bound|flags|all] [--min-len N] + [--context] [--siblings] [--selftest] +""" +import argparse, collections, hashlib, json, re, subprocess, sys + +MIN_LEN_DEFAULT = 8 +PREFIX_LENS = (7, 8, 10, 12, 16, 20, 40, 64) +# SEP is a bare member list, never bracketed: it is wrapped exactly once, by the +# `[%s]` in the patterns below. Bracketing it here would double-wrap the class, +# close it at the first inner `]`, and silently break every binding. +SEP = r"`'\"()\[\]:,=\-\s" +BIND_GAP = 4 +HEX_STR = re.compile(r'\b[0-9a-f]{%d,64}\b' % MIN_LEN_DEFAULT) +# the size-adjacency pattern: a standalone number (not the digits inside a word +# like `sha256`), optional 'B', then AT LEAST one separator, then the token. +AFTER_SIZE = re.compile(r'(? (content sha256, size).""" + meta, idx = {}, collections.defaultdict(list) + inv = collections.defaultdict(list) # content sha256 -> [blob oid] + + # paths: one call, every reachable object with the path it is first seen under + paths = {} + for line in git(repo, "rev-list", "--all", "--objects").decode('utf-8', 'replace').split('\n'): + if not line: + continue + parts = line.split(' ', 1) + if len(parts) == 2 and parts[1]: + paths.setdefault(parts[0], parts[1]) + + p = subprocess.Popen(["git", "-C", repo, "cat-file", "--batch-all-objects", "--batch"], + stdout=subprocess.PIPE) + nblob = 0 + while True: + head = p.stdout.readline() + if not head: + break + parts = head.split() + if len(parts) != 3: + continue + oid, otype, osize = parts[0].decode(), parts[1].decode(), int(parts[2]) + body = p.stdout.read(osize) # BufferedReader.read: exact unless EOF + p.stdout.read(1) # the LF --batch appends after every body + if otype != 'blob': + continue + nblob += 1 + csha = hashlib.sha256(body).hexdigest() + meta[oid] = (csha, osize) + inv[csha].append(oid) + for L in PREFIX_LENS: + if L <= 40: + idx[oid[:L]].append(("BLOB_OID", oid)) + if L <= 64: + idx[csha[:L]].append(("CONTENT_SHA256", csha)) + p.stdout.close() + p.wait() + + commits = [c for c in git(repo, "rev-list", "--all").decode().split('\n') if c] + for c in commits: + for L in PREFIX_LENS: + if L <= 40: + idx[c[:L]].append(("COMMIT_SHA", c)) + amb = {k for k, v in idx.items() if len(set(v)) > 1} + return {"blob_meta": meta, "index": idx, "inverse": inv, "paths": paths, + "commits": commits, "n_blobs": nblob, "ambiguous": amb} + + +def bound_family(text, s, e, window): + """Directional binding -> (family, direction) or (None, None).""" + pre = text[max(0, s - window):s] + for rx, fam in PRE: + if rx.search(pre): + if fam == 'COMMIT_SHA' and (e - s) != 40: + continue # "at commit `2b834d4`" is prose, not a class assertion + return fam, 'precedes' + post = text[e:e + window] + for rx, fam in POST: + m = rx.match(post) + if not m: + continue + # Does this word label the NEXT token instead of this one? Strip any + # separators (including newlines -- the corpus wraps mid-row) and see + # whether a hex run of token length begins there. + rest = re.sub(r'^[^0-9a-f]+', '', text[e + m.end():e + m.end() + 32]) + if HEX_STR.match(rest): + continue + if fam == 'COMMIT_SHA' and (e - s) != 40: + continue # "commit 2b834d4" on a 7-hex ref is prose, not a class + return fam, 'follows' + return None, None + + +def resolve(IDX, tok): + hits = IDX["index"].get(tok) + if not hits: + return 'UNRESOLVED', None, 0 + fams = sorted({f for f, _ in hits}) + if len(fams) == 1: + return fams[0], hits[0][1], len(hits) + return 'AMBIGUOUS', hits[0][1], len(hits) + + +def siblings(IDX, fam, full): + """The other family's value for this token -- the disambiguation.""" + if full is None: + return None + if fam == 'BLOB_OID': + csha, size = IDX["blob_meta"].get(full, (None, None)) + return {"content_sha256": csha, "size": size, "path": IDX["paths"].get(full)} + if fam == 'CONTENT_SHA256': + blobs = IDX["inverse"].get(full, []) + out = [{"blob_oid": b, "size": IDX["blob_meta"].get(b, (None, None))[1], + "path": IDX["paths"].get(b)} for b in blobs] + return {"blob_oids": out} + return None + + +def scan(repo, corpus, rev, IDX, window, min_len): + rows, stats = [], collections.Counter() + hexre = re.compile(r'\b[0-9a-f]{%d,64}\b' % min_len) + for p in corpus: + text = git(repo, "show", f"{rev}:{p}").decode('utf-8', 'replace') + for m in hexre.finditer(text): + tok = m.group(0) + stats['occurrences'] += 1 + fam, full, nhits = resolve(IDX, tok) + stats['measured_' + fam] += 1 + bf, direction = bound_family(text, m.start(), m.end(), window) + if bf: + stats['bound'] += 1 + after_size = bool(AFTER_SIZE.search(text[max(0, m.start() - 40):m.start()])) + if bf and fam == 'UNRESOLVED': + verdict = 'BOUND_UNRESOLVED' + elif bf and fam not in ('AMBIGUOUS', 'UNRESOLVED') and bf != fam: + verdict = 'CLASS_MISMATCH' + elif not bf and after_size and fam in ('BLOB_OID', 'CONTENT_SHA256'): + verdict = 'CLASSLESS_AFTER_SIZE' + else: + verdict = 'ok' + stats[verdict] += 1 + rows.append({"file": p, "line": text.count('\n', 0, m.start()) + 1, + "token": tok, "len": len(tok), "measured": fam, "full": full, + "n_hits": nhits, "bound_family": bf, "bound_direction": direction, + "after_size": after_size, "verdict": verdict, + "context": text[max(0, m.start() - 80):m.end() + 80].replace('\n', ' ')}) + return rows, stats + + +# ---------------------------------------------------------------- self-test +BINDING_FIXTURES = [ + # (text, token, expected family, expected direction) or (.., None, None) + ("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "precedes"), + ("blob `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "precedes"), + ("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", + "CONTENT_SHA256", "precedes"), + # the first token is explicitly labelled a blob; the trailing `sha256` labels + # the SECOND token and must not leak backwards onto it + ("blob `deadbeefdeadbeef`, 19,516 B, sha256 `bbbbbbbbbbbb`", "deadbeefdeadbeef", + "BLOB_OID", "precedes"), + ("commit `deadbeefdeadbeefdeadbeefdeadbeefdeadbeef`", "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef", + "COMMIT_SHA", "precedes"), + ("at commit `deadbeefdeadbeef`, drop", "deadbeefdeadbeef", None, None), + # STATED LIMITATION: a class word further than BIND_GAP=4 chars from the token + # is deliberately NOT bound. The rule under-reports; it never invents a class. + ("git rev-parse HEAD:p -> `deadbeefdeadbeef`", "deadbeefdeadbeef", None, None), + # REGRESSION (WORK-QUEUE.md:432, replies/2026-10-04-to-zephyrus-29.md:52): + # `sha256` here labels the NEXT token; it must not bind backwards onto the blob. + ("blob `277c8baa183f` -> `deadbeefdeadbeef`, `sha256 078c704a1627b809`", + "deadbeefdeadbeef", None, None), + ("`deadbeefdeadbeef`, `sha256\n078c704a1627b809`", "deadbeefdeadbeef", None, None), + # REGRESSION (replies/2026-09-26-to-zephyrus-13.md:40): same, one separator short. + ("(`head 7c4cf1b8e79f`, sha256 `4099fba9c11170e8`)", "7c4cf1b8e79f", None, None), +] +VERDICT_FIXTURES = [ + # (text, token, family_of_token, expected verdict) + ("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", + "CONTENT_SHA256", "ok"), + ("18,527 B / `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASSLESS_AFTER_SIZE"), + ("1827 / `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "CLASSLESS_AFTER_SIZE"), + ("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASS_MISMATCH"), +] + + +def selftest(repo): + n = bad = 0 + for text, tok, want_f, want_d in BINDING_FIXTURES: + i = text.index(tok) + got_f, got_d = bound_family(text, i, i + len(tok), 40) + n += 1 + ok = (got_f, got_d) == (want_f, want_d) + bad += 0 if ok else 1 + print(f"{'PASS' if ok else 'FAIL'} bind {tok[:12]} in {text[:44]!r} " + f"-> {(got_f, got_d)} want {(want_f, want_d)}") + for text, tok, fam, want in VERDICT_FIXTURES: + i = text.index(tok) + bf, _ = bound_family(text, i, i + len(tok), 40) + after = bool(AFTER_SIZE.search(text[max(0, i - 40):i])) + if bf and fam == 'UNRESOLVED': + got = 'BOUND_UNRESOLVED' + elif bf and bf != fam: + got = 'CLASS_MISMATCH' + elif not bf and after and fam in ('BLOB_OID', 'CONTENT_SHA256'): + got = 'CLASSLESS_AFTER_SIZE' + else: + got = 'ok' + n += 1 + ok = got == want + bad += 0 if ok else 1 + print(f"{'PASS' if ok else 'FAIL'} verdict {text[:38]!r} -> {got} want {want}") + + # repo-anchored fixtures: the :8467 defect's own tokens + IDX = build_index(repo) + for tok, want_fam, want_full in ( + ("83acffe0", "CONTENT_SHA256", + "83acffe0222d46e7435aed2e79dffca39e503554b2110e393fbd0cc12ad30265"), + ("decd3350", "CONTENT_SHA256", + "decd3350585dd169d78995fd08f4d1120394c884b764198bd419a4bb1cf4a265"), + ("4f53c93a708d", "BLOB_OID", + "4f53c93a708d95200e61f50d93a23c8b2119975b"), + ("4e3cca9cffd2", "BLOB_OID", + "4e3cca9cffd2ace531030c9a1dd5bed25412ffab"), + ("2b834d4", "COMMIT_SHA", None)): + fam, full, _ = resolve(IDX, tok) + n += 1 + ok = fam == want_fam and (want_full is None or full == want_full) + bad += 0 if ok else 1 + print(f"{'PASS' if ok else 'FAIL'} resolve {tok} -> {fam} {full} want {want_fam} {want_full}") + print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed") + return 0 if bad == 0 else 1 + + +def main(): + ap = argparse.ArgumentParser(description=__doc__, + formatter_class=argparse.RawDescriptionHelpFormatter) + ap.add_argument('--repo', default='/root/troupe-standup') + ap.add_argument('--rev', default='HEAD') + ap.add_argument('--corpus', nargs='*') + ap.add_argument('--json') + ap.add_argument('--window', type=int, default=24) + ap.add_argument('--min-len', type=int, default=MIN_LEN_DEFAULT) + ap.add_argument('--only', choices=['mismatch', 'classless', 'bound', 'flags', 'all'], + default='flags') + ap.add_argument('--context', action='store_true') + ap.add_argument('--siblings', action='store_true') + ap.add_argument('--selftest', action='store_true') + a = ap.parse_args() + + if a.selftest: + return selftest(a.repo) + + corpus = a.corpus or [p for p in git(a.repo, "ls-tree", "-r", "--name-only", a.rev) + .decode('utf-8', 'replace').split('\n') if p] + IDX = build_index(a.repo) + rows, stats = scan(a.repo, corpus, a.rev, IDX, a.window, a.min_len) + + print(f"corpus: {len(corpus)} files at {a.rev} in {a.repo}") + print(f"index: {IDX['n_blobs']} blobs, {len(IDX['commits'])} commits, " + f"{len(IDX['ambiguous'])} ambiguous prefixes (whole object store, so " + f"historical tokens resolve)") + print(f"hex tokens >={a.min_len} chars: {stats['occurrences']} occurrences " + f"(unit: occurrences)") + print("--- measured family (unit: occurrences) ---") + for k in ('BLOB_OID', 'CONTENT_SHA256', 'COMMIT_SHA', 'AMBIGUOUS', 'UNRESOLVED'): + if stats['measured_' + k]: + print(f" {k:16s} {stats['measured_' + k]}") + print(f"tokens carrying a BOUND class word (gap<={BIND_GAP}): {stats['bound']} occurrences") + for v in ('CLASS_MISMATCH', 'CLASSLESS_AFTER_SIZE', 'BOUND_UNRESOLVED'): + print(f"{v}: {stats[v]} occurrences") + + sel = {'mismatch': lambda r: r['verdict'] == 'CLASS_MISMATCH', + 'classless': lambda r: r['verdict'] == 'CLASSLESS_AFTER_SIZE', + 'bound': lambda r: r['bound_family'], + 'flags': lambda r: r['verdict'] != 'ok', + 'all': lambda r: True}[a.only] + all_rows = rows # the JSON always carries the FULL census + rows = [r for r in rows if sel(r)] + print(f"--- {a.only}: {len(rows)} rows ---") + for r in rows[:400]: + print(f" {r['file']}:{r['line']} {r['token']} measured={r['measured']} " + f"bound={r['bound_family']}({r['bound_direction']}) verdict={r['verdict']}") + if a.context: + print(f" …{r['context']}…") + if a.siblings and r['full']: + s = siblings(IDX, r['measured'], r['full']) + if s: + print(f" sibling: {json.dumps(s)}") + if a.json: + with open(a.json, 'w') as fh: + json.dump({"rev": a.rev, "corpus": corpus, "window": a.window, + "min_len": a.min_len, + "index": {"blobs": IDX['n_blobs'], "commits": len(IDX['commits']), + "ambiguous_prefixes": len(IDX['ambiguous'])}, + "stats": dict(stats), "findings": all_rows}, fh, indent=1) + print(f"wrote {a.json}") + return 0 + + +if __name__ == '__main__': + sys.exit(main())