volt
9188dd37d0
The defect class WORK-QUEUE.md:8510 names in one sentence -- "a token quoted without its class ... a hash family swapped for its sibling, with no byte figure moving" -- is invisible to a reader (both families are hex) and cheap for a script. tools/citation_hash_class.py indexes the whole object store (868 blobs, 678 commits, 0 ambiguous prefixes) and resolves every token by prefix, so historically-truncated tokens resolve too. Measured at HEAD, 306 files, 9,296 hex tokens >=8 chars: BLOB_OID 584, CONTENT_SHA256 1,049, COMMIT_SHA 220, UNRESOLVED 7,443. A class word is BOUND to a token 1,013 times and contradicts the measurement 0 times. The exposure is a MISSING label, not a wrong one: 485 occurrences in 54 files where a hash sits directly after a bare byte figure. Acceptance: it flags :8467's two tokens (83acffe0, decd3350) as CLASSLESS_AFTER_SIZE with measured=CONTENT_SHA256 and prints each sibling -- 83acffe0 -> blob4f53c93a70/ 13,218 B, decd3350 -> blob4e3cca9cff/ 18,527 B -- reproducing :8485's erratum from the artifacts alone. :8485 itself flags nothing. Self-test 19 fixtures, 19 passed, 0 failed, rc 0. Four defects of my own found by running it are logged in checkins/2026-10-15-volt.md Block 2, including the 12-char floor that made the tool miss the 8-char tokens it exists for, and the schema bug that let --json write an empty findings array under --only mismatch.
373 lines
18 KiB
Python
373 lines
18 KiB
Python
#!/usr/bin/env python3
|
|
"""citation_hash_class.py -- resolve every hex token in a corpus to its hash family,
|
|
and flag the tokens that carry no stated class.
|
|
|
|
WHY THIS EXISTS
|
|
The troupe's most-recurring manual defect is "a token quoted without its class":
|
|
a git blob oid cited where a content sha256 belongs, or a sha256 cited where a
|
|
blob oid belongs. Both families are hex, so to the eye they are the same token.
|
|
It is logged at least three times on this repo:
|
|
WORK-QUEUE.md:8467 two `git cat-file -s` rows print the seed's CONTENT digests
|
|
(`83acffe0...`, `decd3350...`) where the git blobs belong
|
|
(`4f53c93a708d...`, `4e3cca9cffd2...`).
|
|
WORK-QUEUE.md:8497 the mirror: a block calls the blob oids "unsupported" and
|
|
proposes the content digests in their place.
|
|
WORK-QUEUE.md:8510 the lane's own summary: "a token quoted without its class
|
|
... a hash family swapped for its sibling, with no byte
|
|
figure moving".
|
|
This tool makes the class explicit by RESOLVING each token against the object
|
|
database instead of trusting the word beside it -- and, for every flag, prints
|
|
the SIBLING value, which is the disambiguation that defect cost a full round trip.
|
|
|
|
WHAT IT MEASURES (read-only; the only write path is --json)
|
|
Built once over the whole object store, so HISTORICAL tokens resolve too:
|
|
blob oid every blob ever (`git cat-file --batch-all-objects`)
|
|
content sha256 sha256 of that blob's bytes (same single streaming pass)
|
|
size byte size of that blob
|
|
path the first path the blob is reachable under (`rev-list --objects`)
|
|
commit sha `git rev-list --all`
|
|
Every token (>= --min-len hex chars) is matched BY PREFIX against the index and
|
|
classified BLOB_OID / CONTENT_SHA256 / COMMIT_SHA / AMBIGUOUS / UNRESOLVED.
|
|
A reverse map (content sha256 -> blob oids) is what yields the sibling.
|
|
|
|
VERDICT CLASSES, EACH WITH ITS UNIT
|
|
CLASS_MISMATCH a class word BOUND to the token names a family the
|
|
measurement contradicts. This is a defect claim.
|
|
CLASSLESS_AFTER_SIZE a token with no bound class word, directly after a bare
|
|
size figure ("18,527 B / `T`" or "13218 / `T`") -- the
|
|
row notation behind the :8467 defect. A RISK flag, not a
|
|
claim: both families are legitimate there, and that is
|
|
exactly why the swap goes unnoticed.
|
|
BOUND_UNRESOLVED a class word bound to a token this object store cannot
|
|
resolve (an external artifact's digest, typically).
|
|
|
|
THE BINDING RULE, AND WHY IT IS DIRECTIONAL
|
|
A class word is bound only when at most BIND_GAP separator characters and nothing
|
|
else stand between it and the token. A word binds the token it directly PRECEDES
|
|
("sha256 `T`"), which is the troupe's notation; it binds the token it FOLLOWS
|
|
only when no other token follows the word. Without that direction the rule reads
|
|
"blob `A`, 19,516 B, sha256 `B`" as labelling A with sha256 -- 896 such
|
|
false positives on this corpus at a 120-char window.
|
|
|
|
USAGE
|
|
python3 citation_hash_class.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
|
|
[--json OUT] [--only mismatch|classless|bound|flags|all] [--min-len N]
|
|
[--context] [--siblings] [--selftest]
|
|
"""
|
|
import argparse, collections, hashlib, json, re, subprocess, sys
|
|
|
|
MIN_LEN_DEFAULT = 8
|
|
PREFIX_LENS = (7, 8, 10, 12, 16, 20, 40, 64)
|
|
# SEP is a bare member list, never bracketed: it is wrapped exactly once, by the
|
|
# `[%s]` in the patterns below. Bracketing it here would double-wrap the class,
|
|
# close it at the first inner `]`, and silently break every binding.
|
|
SEP = r"`'\"()\[\]:,=\-\s"
|
|
BIND_GAP = 4
|
|
HEX_STR = re.compile(r'\b[0-9a-f]{%d,64}\b' % MIN_LEN_DEFAULT)
|
|
# the size-adjacency pattern: a standalone number (not the digits inside a word
|
|
# like `sha256`), optional 'B', then AT LEAST one separator, then the token.
|
|
AFTER_SIZE = re.compile(r'(?<![0-9a-zA-Z])\d[\d,]*(?:\s*B)?[`\'"()\[\]:,=\-/*_\s]{1,10}$')
|
|
|
|
WORDS = [
|
|
('content sha256', 'CONTENT_SHA256'), ('content digest', 'CONTENT_SHA256'),
|
|
('content sha', 'CONTENT_SHA256'), ('content-sha', 'CONTENT_SHA256'),
|
|
('sha256', 'CONTENT_SHA256'), ('sha-256', 'CONTENT_SHA256'),
|
|
('git hash-object', 'BLOB_OID'), ('git rev-parse', 'BLOB_OID'),
|
|
('blob oid', 'BLOB_OID'), ('blob-oid', 'BLOB_OID'), ('blob', 'BLOB_OID'),
|
|
('commit', 'COMMIT_SHA'),
|
|
]
|
|
WORDS.sort(key=lambda t: -len(t[0]))
|
|
PRE = [(re.compile(r'(?<![0-9a-z])%s[%s]{0,%d}$' % (re.escape(w), SEP, BIND_GAP), re.I), f)
|
|
for w, f in WORDS]
|
|
POST = [(re.compile(r'^[%s]{0,%d}(?<![0-9a-z])%s(?![0-9a-z])' % (SEP, BIND_GAP, re.escape(w)), re.I), f)
|
|
for w, f in WORDS]
|
|
|
|
|
|
def git(repo, *a):
|
|
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
|
|
|
|
|
|
def build_index(repo):
|
|
"""One pass over the whole object store: blob oid -> (content sha256, size)."""
|
|
meta, idx = {}, collections.defaultdict(list)
|
|
inv = collections.defaultdict(list) # content sha256 -> [blob oid]
|
|
|
|
# paths: one call, every reachable object with the path it is first seen under
|
|
paths = {}
|
|
for line in git(repo, "rev-list", "--all", "--objects").decode('utf-8', 'replace').split('\n'):
|
|
if not line:
|
|
continue
|
|
parts = line.split(' ', 1)
|
|
if len(parts) == 2 and parts[1]:
|
|
paths.setdefault(parts[0], parts[1])
|
|
|
|
p = subprocess.Popen(["git", "-C", repo, "cat-file", "--batch-all-objects", "--batch"],
|
|
stdout=subprocess.PIPE)
|
|
nblob = 0
|
|
while True:
|
|
head = p.stdout.readline()
|
|
if not head:
|
|
break
|
|
parts = head.split()
|
|
if len(parts) != 3:
|
|
continue
|
|
oid, otype, osize = parts[0].decode(), parts[1].decode(), int(parts[2])
|
|
body = p.stdout.read(osize) # BufferedReader.read: exact unless EOF
|
|
p.stdout.read(1) # the LF --batch appends after every body
|
|
if otype != 'blob':
|
|
continue
|
|
nblob += 1
|
|
csha = hashlib.sha256(body).hexdigest()
|
|
meta[oid] = (csha, osize)
|
|
inv[csha].append(oid)
|
|
for L in PREFIX_LENS:
|
|
if L <= 40:
|
|
idx[oid[:L]].append(("BLOB_OID", oid))
|
|
if L <= 64:
|
|
idx[csha[:L]].append(("CONTENT_SHA256", csha))
|
|
p.stdout.close()
|
|
p.wait()
|
|
|
|
commits = [c for c in git(repo, "rev-list", "--all").decode().split('\n') if c]
|
|
for c in commits:
|
|
for L in PREFIX_LENS:
|
|
if L <= 40:
|
|
idx[c[:L]].append(("COMMIT_SHA", c))
|
|
amb = {k for k, v in idx.items() if len(set(v)) > 1}
|
|
return {"blob_meta": meta, "index": idx, "inverse": inv, "paths": paths,
|
|
"commits": commits, "n_blobs": nblob, "ambiguous": amb}
|
|
|
|
|
|
def bound_family(text, s, e, window):
|
|
"""Directional binding -> (family, direction) or (None, None)."""
|
|
pre = text[max(0, s - window):s]
|
|
for rx, fam in PRE:
|
|
if rx.search(pre):
|
|
if fam == 'COMMIT_SHA' and (e - s) != 40:
|
|
continue # "at commit `2b834d4`" is prose, not a class assertion
|
|
return fam, 'precedes'
|
|
post = text[e:e + window]
|
|
for rx, fam in POST:
|
|
m = rx.match(post)
|
|
if not m:
|
|
continue
|
|
# Does this word label the NEXT token instead of this one? Strip any
|
|
# separators (including newlines -- the corpus wraps mid-row) and see
|
|
# whether a hex run of token length begins there.
|
|
rest = re.sub(r'^[^0-9a-f]+', '', text[e + m.end():e + m.end() + 32])
|
|
if HEX_STR.match(rest):
|
|
continue
|
|
if fam == 'COMMIT_SHA' and (e - s) != 40:
|
|
continue # "commit 2b834d4" on a 7-hex ref is prose, not a class
|
|
return fam, 'follows'
|
|
return None, None
|
|
|
|
|
|
def resolve(IDX, tok):
|
|
hits = IDX["index"].get(tok)
|
|
if not hits:
|
|
return 'UNRESOLVED', None, 0
|
|
fams = sorted({f for f, _ in hits})
|
|
if len(fams) == 1:
|
|
return fams[0], hits[0][1], len(hits)
|
|
return 'AMBIGUOUS', hits[0][1], len(hits)
|
|
|
|
|
|
def siblings(IDX, fam, full):
|
|
"""The other family's value for this token -- the disambiguation."""
|
|
if full is None:
|
|
return None
|
|
if fam == 'BLOB_OID':
|
|
csha, size = IDX["blob_meta"].get(full, (None, None))
|
|
return {"content_sha256": csha, "size": size, "path": IDX["paths"].get(full)}
|
|
if fam == 'CONTENT_SHA256':
|
|
blobs = IDX["inverse"].get(full, [])
|
|
out = [{"blob_oid": b, "size": IDX["blob_meta"].get(b, (None, None))[1],
|
|
"path": IDX["paths"].get(b)} for b in blobs]
|
|
return {"blob_oids": out}
|
|
return None
|
|
|
|
|
|
def scan(repo, corpus, rev, IDX, window, min_len):
|
|
rows, stats = [], collections.Counter()
|
|
hexre = re.compile(r'\b[0-9a-f]{%d,64}\b' % min_len)
|
|
for p in corpus:
|
|
text = git(repo, "show", f"{rev}:{p}").decode('utf-8', 'replace')
|
|
for m in hexre.finditer(text):
|
|
tok = m.group(0)
|
|
stats['occurrences'] += 1
|
|
fam, full, nhits = resolve(IDX, tok)
|
|
stats['measured_' + fam] += 1
|
|
bf, direction = bound_family(text, m.start(), m.end(), window)
|
|
if bf:
|
|
stats['bound'] += 1
|
|
after_size = bool(AFTER_SIZE.search(text[max(0, m.start() - 40):m.start()]))
|
|
if bf and fam == 'UNRESOLVED':
|
|
verdict = 'BOUND_UNRESOLVED'
|
|
elif bf and fam not in ('AMBIGUOUS', 'UNRESOLVED') and bf != fam:
|
|
verdict = 'CLASS_MISMATCH'
|
|
elif not bf and after_size and fam in ('BLOB_OID', 'CONTENT_SHA256'):
|
|
verdict = 'CLASSLESS_AFTER_SIZE'
|
|
else:
|
|
verdict = 'ok'
|
|
stats[verdict] += 1
|
|
rows.append({"file": p, "line": text.count('\n', 0, m.start()) + 1,
|
|
"token": tok, "len": len(tok), "measured": fam, "full": full,
|
|
"n_hits": nhits, "bound_family": bf, "bound_direction": direction,
|
|
"after_size": after_size, "verdict": verdict,
|
|
"context": text[max(0, m.start() - 80):m.end() + 80].replace('\n', ' ')})
|
|
return rows, stats
|
|
|
|
|
|
# ---------------------------------------------------------------- self-test
|
|
BINDING_FIXTURES = [
|
|
# (text, token, expected family, expected direction) or (.., None, None)
|
|
("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "precedes"),
|
|
("blob `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "precedes"),
|
|
("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef",
|
|
"CONTENT_SHA256", "precedes"),
|
|
# the first token is explicitly labelled a blob; the trailing `sha256` labels
|
|
# the SECOND token and must not leak backwards onto it
|
|
("blob `deadbeefdeadbeef`, 19,516 B, sha256 `bbbbbbbbbbbb`", "deadbeefdeadbeef",
|
|
"BLOB_OID", "precedes"),
|
|
("commit `deadbeefdeadbeefdeadbeefdeadbeefdeadbeef`", "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef",
|
|
"COMMIT_SHA", "precedes"),
|
|
("at commit `deadbeefdeadbeef`, drop", "deadbeefdeadbeef", None, None),
|
|
# STATED LIMITATION: a class word further than BIND_GAP=4 chars from the token
|
|
# is deliberately NOT bound. The rule under-reports; it never invents a class.
|
|
("git rev-parse HEAD:p -> `deadbeefdeadbeef`", "deadbeefdeadbeef", None, None),
|
|
# REGRESSION (WORK-QUEUE.md:432, replies/2026-10-04-to-zephyrus-29.md:52):
|
|
# `sha256` here labels the NEXT token; it must not bind backwards onto the blob.
|
|
("blob `277c8baa183f` -> `deadbeefdeadbeef`, `sha256 078c704a1627b809`",
|
|
"deadbeefdeadbeef", None, None),
|
|
("`deadbeefdeadbeef`, `sha256\n078c704a1627b809`", "deadbeefdeadbeef", None, None),
|
|
# REGRESSION (replies/2026-09-26-to-zephyrus-13.md:40): same, one separator short.
|
|
("(`head 7c4cf1b8e79f`, sha256 `4099fba9c11170e8`)", "7c4cf1b8e79f", None, None),
|
|
]
|
|
VERDICT_FIXTURES = [
|
|
# (text, token, family_of_token, expected verdict)
|
|
("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef",
|
|
"CONTENT_SHA256", "ok"),
|
|
("18,527 B / `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASSLESS_AFTER_SIZE"),
|
|
("1827 / `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "CLASSLESS_AFTER_SIZE"),
|
|
("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASS_MISMATCH"),
|
|
]
|
|
|
|
|
|
def selftest(repo):
|
|
n = bad = 0
|
|
for text, tok, want_f, want_d in BINDING_FIXTURES:
|
|
i = text.index(tok)
|
|
got_f, got_d = bound_family(text, i, i + len(tok), 40)
|
|
n += 1
|
|
ok = (got_f, got_d) == (want_f, want_d)
|
|
bad += 0 if ok else 1
|
|
print(f"{'PASS' if ok else 'FAIL'} bind {tok[:12]} in {text[:44]!r} "
|
|
f"-> {(got_f, got_d)} want {(want_f, want_d)}")
|
|
for text, tok, fam, want in VERDICT_FIXTURES:
|
|
i = text.index(tok)
|
|
bf, _ = bound_family(text, i, i + len(tok), 40)
|
|
after = bool(AFTER_SIZE.search(text[max(0, i - 40):i]))
|
|
if bf and fam == 'UNRESOLVED':
|
|
got = 'BOUND_UNRESOLVED'
|
|
elif bf and bf != fam:
|
|
got = 'CLASS_MISMATCH'
|
|
elif not bf and after and fam in ('BLOB_OID', 'CONTENT_SHA256'):
|
|
got = 'CLASSLESS_AFTER_SIZE'
|
|
else:
|
|
got = 'ok'
|
|
n += 1
|
|
ok = got == want
|
|
bad += 0 if ok else 1
|
|
print(f"{'PASS' if ok else 'FAIL'} verdict {text[:38]!r} -> {got} want {want}")
|
|
|
|
# repo-anchored fixtures: the :8467 defect's own tokens
|
|
IDX = build_index(repo)
|
|
for tok, want_fam, want_full in (
|
|
("83acffe0", "CONTENT_SHA256",
|
|
"83acffe0222d46e7435aed2e79dffca39e503554b2110e393fbd0cc12ad30265"),
|
|
("decd3350", "CONTENT_SHA256",
|
|
"decd3350585dd169d78995fd08f4d1120394c884b764198bd419a4bb1cf4a265"),
|
|
("4f53c93a708d", "BLOB_OID",
|
|
"4f53c93a708d95200e61f50d93a23c8b2119975b"),
|
|
("4e3cca9cffd2", "BLOB_OID",
|
|
"4e3cca9cffd2ace531030c9a1dd5bed25412ffab"),
|
|
("2b834d4", "COMMIT_SHA", None)):
|
|
fam, full, _ = resolve(IDX, tok)
|
|
n += 1
|
|
ok = fam == want_fam and (want_full is None or full == want_full)
|
|
bad += 0 if ok else 1
|
|
print(f"{'PASS' if ok else 'FAIL'} resolve {tok} -> {fam} {full} want {want_fam} {want_full}")
|
|
print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed")
|
|
return 0 if bad == 0 else 1
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser(description=__doc__,
|
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument('--repo', default='/root/troupe-standup')
|
|
ap.add_argument('--rev', default='HEAD')
|
|
ap.add_argument('--corpus', nargs='*')
|
|
ap.add_argument('--json')
|
|
ap.add_argument('--window', type=int, default=24)
|
|
ap.add_argument('--min-len', type=int, default=MIN_LEN_DEFAULT)
|
|
ap.add_argument('--only', choices=['mismatch', 'classless', 'bound', 'flags', 'all'],
|
|
default='flags')
|
|
ap.add_argument('--context', action='store_true')
|
|
ap.add_argument('--siblings', action='store_true')
|
|
ap.add_argument('--selftest', action='store_true')
|
|
a = ap.parse_args()
|
|
|
|
if a.selftest:
|
|
return selftest(a.repo)
|
|
|
|
corpus = a.corpus or [p for p in git(a.repo, "ls-tree", "-r", "--name-only", a.rev)
|
|
.decode('utf-8', 'replace').split('\n') if p]
|
|
IDX = build_index(a.repo)
|
|
rows, stats = scan(a.repo, corpus, a.rev, IDX, a.window, a.min_len)
|
|
|
|
print(f"corpus: {len(corpus)} files at {a.rev} in {a.repo}")
|
|
print(f"index: {IDX['n_blobs']} blobs, {len(IDX['commits'])} commits, "
|
|
f"{len(IDX['ambiguous'])} ambiguous prefixes (whole object store, so "
|
|
f"historical tokens resolve)")
|
|
print(f"hex tokens >={a.min_len} chars: {stats['occurrences']} occurrences "
|
|
f"(unit: occurrences)")
|
|
print("--- measured family (unit: occurrences) ---")
|
|
for k in ('BLOB_OID', 'CONTENT_SHA256', 'COMMIT_SHA', 'AMBIGUOUS', 'UNRESOLVED'):
|
|
if stats['measured_' + k]:
|
|
print(f" {k:16s} {stats['measured_' + k]}")
|
|
print(f"tokens carrying a BOUND class word (gap<={BIND_GAP}): {stats['bound']} occurrences")
|
|
for v in ('CLASS_MISMATCH', 'CLASSLESS_AFTER_SIZE', 'BOUND_UNRESOLVED'):
|
|
print(f"{v}: {stats[v]} occurrences")
|
|
|
|
sel = {'mismatch': lambda r: r['verdict'] == 'CLASS_MISMATCH',
|
|
'classless': lambda r: r['verdict'] == 'CLASSLESS_AFTER_SIZE',
|
|
'bound': lambda r: r['bound_family'],
|
|
'flags': lambda r: r['verdict'] != 'ok',
|
|
'all': lambda r: True}[a.only]
|
|
all_rows = rows # the JSON always carries the FULL census
|
|
rows = [r for r in rows if sel(r)]
|
|
print(f"--- {a.only}: {len(rows)} rows ---")
|
|
for r in rows[:400]:
|
|
print(f" {r['file']}:{r['line']} {r['token']} measured={r['measured']} "
|
|
f"bound={r['bound_family']}({r['bound_direction']}) verdict={r['verdict']}")
|
|
if a.context:
|
|
print(f" …{r['context']}…")
|
|
if a.siblings and r['full']:
|
|
s = siblings(IDX, r['measured'], r['full'])
|
|
if s:
|
|
print(f" sibling: {json.dumps(s)}")
|
|
if a.json:
|
|
with open(a.json, 'w') as fh:
|
|
json.dump({"rev": a.rev, "corpus": corpus, "window": a.window,
|
|
"min_len": a.min_len,
|
|
"index": {"blobs": IDX['n_blobs'], "commits": len(IDX['commits']),
|
|
"ambiguous_prefixes": len(IDX['ambiguous'])},
|
|
"stats": dict(stats), "findings": all_rows}, fh, indent=1)
|
|
print(f"wrote {a.json}")
|
|
return 0
|
|
|
|
|
|
if __name__ == '__main__':
|
|
sys.exit(main())
|