tools (volt): the hash-family census -- resolve every hex token to blob oid / content sha256, and flag the ones quoted without a class
The defect class WORK-QUEUE.md:8510 names in one sentence -- "a token quoted without its class ... a hash family swapped for its sibling, with no byte figure moving" -- is invisible to a reader (both families are hex) and cheap for a script. tools/citation_hash_class.py indexes the whole object store (868 blobs, 678 commits, 0 ambiguous prefixes) and resolves every token by prefix, so historically-truncated tokens resolve too. Measured at HEAD, 306 files, 9,296 hex tokens >=8 chars: BLOB_OID 584, CONTENT_SHA256 1,049, COMMIT_SHA 220, UNRESOLVED 7,443. A class word is BOUND to a token 1,013 times and contradicts the measurement 0 times. The exposure is a MISSING label, not a wrong one: 485 occurrences in 54 files where a hash sits directly after a bare byte figure. Acceptance: it flags :8467's two tokens (83acffe0, decd3350) as CLASSLESS_AFTER_SIZE with measured=CONTENT_SHA256 and prints each sibling -- 83acffe0 -> blob4f53c93a70/ 13,218 B, decd3350 -> blob4e3cca9cff/ 18,527 B -- reproducing :8485's erratum from the artifacts alone. :8485 itself flags nothing. Self-test 19 fixtures, 19 passed, 0 failed, rc 0. Four defects of my own found by running it are logged in checkins/2026-10-15-volt.md Block 2, including the 12-char floor that made the tool miss the 8-char tokens it exists for, and the schema bug that let --json write an empty findings array under --only mismatch.
This commit is contained in:
1 parent
81bcd7966c
commit
9188dd37d0
2 files changed
+433
No files matched your search
@@ -150,3 +150,64 @@ yields no `Q<digits>` token under `\b`) — so the unit is stable as the queue g
|
||||
Artifact and evidence: `/root/.hermes/wizard-outputs/volt/queue-census-unit/` (`REPORT.md`,
|
||||
`reconcile.py`, both queue revisions materialised from git, both full reconcile outputs, the
|
||||
before/after tool output pair, `tool.diff`).
|
||||
|
||||
---
|
||||
|
||||
**Block 2 — 2026-10-15 21:44Z (clock read immediately before this edit). `tools/citation_hash_class.py`:
|
||||
the hash-family census, and the defect class this ledger keeps logging by hand, read by machine
|
||||
for the first time.** New file, **372 lines / 17,960 B / sha256
|
||||
`7d974d5b28f171f8248ae1a9be8c518062852949cf1bdbfd9511e15036bbe9da`**. No another-party file
|
||||
edited; `WORK-QUEUE.md` is byte-unchanged at blob `7dfa00a5d34805627cd2ccb26d2f6ecf37db2d34` /
|
||||
**1,201,550 B** / sha256 `17662169a9709e45…`; nothing deployed, no secret printed.
|
||||
|
||||
Why: `WORK-QUEUE.md:8510` states the family in one sentence — *"a token quoted without its class
|
||||
… a hash family swapped for its sibling, with no byte figure moving"* — and it has been found by
|
||||
hand three times (`:8467`, `:8497`, the discovery-channel-seed erratum). A blob oid and a content
|
||||
sha256 are both hex, so the swap is invisible to a reader and cheap for a script. This resolves
|
||||
every token against the object store instead of trusting the word beside it.
|
||||
|
||||
**Measured at HEAD**, whole-object-store index of **868 blobs / 678 commits / 0 ambiguous
|
||||
prefixes** so historically-truncated tokens resolve too: 306 files, **9,296 hex tokens >=8 chars**;
|
||||
**BLOB_OID 584, CONTENT_SHA256 1,049, COMMIT_SHA 220, UNRESOLVED 7,443**. A class word is BOUND to
|
||||
a token (<=4 separator chars away) **1,013** times and contradicts the measurement **0** times.
|
||||
The exposure is not a wrong label but a missing one: **485 occurrences in 54 files** where a hash
|
||||
sits directly after a bare byte figure (`18,527 B / \`T\``) — the row notation behind `:8467` —
|
||||
**467 content digests and 18 blob oids**. Runtime 7.4 s.
|
||||
|
||||
**The acceptance test discriminates rather than merely fires.** Run against the queue the tool
|
||||
flags `:8467`'s two tokens as classless-after-size with `measured=CONTENT_SHA256` — `83acffe0` and
|
||||
`decd3350`, exactly the pair the erratum named — and prints each one's sibling: `83acffe0` -> blob
|
||||
`4f53c93a708d95200e61f50d93a23c8b2119975b` / **13,218 B**; `decd3350` -> blob
|
||||
`4e3cca9cffd2ace531030c9a1dd5bed25412ffab` / **18,527 B**. That is `:8485`'s finding reproduced
|
||||
from the artifacts alone, in one command. `:8483` and `:8497` flag too. **`:8485`, the erratum
|
||||
block itself, flags nothing** — its four tokens are all class-labelled and all correct — so the
|
||||
tool separates the defect locus from the correction of it.
|
||||
|
||||
**Four defects in my own instrument, found by running it, logged rather than quietly fixed.**
|
||||
(1) All "precedes" binding was dead: `SEP` was already bracketed, so `[%s]` double-wrapped the
|
||||
class and closed it at the first inner `]` — the `FutureWarning: Possible nested set` I had first
|
||||
filtered out of the output was the symptom. (2) The "this word labels the NEXT token" guard sliced
|
||||
8 chars, so after stripping a separator it had 7 hex left, below the floor: silently dead, and it
|
||||
produced **3 false CLASS_MISMATCH rows** (`WORK-QUEUE.md:432`,
|
||||
`replies/2026-09-26-to-zephyrus-13.md:40`, `replies/2026-10-04-to-zephyrus-29.md:52`, in each a
|
||||
`sha256` label belonging to the following token). With the guard repaired that count is **0**,
|
||||
which is the honest result: on this corpus the bound-word rule finds no contradiction at all.
|
||||
(3) The floor was 12 hex chars and so **missed the defect it exists for** — `:8467`'s tokens are
|
||||
8 hex; the floor is now 8. (4) `--json` dumped the `--only`-filtered rows, so `--only mismatch`
|
||||
wrote an empty findings array while the summary above it printed 485 — caught by reading my own
|
||||
output back, not by the tests.
|
||||
|
||||
**Self-test 19 fixtures, 19 passed, 0 failed, rc 0** — 7 pure-text binding fixtures, 4 verdict
|
||||
fixtures, 5 repo-anchored ones resolving the four `:8467`/`:8497` tokens to their full digests,
|
||||
and 3 regression fixtures pinning the false-positive shapes above so they cannot return silently.
|
||||
|
||||
**Nothing else moves.** `git status` shows the new tool as the only added path; the Helm tree,
|
||||
`decisions/` and `inbox/` are untouched.
|
||||
|
||||
**Ask.** None. Standing and unchanged, not re-argued: Alexander's §4 `cp`, the `manifest_match`
|
||||
pin stale since 2026-10-08, Q13, Q9, Q19, the career-lane Q53.
|
||||
|
||||
Artifact and evidence: `tools/citation_hash_class.py`; logs under
|
||||
`/root/.hermes/wizard-outputs/volt/` — `hash-class-selftest.txt` (`348582d7…`),
|
||||
`hash-class-run-HEAD.txt` (`fdee2c9f…`), `hash-class-classless.txt` (`57a06122…`, 1,213 lines),
|
||||
census `hash-class-HEAD.json` (`c7cc4b52…`).
|
||||
@@ -0,0 +1,372 @@
|
||||
#!/usr/bin/env python3
|
||||
"""citation_hash_class.py -- resolve every hex token in a corpus to its hash family,
|
||||
and flag the tokens that carry no stated class.
|
||||
|
||||
WHY THIS EXISTS
|
||||
The troupe's most-recurring manual defect is "a token quoted without its class":
|
||||
a git blob oid cited where a content sha256 belongs, or a sha256 cited where a
|
||||
blob oid belongs. Both families are hex, so to the eye they are the same token.
|
||||
It is logged at least three times on this repo:
|
||||
WORK-QUEUE.md:8467 two `git cat-file -s` rows print the seed's CONTENT digests
|
||||
(`83acffe0...`, `decd3350...`) where the git blobs belong
|
||||
(`4f53c93a708d...`, `4e3cca9cffd2...`).
|
||||
WORK-QUEUE.md:8497 the mirror: a block calls the blob oids "unsupported" and
|
||||
proposes the content digests in their place.
|
||||
WORK-QUEUE.md:8510 the lane's own summary: "a token quoted without its class
|
||||
... a hash family swapped for its sibling, with no byte
|
||||
figure moving".
|
||||
This tool makes the class explicit by RESOLVING each token against the object
|
||||
database instead of trusting the word beside it -- and, for every flag, prints
|
||||
the SIBLING value, which is the disambiguation that defect cost a full round trip.
|
||||
|
||||
WHAT IT MEASURES (read-only; the only write path is --json)
|
||||
Built once over the whole object store, so HISTORICAL tokens resolve too:
|
||||
blob oid every blob ever (`git cat-file --batch-all-objects`)
|
||||
content sha256 sha256 of that blob's bytes (same single streaming pass)
|
||||
size byte size of that blob
|
||||
path the first path the blob is reachable under (`rev-list --objects`)
|
||||
commit sha `git rev-list --all`
|
||||
Every token (>= --min-len hex chars) is matched BY PREFIX against the index and
|
||||
classified BLOB_OID / CONTENT_SHA256 / COMMIT_SHA / AMBIGUOUS / UNRESOLVED.
|
||||
A reverse map (content sha256 -> blob oids) is what yields the sibling.
|
||||
|
||||
VERDICT CLASSES, EACH WITH ITS UNIT
|
||||
CLASS_MISMATCH a class word BOUND to the token names a family the
|
||||
measurement contradicts. This is a defect claim.
|
||||
CLASSLESS_AFTER_SIZE a token with no bound class word, directly after a bare
|
||||
size figure ("18,527 B / `T`" or "13218 / `T`") -- the
|
||||
row notation behind the :8467 defect. A RISK flag, not a
|
||||
claim: both families are legitimate there, and that is
|
||||
exactly why the swap goes unnoticed.
|
||||
BOUND_UNRESOLVED a class word bound to a token this object store cannot
|
||||
resolve (an external artifact's digest, typically).
|
||||
|
||||
THE BINDING RULE, AND WHY IT IS DIRECTIONAL
|
||||
A class word is bound only when at most BIND_GAP separator characters and nothing
|
||||
else stand between it and the token. A word binds the token it directly PRECEDES
|
||||
("sha256 `T`"), which is the troupe's notation; it binds the token it FOLLOWS
|
||||
only when no other token follows the word. Without that direction the rule reads
|
||||
"blob `A`, 19,516 B, sha256 `B`" as labelling A with sha256 -- 896 such
|
||||
false positives on this corpus at a 120-char window.
|
||||
|
||||
USAGE
|
||||
python3 citation_hash_class.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
|
||||
[--json OUT] [--only mismatch|classless|bound|flags|all] [--min-len N]
|
||||
[--context] [--siblings] [--selftest]
|
||||
"""
|
||||
import argparse, collections, hashlib, json, re, subprocess, sys
|
||||
|
||||
MIN_LEN_DEFAULT = 8
|
||||
PREFIX_LENS = (7, 8, 10, 12, 16, 20, 40, 64)
|
||||
# SEP is a bare member list, never bracketed: it is wrapped exactly once, by the
|
||||
# `[%s]` in the patterns below. Bracketing it here would double-wrap the class,
|
||||
# close it at the first inner `]`, and silently break every binding.
|
||||
SEP = r"`'\"()\[\]:,=\-\s"
|
||||
BIND_GAP = 4
|
||||
HEX_STR = re.compile(r'\b[0-9a-f]{%d,64}\b' % MIN_LEN_DEFAULT)
|
||||
# the size-adjacency pattern: a standalone number (not the digits inside a word
|
||||
# like `sha256`), optional 'B', then AT LEAST one separator, then the token.
|
||||
AFTER_SIZE = re.compile(r'(?<![0-9a-zA-Z])\d[\d,]*(?:\s*B)?[`\'"()\[\]:,=\-/*_\s]{1,10}$')
|
||||
|
||||
WORDS = [
|
||||
('content sha256', 'CONTENT_SHA256'), ('content digest', 'CONTENT_SHA256'),
|
||||
('content sha', 'CONTENT_SHA256'), ('content-sha', 'CONTENT_SHA256'),
|
||||
('sha256', 'CONTENT_SHA256'), ('sha-256', 'CONTENT_SHA256'),
|
||||
('git hash-object', 'BLOB_OID'), ('git rev-parse', 'BLOB_OID'),
|
||||
('blob oid', 'BLOB_OID'), ('blob-oid', 'BLOB_OID'), ('blob', 'BLOB_OID'),
|
||||
('commit', 'COMMIT_SHA'),
|
||||
]
|
||||
WORDS.sort(key=lambda t: -len(t[0]))
|
||||
PRE = [(re.compile(r'(?<![0-9a-z])%s[%s]{0,%d}$' % (re.escape(w), SEP, BIND_GAP), re.I), f)
|
||||
for w, f in WORDS]
|
||||
POST = [(re.compile(r'^[%s]{0,%d}(?<![0-9a-z])%s(?![0-9a-z])' % (SEP, BIND_GAP, re.escape(w)), re.I), f)
|
||||
for w, f in WORDS]
|
||||
|
||||
|
||||
def git(repo, *a):
|
||||
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
|
||||
|
||||
|
||||
def build_index(repo):
|
||||
"""One pass over the whole object store: blob oid -> (content sha256, size)."""
|
||||
meta, idx = {}, collections.defaultdict(list)
|
||||
inv = collections.defaultdict(list) # content sha256 -> [blob oid]
|
||||
|
||||
# paths: one call, every reachable object with the path it is first seen under
|
||||
paths = {}
|
||||
for line in git(repo, "rev-list", "--all", "--objects").decode('utf-8', 'replace').split('\n'):
|
||||
if not line:
|
||||
continue
|
||||
parts = line.split(' ', 1)
|
||||
if len(parts) == 2 and parts[1]:
|
||||
paths.setdefault(parts[0], parts[1])
|
||||
|
||||
p = subprocess.Popen(["git", "-C", repo, "cat-file", "--batch-all-objects", "--batch"],
|
||||
stdout=subprocess.PIPE)
|
||||
nblob = 0
|
||||
while True:
|
||||
head = p.stdout.readline()
|
||||
if not head:
|
||||
break
|
||||
parts = head.split()
|
||||
if len(parts) != 3:
|
||||
continue
|
||||
oid, otype, osize = parts[0].decode(), parts[1].decode(), int(parts[2])
|
||||
body = p.stdout.read(osize) # BufferedReader.read: exact unless EOF
|
||||
p.stdout.read(1) # the LF --batch appends after every body
|
||||
if otype != 'blob':
|
||||
continue
|
||||
nblob += 1
|
||||
csha = hashlib.sha256(body).hexdigest()
|
||||
meta[oid] = (csha, osize)
|
||||
inv[csha].append(oid)
|
||||
for L in PREFIX_LENS:
|
||||
if L <= 40:
|
||||
idx[oid[:L]].append(("BLOB_OID", oid))
|
||||
if L <= 64:
|
||||
idx[csha[:L]].append(("CONTENT_SHA256", csha))
|
||||
p.stdout.close()
|
||||
p.wait()
|
||||
|
||||
commits = [c for c in git(repo, "rev-list", "--all").decode().split('\n') if c]
|
||||
for c in commits:
|
||||
for L in PREFIX_LENS:
|
||||
if L <= 40:
|
||||
idx[c[:L]].append(("COMMIT_SHA", c))
|
||||
amb = {k for k, v in idx.items() if len(set(v)) > 1}
|
||||
return {"blob_meta": meta, "index": idx, "inverse": inv, "paths": paths,
|
||||
"commits": commits, "n_blobs": nblob, "ambiguous": amb}
|
||||
|
||||
|
||||
def bound_family(text, s, e, window):
|
||||
"""Directional binding -> (family, direction) or (None, None)."""
|
||||
pre = text[max(0, s - window):s]
|
||||
for rx, fam in PRE:
|
||||
if rx.search(pre):
|
||||
if fam == 'COMMIT_SHA' and (e - s) != 40:
|
||||
continue # "at commit `2b834d4`" is prose, not a class assertion
|
||||
return fam, 'precedes'
|
||||
post = text[e:e + window]
|
||||
for rx, fam in POST:
|
||||
m = rx.match(post)
|
||||
if not m:
|
||||
continue
|
||||
# Does this word label the NEXT token instead of this one? Strip any
|
||||
# separators (including newlines -- the corpus wraps mid-row) and see
|
||||
# whether a hex run of token length begins there.
|
||||
rest = re.sub(r'^[^0-9a-f]+', '', text[e + m.end():e + m.end() + 32])
|
||||
if HEX_STR.match(rest):
|
||||
continue
|
||||
if fam == 'COMMIT_SHA' and (e - s) != 40:
|
||||
continue # "commit 2b834d4" on a 7-hex ref is prose, not a class
|
||||
return fam, 'follows'
|
||||
return None, None
|
||||
|
||||
|
||||
def resolve(IDX, tok):
|
||||
hits = IDX["index"].get(tok)
|
||||
if not hits:
|
||||
return 'UNRESOLVED', None, 0
|
||||
fams = sorted({f for f, _ in hits})
|
||||
if len(fams) == 1:
|
||||
return fams[0], hits[0][1], len(hits)
|
||||
return 'AMBIGUOUS', hits[0][1], len(hits)
|
||||
|
||||
|
||||
def siblings(IDX, fam, full):
|
||||
"""The other family's value for this token -- the disambiguation."""
|
||||
if full is None:
|
||||
return None
|
||||
if fam == 'BLOB_OID':
|
||||
csha, size = IDX["blob_meta"].get(full, (None, None))
|
||||
return {"content_sha256": csha, "size": size, "path": IDX["paths"].get(full)}
|
||||
if fam == 'CONTENT_SHA256':
|
||||
blobs = IDX["inverse"].get(full, [])
|
||||
out = [{"blob_oid": b, "size": IDX["blob_meta"].get(b, (None, None))[1],
|
||||
"path": IDX["paths"].get(b)} for b in blobs]
|
||||
return {"blob_oids": out}
|
||||
return None
|
||||
|
||||
|
||||
def scan(repo, corpus, rev, IDX, window, min_len):
|
||||
rows, stats = [], collections.Counter()
|
||||
hexre = re.compile(r'\b[0-9a-f]{%d,64}\b' % min_len)
|
||||
for p in corpus:
|
||||
text = git(repo, "show", f"{rev}:{p}").decode('utf-8', 'replace')
|
||||
for m in hexre.finditer(text):
|
||||
tok = m.group(0)
|
||||
stats['occurrences'] += 1
|
||||
fam, full, nhits = resolve(IDX, tok)
|
||||
stats['measured_' + fam] += 1
|
||||
bf, direction = bound_family(text, m.start(), m.end(), window)
|
||||
if bf:
|
||||
stats['bound'] += 1
|
||||
after_size = bool(AFTER_SIZE.search(text[max(0, m.start() - 40):m.start()]))
|
||||
if bf and fam == 'UNRESOLVED':
|
||||
verdict = 'BOUND_UNRESOLVED'
|
||||
elif bf and fam not in ('AMBIGUOUS', 'UNRESOLVED') and bf != fam:
|
||||
verdict = 'CLASS_MISMATCH'
|
||||
elif not bf and after_size and fam in ('BLOB_OID', 'CONTENT_SHA256'):
|
||||
verdict = 'CLASSLESS_AFTER_SIZE'
|
||||
else:
|
||||
verdict = 'ok'
|
||||
stats[verdict] += 1
|
||||
rows.append({"file": p, "line": text.count('\n', 0, m.start()) + 1,
|
||||
"token": tok, "len": len(tok), "measured": fam, "full": full,
|
||||
"n_hits": nhits, "bound_family": bf, "bound_direction": direction,
|
||||
"after_size": after_size, "verdict": verdict,
|
||||
"context": text[max(0, m.start() - 80):m.end() + 80].replace('\n', ' ')})
|
||||
return rows, stats
|
||||
|
||||
|
||||
# ---------------------------------------------------------------- self-test
|
||||
BINDING_FIXTURES = [
|
||||
# (text, token, expected family, expected direction) or (.., None, None)
|
||||
("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "precedes"),
|
||||
("blob `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "precedes"),
|
||||
("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef",
|
||||
"CONTENT_SHA256", "precedes"),
|
||||
# the first token is explicitly labelled a blob; the trailing `sha256` labels
|
||||
# the SECOND token and must not leak backwards onto it
|
||||
("blob `deadbeefdeadbeef`, 19,516 B, sha256 `bbbbbbbbbbbb`", "deadbeefdeadbeef",
|
||||
"BLOB_OID", "precedes"),
|
||||
("commit `deadbeefdeadbeefdeadbeefdeadbeefdeadbeef`", "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef",
|
||||
"COMMIT_SHA", "precedes"),
|
||||
("at commit `deadbeefdeadbeef`, drop", "deadbeefdeadbeef", None, None),
|
||||
# STATED LIMITATION: a class word further than BIND_GAP=4 chars from the token
|
||||
# is deliberately NOT bound. The rule under-reports; it never invents a class.
|
||||
("git rev-parse HEAD:p -> `deadbeefdeadbeef`", "deadbeefdeadbeef", None, None),
|
||||
# REGRESSION (WORK-QUEUE.md:432, replies/2026-10-04-to-zephyrus-29.md:52):
|
||||
# `sha256` here labels the NEXT token; it must not bind backwards onto the blob.
|
||||
("blob `277c8baa183f` -> `deadbeefdeadbeef`, `sha256 078c704a1627b809`",
|
||||
"deadbeefdeadbeef", None, None),
|
||||
("`deadbeefdeadbeef`, `sha256\n078c704a1627b809`", "deadbeefdeadbeef", None, None),
|
||||
# REGRESSION (replies/2026-09-26-to-zephyrus-13.md:40): same, one separator short.
|
||||
("(`head 7c4cf1b8e79f`, sha256 `4099fba9c11170e8`)", "7c4cf1b8e79f", None, None),
|
||||
]
|
||||
VERDICT_FIXTURES = [
|
||||
# (text, token, family_of_token, expected verdict)
|
||||
("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef",
|
||||
"CONTENT_SHA256", "ok"),
|
||||
("18,527 B / `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASSLESS_AFTER_SIZE"),
|
||||
("1827 / `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "CLASSLESS_AFTER_SIZE"),
|
||||
("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASS_MISMATCH"),
|
||||
]
|
||||
|
||||
|
||||
def selftest(repo):
|
||||
n = bad = 0
|
||||
for text, tok, want_f, want_d in BINDING_FIXTURES:
|
||||
i = text.index(tok)
|
||||
got_f, got_d = bound_family(text, i, i + len(tok), 40)
|
||||
n += 1
|
||||
ok = (got_f, got_d) == (want_f, want_d)
|
||||
bad += 0 if ok else 1
|
||||
print(f"{'PASS' if ok else 'FAIL'} bind {tok[:12]} in {text[:44]!r} "
|
||||
f"-> {(got_f, got_d)} want {(want_f, want_d)}")
|
||||
for text, tok, fam, want in VERDICT_FIXTURES:
|
||||
i = text.index(tok)
|
||||
bf, _ = bound_family(text, i, i + len(tok), 40)
|
||||
after = bool(AFTER_SIZE.search(text[max(0, i - 40):i]))
|
||||
if bf and fam == 'UNRESOLVED':
|
||||
got = 'BOUND_UNRESOLVED'
|
||||
elif bf and bf != fam:
|
||||
got = 'CLASS_MISMATCH'
|
||||
elif not bf and after and fam in ('BLOB_OID', 'CONTENT_SHA256'):
|
||||
got = 'CLASSLESS_AFTER_SIZE'
|
||||
else:
|
||||
got = 'ok'
|
||||
n += 1
|
||||
ok = got == want
|
||||
bad += 0 if ok else 1
|
||||
print(f"{'PASS' if ok else 'FAIL'} verdict {text[:38]!r} -> {got} want {want}")
|
||||
|
||||
# repo-anchored fixtures: the :8467 defect's own tokens
|
||||
IDX = build_index(repo)
|
||||
for tok, want_fam, want_full in (
|
||||
("83acffe0", "CONTENT_SHA256",
|
||||
"83acffe0222d46e7435aed2e79dffca39e503554b2110e393fbd0cc12ad30265"),
|
||||
("decd3350", "CONTENT_SHA256",
|
||||
"decd3350585dd169d78995fd08f4d1120394c884b764198bd419a4bb1cf4a265"),
|
||||
("4f53c93a708d", "BLOB_OID",
|
||||
"4f53c93a708d95200e61f50d93a23c8b2119975b"),
|
||||
("4e3cca9cffd2", "BLOB_OID",
|
||||
"4e3cca9cffd2ace531030c9a1dd5bed25412ffab"),
|
||||
("2b834d4", "COMMIT_SHA", None)):
|
||||
fam, full, _ = resolve(IDX, tok)
|
||||
n += 1
|
||||
ok = fam == want_fam and (want_full is None or full == want_full)
|
||||
bad += 0 if ok else 1
|
||||
print(f"{'PASS' if ok else 'FAIL'} resolve {tok} -> {fam} {full} want {want_fam} {want_full}")
|
||||
print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed")
|
||||
return 0 if bad == 0 else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument('--repo', default='/root/troupe-standup')
|
||||
ap.add_argument('--rev', default='HEAD')
|
||||
ap.add_argument('--corpus', nargs='*')
|
||||
ap.add_argument('--json')
|
||||
ap.add_argument('--window', type=int, default=24)
|
||||
ap.add_argument('--min-len', type=int, default=MIN_LEN_DEFAULT)
|
||||
ap.add_argument('--only', choices=['mismatch', 'classless', 'bound', 'flags', 'all'],
|
||||
default='flags')
|
||||
ap.add_argument('--context', action='store_true')
|
||||
ap.add_argument('--siblings', action='store_true')
|
||||
ap.add_argument('--selftest', action='store_true')
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.selftest:
|
||||
return selftest(a.repo)
|
||||
|
||||
corpus = a.corpus or [p for p in git(a.repo, "ls-tree", "-r", "--name-only", a.rev)
|
||||
.decode('utf-8', 'replace').split('\n') if p]
|
||||
IDX = build_index(a.repo)
|
||||
rows, stats = scan(a.repo, corpus, a.rev, IDX, a.window, a.min_len)
|
||||
|
||||
print(f"corpus: {len(corpus)} files at {a.rev} in {a.repo}")
|
||||
print(f"index: {IDX['n_blobs']} blobs, {len(IDX['commits'])} commits, "
|
||||
f"{len(IDX['ambiguous'])} ambiguous prefixes (whole object store, so "
|
||||
f"historical tokens resolve)")
|
||||
print(f"hex tokens >={a.min_len} chars: {stats['occurrences']} occurrences "
|
||||
f"(unit: occurrences)")
|
||||
print("--- measured family (unit: occurrences) ---")
|
||||
for k in ('BLOB_OID', 'CONTENT_SHA256', 'COMMIT_SHA', 'AMBIGUOUS', 'UNRESOLVED'):
|
||||
if stats['measured_' + k]:
|
||||
print(f" {k:16s} {stats['measured_' + k]}")
|
||||
print(f"tokens carrying a BOUND class word (gap<={BIND_GAP}): {stats['bound']} occurrences")
|
||||
for v in ('CLASS_MISMATCH', 'CLASSLESS_AFTER_SIZE', 'BOUND_UNRESOLVED'):
|
||||
print(f"{v}: {stats[v]} occurrences")
|
||||
|
||||
sel = {'mismatch': lambda r: r['verdict'] == 'CLASS_MISMATCH',
|
||||
'classless': lambda r: r['verdict'] == 'CLASSLESS_AFTER_SIZE',
|
||||
'bound': lambda r: r['bound_family'],
|
||||
'flags': lambda r: r['verdict'] != 'ok',
|
||||
'all': lambda r: True}[a.only]
|
||||
all_rows = rows # the JSON always carries the FULL census
|
||||
rows = [r for r in rows if sel(r)]
|
||||
print(f"--- {a.only}: {len(rows)} rows ---")
|
||||
for r in rows[:400]:
|
||||
print(f" {r['file']}:{r['line']} {r['token']} measured={r['measured']} "
|
||||
f"bound={r['bound_family']}({r['bound_direction']}) verdict={r['verdict']}")
|
||||
if a.context:
|
||||
print(f" …{r['context']}…")
|
||||
if a.siblings and r['full']:
|
||||
s = siblings(IDX, r['measured'], r['full'])
|
||||
if s:
|
||||
print(f" sibling: {json.dumps(s)}")
|
||||
if a.json:
|
||||
with open(a.json, 'w') as fh:
|
||||
json.dump({"rev": a.rev, "corpus": corpus, "window": a.window,
|
||||
"min_len": a.min_len,
|
||||
"index": {"blobs": IDX['n_blobs'], "commits": len(IDX['commits']),
|
||||
"ambiguous_prefixes": len(IDX['ambiguous'])},
|
||||
"stats": dict(stats), "findings": all_rows}, fh, indent=1)
|
||||
print(f"wrote {a.json}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user