#!/usr/bin/env python3
"""citation_hash_class.py -- resolve every hex token in a corpus to its hash family,
and flag the tokens that carry no stated class.

WHY THIS EXISTS
  The troupe's most-recurring manual defect is "a token quoted without its class":
  a git blob oid cited where a content sha256 belongs, or a sha256 cited where a
  blob oid belongs. Both families are hex, so to the eye they are the same token.
  It is logged at least three times on this repo:
    WORK-QUEUE.md:8467  two `git cat-file -s` rows print the seed's CONTENT digests
                        (`83acffe0...`, `decd3350...`) where the git blobs belong
                        (`4f53c93a708d...`, `4e3cca9cffd2...`).
    WORK-QUEUE.md:8497  the mirror: a block calls the blob oids "unsupported" and
                        proposes the content digests in their place.
    WORK-QUEUE.md:8510  the lane's own summary: "a token quoted without its class
                        ... a hash family swapped for its sibling, with no byte
                        figure moving".
  This tool makes the class explicit by RESOLVING each token against the object
  database instead of trusting the word beside it -- and, for every flag, prints
  the SIBLING value, which is the disambiguation that defect cost a full round trip.

WHAT IT MEASURES (read-only; the only write path is --json)
  Built once over the whole object store, so HISTORICAL tokens resolve too:
    blob oid        every blob ever (`git cat-file --batch-all-objects`)
    content sha256  sha256 of that blob's bytes (same single streaming pass)
    size            byte size of that blob
    path            the first path the blob is reachable under (`rev-list --objects`)
    commit sha      `git rev-list --all`
  Every token (>= --min-len hex chars) is matched BY PREFIX against the index and
  classified BLOB_OID / CONTENT_SHA256 / COMMIT_SHA / AMBIGUOUS / UNRESOLVED.
  A reverse map (content sha256 -> blob oids) is what yields the sibling.

SCOPING LIMITS -- THE TWO COUNTS ON THE `index:` LINE ARE NOT ONE UNIT (measured)
  The `index:` line has printed a blob count and a commit count on one line since
  v1, and this tool's first reader (replies/2026-10-15-to-volt-02.md, note 2) was
  right that they must not be cited as repo invariants. Re-measured here
  2026-10-15 at rev 9188dd3, same corpus both ways:

    blobs   = every blob in the LOCAL object store (`--batch-all-objects`),
              reachable or not. CLONE-LOCAL. Measured: this working clone 871,
              a true clone (`git clone --no-local`, and a clone off the forge)
              866. The +5 is exactly what `git fsck --unreachable --no-reflogs`
              lists as unreachable loose blobs (5): objects a fresh clone does
              not carry. NEVER cite a blob count without naming the clone.
    commits = `git rev-list --all` only: ref-REACHABLE commits. It follows the
              refs, not the object store, so it is invariant across clones at
              equal refs (both clones above print 680, refs identical at
              bbe8d0d) and unreachable commits are invisible to it. The two
              counts therefore move for different reasons.

  WHAT DOES CARRY across clones: every token figure -- corpus files, occurrences,
  the measured family, BOUND, and all three verdict classes. Identical on both
  clones at 9188dd3: 307 files / 9,355 occurrences / 594 BLOB_OID / 1,062
  CONTENT_SHA256 / 222 COMMIT_SHA / 7,477 UNRESOLVED / BOUND 1,034 /
  CLASS_MISMATCH 0 / CLASSLESS_AFTER_SIZE 485 / BOUND_UNRESOLVED 615. Cite those
  freely; cite the index with its clone named.

  NOT YET IN SCOPE, stated rather than implied: a commit cannot measure the file
  it introduces, so a census at the rev that ships this tool reports one file and
  its tokens more than the same tool reports at that rev's parent (307/9,355 at
  9188dd3 against 306/9,296 at 81bcd796). Name the rev with the number.

VERDICT CLASSES, EACH WITH ITS UNIT
  CLASS_MISMATCH          a class word BOUND to the token names a family the
                          measurement contradicts. This is a defect claim.
  CLASSLESS_AFTER_SIZE    a token with no bound class word, directly after a bare
                          size figure ("18,527 B / `T`" or "13218 / `T`") -- the
                          row notation behind the :8467 defect. A RISK flag, not a
                          claim: both families are legitimate there, and that is
                          exactly why the swap goes unnoticed.
  BOUND_UNRESOLVED        a class word bound to a token this object store cannot
                          resolve (an external artifact's digest, typically).

THE BINDING RULE, AND WHY IT IS DIRECTIONAL
  A class word is bound only when at most BIND_GAP separator characters and nothing
  else stand between it and the token. A word binds the token it directly PRECEDES
  ("sha256 `T`"), which is the troupe's notation; it binds the token it FOLLOWS
  only when no other token follows the word. Without that direction the rule reads
  "blob `A`, 19,516 B, sha256 `B`" as labelling A with sha256 -- 896 such
  false positives on this corpus at a 120-char window.

USAGE
  python3 citation_hash_class.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
      [--json OUT] [--only mismatch|classless|bound|flags|all] [--min-len N]
      [--context] [--siblings] [--selftest]        # 19 fixtures
"""
import argparse, collections, hashlib, json, re, subprocess, sys

MIN_LEN_DEFAULT = 8
PREFIX_LENS = (7, 8, 10, 12, 16, 20, 40, 64)
# SEP is a bare member list, never bracketed: it is wrapped exactly once, by the
# `[%s]` in the patterns below. Bracketing it here would double-wrap the class,
# close it at the first inner `]`, and silently break every binding.
SEP = r"`'\"()\[\]:,=\-\s"
BIND_GAP = 4
HEX_STR = re.compile(r'\b[0-9a-f]{%d,64}\b' % MIN_LEN_DEFAULT)
# the size-adjacency pattern: a standalone number (not the digits inside a word
# like `sha256`), optional 'B', then AT LEAST one separator, then the token.
AFTER_SIZE = re.compile(r'(?<![0-9a-zA-Z])\d[\d,]*(?:\s*B)?[`\'"()\[\]:,=\-/*_\s]{1,10}$')

WORDS = [
    ('content sha256', 'CONTENT_SHA256'), ('content digest', 'CONTENT_SHA256'),
    ('content sha', 'CONTENT_SHA256'), ('content-sha', 'CONTENT_SHA256'),
    ('sha256', 'CONTENT_SHA256'), ('sha-256', 'CONTENT_SHA256'),
    ('git hash-object', 'BLOB_OID'), ('git rev-parse', 'BLOB_OID'),
    ('blob oid', 'BLOB_OID'), ('blob-oid', 'BLOB_OID'), ('blob', 'BLOB_OID'),
    ('commit', 'COMMIT_SHA'),
]
WORDS.sort(key=lambda t: -len(t[0]))
PRE = [(re.compile(r'(?<![0-9a-z])%s[%s]{0,%d}$' % (re.escape(w), SEP, BIND_GAP), re.I), f)
       for w, f in WORDS]
POST = [(re.compile(r'^[%s]{0,%d}(?<![0-9a-z])%s(?![0-9a-z])' % (SEP, BIND_GAP, re.escape(w)), re.I), f)
        for w, f in WORDS]


def git(repo, *a):
    return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout


def precondition(repo, rev):
    """None when `repo` is a git work tree that carries `rev`, else why not.

    The caller turns this into exit 2. It exists because every reader below
    swallows git's exit status: an unreadable repo or a mistyped revision came
    back as an empty tree, and the census printed zeros over it with rc 0.
    """
    p = subprocess.run(["git", "-C", repo, "rev-parse", "--verify",
                        rev + "^{commit}"], capture_output=True)
    if p.returncode == 0:
        return None
    why = p.stderr.decode("utf-8", "replace").strip().splitlines()
    return ("cannot read %s at %s: %s" % (repo, rev, why[-1] if why else "git rev-parse failed"))


def build_index(repo):
    """One pass over the whole object store: blob oid -> (content sha256, size)."""
    meta, idx = {}, collections.defaultdict(list)
    inv = collections.defaultdict(list)          # content sha256 -> [blob oid]

    # paths: one call, every reachable object with the path it is first seen under
    paths = {}
    for line in git(repo, "rev-list", "--all", "--objects").decode('utf-8', 'replace').split('\n'):
        if not line:
            continue
        parts = line.split(' ', 1)
        if len(parts) == 2 and parts[1]:
            paths.setdefault(parts[0], parts[1])

    p = subprocess.Popen(["git", "-C", repo, "cat-file", "--batch-all-objects", "--batch"],
                         stdout=subprocess.PIPE)
    nblob = 0
    while True:
        head = p.stdout.readline()
        if not head:
            break
        parts = head.split()
        if len(parts) != 3:
            continue
        oid, otype, osize = parts[0].decode(), parts[1].decode(), int(parts[2])
        body = p.stdout.read(osize)              # BufferedReader.read: exact unless EOF
        p.stdout.read(1)                         # the LF --batch appends after every body
        if otype != 'blob':
            continue
        nblob += 1
        csha = hashlib.sha256(body).hexdigest()
        meta[oid] = (csha, osize)
        inv[csha].append(oid)
        for L in PREFIX_LENS:
            if L <= 40:
                idx[oid[:L]].append(("BLOB_OID", oid))
            if L <= 64:
                idx[csha[:L]].append(("CONTENT_SHA256", csha))
    p.stdout.close()
    p.wait()

    commits = [c for c in git(repo, "rev-list", "--all").decode().split('\n') if c]
    for c in commits:
        for L in PREFIX_LENS:
            if L <= 40:
                idx[c[:L]].append(("COMMIT_SHA", c))
    amb = {k for k, v in idx.items() if len(set(v)) > 1}
    return {"blob_meta": meta, "index": idx, "inverse": inv, "paths": paths,
            "commits": commits, "n_blobs": nblob, "ambiguous": amb}


def bound_family(text, s, e, window):
    """Directional binding -> (family, direction) or (None, None)."""
    pre = text[max(0, s - window):s]
    for rx, fam in PRE:
        if rx.search(pre):
            if fam == 'COMMIT_SHA' and (e - s) != 40:
                continue      # "at commit `2b834d4`" is prose, not a class assertion
            return fam, 'precedes'
    post = text[e:e + window]
    for rx, fam in POST:
        m = rx.match(post)
        if not m:
            continue
        # Does this word label the NEXT token instead of this one? Strip any
        # separators (including newlines -- the corpus wraps mid-row) and see
        # whether a hex run of token length begins there.
        rest = re.sub(r'^[^0-9a-f]+', '', text[e + m.end():e + m.end() + 32])
        if HEX_STR.match(rest):
            continue
        if fam == 'COMMIT_SHA' and (e - s) != 40:
            continue          # "commit 2b834d4" on a 7-hex ref is prose, not a class
        return fam, 'follows'
    return None, None


def resolve(IDX, tok):
    hits = IDX["index"].get(tok)
    if not hits:
        return 'UNRESOLVED', None, 0
    fams = sorted({f for f, _ in hits})
    if len(fams) == 1:
        return fams[0], hits[0][1], len(hits)
    return 'AMBIGUOUS', hits[0][1], len(hits)


def siblings(IDX, fam, full):
    """The other family's value for this token -- the disambiguation."""
    if full is None:
        return None
    if fam == 'BLOB_OID':
        csha, size = IDX["blob_meta"].get(full, (None, None))
        return {"content_sha256": csha, "size": size, "path": IDX["paths"].get(full)}
    if fam == 'CONTENT_SHA256':
        blobs = IDX["inverse"].get(full, [])
        out = [{"blob_oid": b, "size": IDX["blob_meta"].get(b, (None, None))[1],
                "path": IDX["paths"].get(b)} for b in blobs]
        return {"blob_oids": out}
    return None


def scan(repo, corpus, rev, IDX, window, min_len):
    rows, stats = [], collections.Counter()
    hexre = re.compile(r'\b[0-9a-f]{%d,64}\b' % min_len)
    for p in corpus:
        text = git(repo, "show", f"{rev}:{p}").decode('utf-8', 'replace')
        for m in hexre.finditer(text):
            tok = m.group(0)
            stats['occurrences'] += 1
            fam, full, nhits = resolve(IDX, tok)
            stats['measured_' + fam] += 1
            bf, direction = bound_family(text, m.start(), m.end(), window)
            if bf:
                stats['bound'] += 1
            after_size = bool(AFTER_SIZE.search(text[max(0, m.start() - 40):m.start()]))
            if bf and fam == 'UNRESOLVED':
                verdict = 'BOUND_UNRESOLVED'
            elif bf and fam not in ('AMBIGUOUS', 'UNRESOLVED') and bf != fam:
                verdict = 'CLASS_MISMATCH'
            elif not bf and after_size and fam in ('BLOB_OID', 'CONTENT_SHA256'):
                verdict = 'CLASSLESS_AFTER_SIZE'
            else:
                verdict = 'ok'
            stats[verdict] += 1
            rows.append({"file": p, "line": text.count('\n', 0, m.start()) + 1,
                         "token": tok, "len": len(tok), "measured": fam, "full": full,
                         "n_hits": nhits, "bound_family": bf, "bound_direction": direction,
                         "after_size": after_size, "verdict": verdict,
                         "context": text[max(0, m.start() - 80):m.end() + 80].replace('\n', ' ')})
    return rows, stats


# ---------------------------------------------------------------- self-test
BINDING_FIXTURES = [
    # (text, token, expected family, expected direction) or (.., None, None)
    ("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "precedes"),
    ("blob `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "precedes"),
    ("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef",
     "CONTENT_SHA256", "precedes"),
    # the first token is explicitly labelled a blob; the trailing `sha256` labels
    # the SECOND token and must not leak backwards onto it
    ("blob `deadbeefdeadbeef`, 19,516 B, sha256 `bbbbbbbbbbbb`", "deadbeefdeadbeef",
     "BLOB_OID", "precedes"),
    ("commit `deadbeefdeadbeefdeadbeefdeadbeefdeadbeef`", "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef",
     "COMMIT_SHA", "precedes"),
    ("at commit `deadbeefdeadbeef`, drop", "deadbeefdeadbeef", None, None),
    # STATED LIMITATION: a class word further than BIND_GAP=4 chars from the token
    # is deliberately NOT bound. The rule under-reports; it never invents a class.
    ("git rev-parse HEAD:p -> `deadbeefdeadbeef`", "deadbeefdeadbeef", None, None),
    # REGRESSION (WORK-QUEUE.md:432, replies/2026-10-04-to-zephyrus-29.md:52):
    # `sha256` here labels the NEXT token; it must not bind backwards onto the blob.
    ("blob `277c8baa183f` -> `deadbeefdeadbeef`, `sha256 078c704a1627b809`",
     "deadbeefdeadbeef", None, None),
    ("`deadbeefdeadbeef`, `sha256\n078c704a1627b809`", "deadbeefdeadbeef", None, None),
    # REGRESSION (replies/2026-09-26-to-zephyrus-13.md:40): same, one separator short.
    ("(`head 7c4cf1b8e79f`, sha256 `4099fba9c11170e8`)", "7c4cf1b8e79f", None, None),
]
VERDICT_FIXTURES = [
    # (text, token, family_of_token, expected verdict)
    ("blob `aaaaaaaaaaaa`, 19,516 B, sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef",
     "CONTENT_SHA256", "ok"),
    ("18,527 B / `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASSLESS_AFTER_SIZE"),
    ("1827 / `deadbeefdeadbeef`", "deadbeefdeadbeef", "CONTENT_SHA256", "CLASSLESS_AFTER_SIZE"),
    ("sha256 `deadbeefdeadbeef`", "deadbeefdeadbeef", "BLOB_OID", "CLASS_MISMATCH"),
]


def selftest(repo):
    n = bad = 0
    for text, tok, want_f, want_d in BINDING_FIXTURES:
        i = text.index(tok)
        got_f, got_d = bound_family(text, i, i + len(tok), 40)
        n += 1
        ok = (got_f, got_d) == (want_f, want_d)
        bad += 0 if ok else 1
        print(f"{'PASS' if ok else 'FAIL'} bind {tok[:12]} in {text[:44]!r} "
              f"-> {(got_f, got_d)} want {(want_f, want_d)}")
    for text, tok, fam, want in VERDICT_FIXTURES:
        i = text.index(tok)
        bf, _ = bound_family(text, i, i + len(tok), 40)
        after = bool(AFTER_SIZE.search(text[max(0, i - 40):i]))
        if bf and fam == 'UNRESOLVED':
            got = 'BOUND_UNRESOLVED'
        elif bf and bf != fam:
            got = 'CLASS_MISMATCH'
        elif not bf and after and fam in ('BLOB_OID', 'CONTENT_SHA256'):
            got = 'CLASSLESS_AFTER_SIZE'
        else:
            got = 'ok'
        n += 1
        ok = got == want
        bad += 0 if ok else 1
        print(f"{'PASS' if ok else 'FAIL'} verdict {text[:38]!r} -> {got} want {want}")

    # repo-anchored fixtures: the :8467 defect's own tokens
    IDX = build_index(repo)
    for tok, want_fam, want_full in (
            ("83acffe0", "CONTENT_SHA256",
             "83acffe0222d46e7435aed2e79dffca39e503554b2110e393fbd0cc12ad30265"),
            ("decd3350", "CONTENT_SHA256",
             "decd3350585dd169d78995fd08f4d1120394c884b764198bd419a4bb1cf4a265"),
            ("4f53c93a708d", "BLOB_OID",
             "4f53c93a708d95200e61f50d93a23c8b2119975b"),
            ("4e3cca9cffd2", "BLOB_OID",
             "4e3cca9cffd2ace531030c9a1dd5bed25412ffab"),
            ("2b834d4", "COMMIT_SHA", None)):
        fam, full, _ = resolve(IDX, tok)
        n += 1
        ok = fam == want_fam and (want_full is None or full == want_full)
        bad += 0 if ok else 1
        print(f"{'PASS' if ok else 'FAIL'} resolve {tok} -> {fam} {full} want {want_fam} {want_full}")
    print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed")
    return 0 if bad == 0 else 1


def main():
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument('--repo', default='/root/troupe-standup')
    ap.add_argument('--rev', default='HEAD')
    ap.add_argument('--corpus', nargs='*')
    ap.add_argument('--json')
    ap.add_argument('--window', type=int, default=24)
    ap.add_argument('--min-len', type=int, default=MIN_LEN_DEFAULT)
    ap.add_argument('--only', choices=['mismatch', 'classless', 'bound', 'flags', 'all'],
                    default='flags')
    ap.add_argument('--context', action='store_true')
    ap.add_argument('--siblings', action='store_true')
    ap.add_argument('--selftest', action='store_true')
    a = ap.parse_args()

    if a.selftest:
        return selftest(a.repo)

    # A repo or revision git cannot read is a precondition failure, never a
    # clean census. `git()` returns stdout only, so `--repo /nonexistent` used to
    # print "corpus: 0 files ... 0 flags" and exit 0 -- a silent green on the one
    # input this tool cannot see. Exit 2 says "did not measure", which is true.
    problem = precondition(a.repo, a.rev)
    if problem:
        print(problem, file=sys.stderr)
        return 2

    corpus = a.corpus or [p for p in git(a.repo, "ls-tree", "-r", "--name-only", a.rev)
                          .decode('utf-8', 'replace').split('\n') if p]
    IDX = build_index(a.repo)
    rows, stats = scan(a.repo, corpus, a.rev, IDX, a.window, a.min_len)

    print(f"corpus: {len(corpus)} files at {a.rev} in {a.repo}")
    # Two counts, two scopes, one line since v1 -- see SCOPING LIMITS in the
    # module docstring before citing either. The token figures below are the
    # clone-invariant ones; this line is not.
    print(f"index: {IDX['n_blobs']} blobs (unit: blobs in THIS clone's local object "
          f"store, reachable or not -- CLONE-LOCAL, not a repo invariant), "
          f"{len(IDX['commits'])} commits (unit: ref-reachable commits, "
          f"`git rev-list --all`), {len(IDX['ambiguous'])} ambiguous prefixes "
          f"(whole object store, so historical tokens resolve)")
    print(f"hex tokens >={a.min_len} chars: {stats['occurrences']} occurrences "
          f"(unit: occurrences)")
    print("--- measured family (unit: occurrences) ---")
    for k in ('BLOB_OID', 'CONTENT_SHA256', 'COMMIT_SHA', 'AMBIGUOUS', 'UNRESOLVED'):
        if stats['measured_' + k]:
            print(f"  {k:16s} {stats['measured_' + k]}")
    print(f"tokens carrying a BOUND class word (gap<={BIND_GAP}): {stats['bound']} occurrences")
    for v in ('CLASS_MISMATCH', 'CLASSLESS_AFTER_SIZE', 'BOUND_UNRESOLVED'):
        print(f"{v}: {stats[v]} occurrences")

    sel = {'mismatch': lambda r: r['verdict'] == 'CLASS_MISMATCH',
           'classless': lambda r: r['verdict'] == 'CLASSLESS_AFTER_SIZE',
           'bound': lambda r: r['bound_family'],
           'flags': lambda r: r['verdict'] != 'ok',
           'all': lambda r: True}[a.only]
    all_rows = rows                      # the JSON always carries the FULL census
    rows = [r for r in rows if sel(r)]
    print(f"--- {a.only}: {len(rows)} rows ---")
    for r in rows[:400]:
        print(f"  {r['file']}:{r['line']}  {r['token']}  measured={r['measured']} "
              f"bound={r['bound_family']}({r['bound_direction']})  verdict={r['verdict']}")
        if a.context:
            print(f"      …{r['context']}…")
        if a.siblings and r['full']:
            s = siblings(IDX, r['measured'], r['full'])
            if s:
                print(f"      sibling: {json.dumps(s)}")
    if a.json:
        with open(a.json, 'w') as fh:
            json.dump({"rev": a.rev, "corpus": corpus, "window": a.window,
                       "min_len": a.min_len,
                       "index": {"blobs": IDX['n_blobs'],
                                 "blobs_scope": "local object store, reachable or "
                                                "not (clone-local, not a repo invariant)",
                                 "commits": len(IDX['commits']),
                                 "commits_scope": "ref-reachable (rev-list --all; "
                                                  "clone-invariant at equal refs)",
                                 "ambiguous_prefixes": len(IDX['ambiguous'])},
                       "stats": dict(stats), "findings": all_rows}, fh, indent=1)
        print(f"wrote {a.json}")
    return 0


if __name__ == '__main__':
    sys.exit(main())