troupe-standup/tools/citation_resolve_check.py

#!/usr/bin/env python3
"""citation_resolve_check.py -- does every `<file>:<line>` citation in this
archive's prose actually resolve, and to which file?

WHY THIS EXISTS
  This archive cites itself by line number. Its most-recurring review defect is
  not a wrong count and not a wrong hash family -- it is a citation that does not
  point where the sentence says it points. Logged in the ledger's own words:

    WORK-QUEUE.md:8586  "the identifiers printed for Zephyrus's newest file are
                        the 11th's ... Filename and *Blocked on* text were right,
                        the identifiers wrong."  Third instance of this class in
                        one week (reply 51 §3 -> :8467; reply 53 §0).
    WORK-QUEUE.md:8003  cites to :2264/:2684/:6847 "read back and checked by hand",
                        because nothing checked them automatically.
    WORK-QUEUE.md:8009  "The hygiene count confirmed at two revisions, not one"
                        -- the same read repeated by hand for want of a tool.

  citation_hash_class.py settles a token's FAMILY. queue_claim_check.py settles
  an item's CLOSURE. Neither settles whether a `path:line` lands on the line the
  citation is offered as evidence for. That is the third axis, and this is it.

WHAT IT MEASURES (read-only; the only write path is --json)
  Two citation forms, in one left-to-right pass per file, paragraph by paragraph.

  EXPLICIT  `<path>:<line>`, and `<path>:<line>-<line>` for a range. The file is
            named AT the citation, so the verdict is a CLAIM:
              RESOLVES       path in the tree at --rev, 1 <= line <= line_count
              STALE_LINE     path in the tree, line > line_count -- the citation
                             points past the end of the file it names
              EXTERNAL_PATH  path not in this tree at --rev (a host path under
                             /root, or another repository's file). OUT OF SCOPE
                             and counted separately: the citation may be
                             perfectly true, this tool cannot read it, and saying
                             so is the honest verdict.

  BARE      `:<line>`, the troupe's inheritance idiom -- "the file last named".
            This is NOT a claim and this tool refuses to make it one. It was
            written to resolve the inherited file, and the first run FALSIFIED
            that rule at scale (see below). Two candidates are offered for every
            bare cite and each is resolved separately:
              candidate 1  the nearest preceding path NAMED in the same
                           blank-line-delimited block (the idiom as written)
              candidate 2  the file the citation is written IN
              AS_NAMED        only candidate 1 has that line
              AS_SELF         only candidate 2 has that line -- the named-file
                              rule mis-binds and the citation is still resolvable
              AMBIGUOUS       both have it, so the number does not decide
              EXTERNAL_NAMED  candidate 1 is not in this tree at all (a host
                              artifact); out of scope, same class as
                              EXTERNAL_PATH -- and ALSO the shape a mis-bind
                              takes when the last-named path is a host file
              UNRESOLVED      no candidate in this tree has the line and nothing
                              external was named: no outside reader can resolve
                              this citation at --rev. This is the defect claim.
            Both candidates and both resolutions are printed for every row, so a
            reader never has to trust the classifier.

WHY BARE CITES ARE NOT RESOLVED BY THE NAMED-FILE RULE -- measured, not argued
  The rule was implemented as a verdict first, and the run falsified it. Three of
  the largest inherited-"stale" groups were mis-binds, each verified by reading
  the cited text at HEAD:
    WORK-QUEUE.md:9804  ":9622's own code span" -- 9622 is a WORK-QUEUE.md line
                        (the queue row describing the tool); the rule had it bound
                        to tools/queue_claim_check.py, named 40 chars earlier.
    WORK-QUEUE.md:8023  the hygiene table's `:6348` column -- WORK-QUEUE.md lines
                        in a row whose LAST column names the artifact the row is
                        ABOUT. A table row names a file for a different purpose
                        than the one the line number belongs to.
    checkins/2026-10-15-merlin.md:739  "`:2264` / `:2684` / `:6847`" -- again
                        WORK-QUEUE.md lines, this time bound to the host file
                        root/crucible/q51/helm-selftest-q51.py named in an
                        earlier sentence of the same block (EXTERNAL_NAMED).
  So "nearest named file" is a candidate; candidate 2 is what rescues the
  residue. AS_SELF is the count of citations the named-file rule alone would have
  reported as defects and that in fact resolve against the file they are in.

THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
    EXPLICIT  410 citations: 113 RESOLVES / 0 STALE_LINE / 297 EXTERNAL_PATH.
              Every explicit citation whose file is in this repo resolves.
    BARE     1612 citations: 119 AS_NAMED / 716 AS_SELF / 51 AMBIGUOUS /
              255 EXTERNAL_NAMED / 471 UNRESOLVED.
              Only 119 of 1612 bare cites are resolvable by the file-naming
              idiom the archive documents; 471 cannot be resolved by any reader
              at HEAD without inferring the referent from context. That is the
              number this tool exists to make checkable, and it is the honest
              form of the ledger's own "a token quoted without its class".
    mentions 6961 file mentions carrying no line -- these seed the scope.
  Read every figure as a citation count, never as a defect count: 471 UNRESOLVED
  is not 471 errors. The tool's own catch is the worked example below.

WORKED CATCH -- this tool's first finding is one of MY OWN artifacts
  checkins/2026-10-15-volt.md:9 reads "the three open items belong to other lanes
  (Q37 `:5278` the pipe's, `:8530` Mercury's, the career-lane apply queue `:7303`
  Alexander's)". Read at HEAD: that file is 281 lines, so neither candidate has
  line 5278; the block's last-named path is tools/queue_claim_check.py, which is
  695 lines. Verdict UNRESOLVED, and it is correct -- the three numbers are
  WORK-QUEUE.md lines (5278 is Q37's Status line, 8530 is Q53's, 7303 is the Q53
  opening), and the citation does not say so. Any troupe reader infers it; a
  stranger cannot check it. Reproduce with:
    python3 citation_resolve_check.py --only unresolved --context | grep volt.md

  NOT in scope, stated rather than implied: whether the cited LINE contains the
  evidence the sentence claims. That needs the claim, not just the pointer. And a
  citation REPRODUCED INSIDE QUOTES is still a citation to this scanner, which
  does not parse quotation marks -- so a document that reports a defective
  citation verbatim appears to contain it. Measured instance: the landing
  check-in for this tool reproduces the defective citations it is reporting,
  verbatim, and the scanner counts every one of them.

USAGE
  python3 citation_resolve_check.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
      [--json OUT] [--only stale|asself|unresolved|ambiguous|external|externalnamed|named|all]
      [--context] [--selftest]
"""
import argparse
import collections
import json
import re
import subprocess
import sys

DEFAULT_REPO = "/root/troupe-standup"

# The extensions this archive actually cites by line. Deliberately a closed list:
# an open `\.[a-z]+` would swallow clock times and domain names.
EXT = (r"md|markdown|py|json|txt|js|mjs|cjs|sh|bash|yml|yaml|html|css|csv|tsv|"
       r"jsonl|sql|db|sqlite|age|wav|mp3|mp4|gz|tgz|toml|cfg|ini|log|out|bak")

# Path branch first, so a named path always wins over the bare branch at the same
# offset. The bare branch requires the colon NOT to follow a digit or a letter --
# without that, `12:08:10Z`, `09:44:40Z` and `2026-10-11-0:0:2` are citations.
# The whole `:line` part is optional, so a bare file MENTION is visible and can
# seed the inheritance scope. (Getting this wrong the first time -- making the
# cite group mandatory -- hid every mention and made every bare cite
# unresolvable; the self-test fixture caught it before the tool was used.)
PAT = re.compile(
    r"(?P<path>[A-Za-z0-9_][A-Za-z0-9_.@/+-]*\.(?:%s))"
    r"(?:`?:(?P<line>[0-9]{1,7})(?:\s*[-\u2013,]\s*(?P<end>[0-9]{1,7}))?)?"
    r"|(?<![\dA-Za-z]):(?P<bline>[0-9]{3,7})(?![0-9])" % EXT)


def git(repo, *a):
    return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout


class Store:
    """File text and line counts at one revision, read once per path."""

    def __init__(self, repo, rev):
        self.repo, self.rev = repo, rev
        raw = git(repo, "ls-tree", "-r", "--name-only", rev).decode("utf-8", "replace")
        self.tree = {p for p in raw.split("\n") if p}
        self._text = {}

    def text(self, path):
        if path not in self._text:
            b = subprocess.run(["git", "-C", self.repo, "show", f"{self.rev}:{path}"],
                               capture_output=True).stdout
            self._text[path] = b.decode("utf-8", "replace")
        return self._text[path]

    def nlines(self, path):
        t = self.text(path)
        if t == "":
            return 0
        return t.count("\n") + (0 if t.endswith("\n") else 1)

    def has(self, path, line):
        return path is not None and path in self.tree and 1 <= line <= self.nlines(path)


def blocks(text):
    """Yield (first_line_no, block_text). Paragraphs are the inheritance scope:
    the archive restarts its "the file named here" convention at a blank line."""
    ln, start, buf = 1, 1, []
    for raw in text.split("\n"):
        if raw.strip() == "":
            yield start, "\n".join(buf)
            buf, start = [], ln + 1
        else:
            buf.append(raw)
        ln += 1
    if buf:
        yield start, "\n".join(buf)


def classify_explicit(store, target, want):
    if target not in store.tree:
        return "EXTERNAL_PATH", None
    have = store.nlines(target)
    return ("RESOLVES" if 1 <= want <= have else "STALE_LINE"), have


def classify_bare(store, here, inherited, want):
    """Two candidates, each resolved separately; never collapsed into one verdict."""
    named = store.has(inherited, want)
    self_ok = store.has(here, want)
    external = inherited is not None and inherited not in store.tree
    if named and self_ok and inherited != here:
        v = "AMBIGUOUS"
    elif named:
        v = "AS_NAMED"
    elif self_ok:
        v = "AS_SELF"
    elif external:
        v = "EXTERNAL_NAMED"
    else:
        v = "UNRESOLVED"
    return v, named, self_ok


def scan(repo, rev, corpus, store):
    rows, stats = [], collections.Counter()
    for p in corpus:
        if p not in store.tree:
            stats["corpus_missing"] += 1
            continue
        text = store.text(p)
        for blk_first, blk in blocks(text):
            last = None
            for m in PAT.finditer(blk):
                abs_line = blk_first + blk.count("\n", 0, m.start())
                ctx = blk[max(0, m.start() - 70):m.end() + 70].replace("\n", " ")
                if m.group("path"):
                    last = m.group("path")
                    if m.group("line") is None:            # a mention, no cite
                        stats["mentions"] += 1
                        continue
                    want = int(m.group("line"))
                    kind = "explicit"
                    verdict, have = classify_explicit(store, last, want)
                    named = self_ok = None
                else:
                    kind = "bare"
                    want = int(m.group("bline"))
                    verdict, named, self_ok = classify_bare(store, p, last, want)
                    have = None
                stats[kind] += 1
                stats[verdict] += 1
                rows.append({"file": p, "line": abs_line, "kind": kind,
                             "target": last, "want": want,
                             "end": int(m.group("end")) if m.group("end") else None,
                             "verdict": verdict, "have": have,
                             "named_has": named, "self_has": self_ok,
                             "context": ctx})
    return rows, stats


# --------------------------------------------------------------- self-test
# Each fixture names its own CONTAINING file, because "candidate 2" is a property
# of where the citation is written, not of the text alone. Line counts at HEAD:
# WORK-QUEUE.md 9,856 / checkins/2026-10-15-volt.md 281 / the reply file 83.
QUEUE = "WORK-QUEUE.md"
VOLT = "checkins/2026-10-15-volt.md"
FIXTURES = [
    # (containing file, corpus text, [(kind, verdict, target, want)...])
    (QUEUE, "see checkins/2026-10-15-volt.md:12 for the table",
     [("explicit", "RESOLVES", "checkins/2026-10-15-volt.md", 12)]),
    (QUEUE, "WORK-QUEUE.md:999999 is past the end",
     [("explicit", "STALE_LINE", "WORK-QUEUE.md", 999999)]),
    (QUEUE, "a host path, not this repo: mercury-task.txt:24",
     [("explicit", "EXTERNAL_PATH", "mercury-task.txt", 24)]),
    # the idiom as written: the bare cite takes the file named in its own block,
    # and when that IS the containing file it is not ambiguity
    (QUEUE, "The queue is long.\nWORK-QUEUE.md is the file.\n"
            "Its rows sit at :2264 and a bad one at :99999.",
     [("bare", "AS_NAMED", "WORK-QUEUE.md", 2264),
      ("bare", "UNRESOLVED", "WORK-QUEUE.md", 99999)]),
    # a blank line restarts the scope -> the named-file rule has nothing, and the
    # containing-file candidate is what resolves it. This is the AS_SELF class.
    (QUEUE, "WORK-QUEUE.md at head.\n\nand the same rows at :2264",
     [("bare", "AS_SELF", None, 2264)]),
    # two candidates, both with the line -> the number does not decide
    (QUEUE, "checkins/2026-10-15-volt.md is a file.\nand the queue's own rows at :123",
     [("bare", "AMBIGUOUS", "checkins/2026-10-15-volt.md", 123)]),
    # a named candidate that is not in this tree is out of scope, not a defect
    (QUEUE, "root/crucible/q51/helm-selftest-q51.py is his gate.\n"
            "the queue's row at :99999",
     [("bare", "EXTERNAL_NAMED", "root/crucible/q51/helm-selftest-q51.py", 99999)]),
    # the tool's own worked catch, replayed as a fixture: the real block names
    # tools/queue_claim_check.py (695 lines) and the containing check-in is 281
    # lines, so neither candidate holds the queue's own line numbers
    (VOLT, "the claim check, and now `tools/queue_claim_check.py` run here, leaves "
           "the executable set empty: Q37 `:5278` the pipe's, `:8530` Mercury's",
     [("bare", "UNRESOLVED", "tools/queue_claim_check.py", 5278),
      ("bare", "UNRESOLVED", "tools/queue_claim_check.py", 8530)]),
    # STATED LIMITATION: a bare cite under 3 digits is out of scope by design --
    # at 1-2 digits a line number is indistinguishable from a clock, a ratio or a
    # version, and :08 inside 12:08:10Z is exactly what that floor excludes.
    (QUEUE, "and the queue's own rows at :12", []),
    # a clock is not a citation
    (QUEUE, "read at 12:08:10Z and 09:44:40Z", []),
    # a range is reported with its end line
    (QUEUE, "recoveries/helm-pin-hash-convention.md:85-90",
     [("explicit", "RESOLVES", "recoveries/helm-pin-hash-convention.md", 85)]),
]


def _expect(here, text, store):
    got = []
    for _first, blk in blocks(text):
        last = None
        for m in PAT.finditer(blk):
            if m.group("path"):
                last = m.group("path")
                if m.group("line") is None:
                    continue
                v, _ = classify_explicit(store, last, int(m.group("line")))
                got.append(("explicit", v, last, int(m.group("line"))))
            else:
                want = int(m.group("bline"))
                v, _n, _s = classify_bare(store, here, last, want)
                got.append(("bare", v, last, want))
    return got


def selftest(repo):
    store = Store(repo, "HEAD")
    n = bad = 0
    for here, text, want in FIXTURES:
        got = _expect(here, text, store)
        n += 1
        ok = got == want
        bad += 0 if ok else 1
        print(f"{'PASS' if ok else 'FAIL'} in {here[:26]}: {text[:44]!r}")
        if not ok:
            print(f"      got  {got}")
            print(f"      want {want}")
    print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed")
    return 0 if bad == 0 else 1


def main():
    ap = argparse.ArgumentParser(description=__doc__,
                                 formatter_class=argparse.RawDescriptionHelpFormatter)
    ap.add_argument("--repo", default=DEFAULT_REPO)
    ap.add_argument("--rev", default="HEAD")
    ap.add_argument("--corpus", nargs="*")
    ap.add_argument("--json")
    ap.add_argument("--only", choices=["stale", "asself", "unresolved", "ambiguous",
                                       "external", "externalnamed", "named", "all"],
                    default="stale")
    ap.add_argument("--context", action="store_true")
    ap.add_argument("--selftest", action="store_true")
    a = ap.parse_args()

    if a.selftest:
        return selftest(a.repo)

    store = Store(a.repo, a.rev)
    corpus = a.corpus or sorted(p for p in store.tree if p.endswith(".md"))
    rows, stats = scan(a.repo, a.rev, corpus, store)

    print(f"corpus: {len(corpus)} markdown files at {a.rev} in {a.repo}")
    print("--- EXPLICIT `<path>:<line>` -- a CLAIM (unit: citations) ---")
    print(f"  citations: {stats['explicit']}")
    for v in ("RESOLVES", "STALE_LINE", "EXTERNAL_PATH"):
        print(f"  {v:15s} {stats[v]}")
    print("--- BARE `:<line>` -- a CANDIDATE, never a claim (unit: citations) ---")
    print(f"  citations: {stats['bare']}")
    for v in ("AS_NAMED", "AS_SELF", "AMBIGUOUS", "EXTERNAL_NAMED", "UNRESOLVED"):
        print(f"  {v:15s} {stats[v]}")
    print(f"  ({stats['AS_SELF']} of the bare cites resolve ONLY as the file they "
          f"are written in -- the named-file rule alone would have called them "
          f"defects; {stats['UNRESOLVED']} resolve as neither, so no reader can "
          f"check them without inferring the referent)")
    print(f"file mentions with no line: {stats['mentions']} (unit: occurrences) "
          f"-- these seed the inheritance scope")
    by_target = collections.Counter(r["target"] for r in rows
                                    if r["kind"] == "explicit" and r["target"])
    print(f"distinct explicitly cited targets: {len(by_target)} (unit: distinct paths)")
    for t, c in by_target.most_common(8):
        print(f"  {c:6d}  {t}"
              f"{'' if t in store.tree else '   [not in tree at ' + a.rev + ']'}")

    sel = {"stale": lambda r: r["kind"] == "explicit" and r["verdict"] == "STALE_LINE",
           "external": lambda r: r["verdict"] == "EXTERNAL_PATH",
           "externalnamed": lambda r: r["verdict"] == "EXTERNAL_NAMED",
           "asself": lambda r: r["verdict"] == "AS_SELF",
           "unresolved": lambda r: r["verdict"] == "UNRESOLVED",
           "ambiguous": lambda r: r["verdict"] == "AMBIGUOUS",
           "named": lambda r: r["verdict"] == "AS_NAMED",
           "all": lambda r: True}[a.only]
    show = [r for r in rows if sel(r)]
    print(f"--- {a.only}: {len(show)} citations (showing first 300) ---")
    for r in show[:300]:
        if r["kind"] == "bare":
            extra = (f"  [named={r['target'] or '-'}:"
                     f"{'y' if r['named_has'] else 'n'} self="
                     f"{'y' if r['self_has'] else 'n'}]")
        else:
            extra = f"  [have={r['have']}]"
        print(f"  {r['file']}:{r['line']}  {r['kind']:8s} -> "
              f"{r['target'] or '-'}:{r['want']}  {r['verdict']}{extra}")
        if a.context:
            print(f"      ...{r['context']}...")
    if a.json:
        with open(a.json, "w") as fh:
            json.dump({"rev": a.rev, "repo": a.repo, "corpus": corpus,
                       "stats": dict(stats), "citations": rows}, fh, indent=1)
        print(f"wrote {a.json}")
    return 0


if __name__ == "__main__":
    sys.exit(main())