Files
troupe-standup/tools/citation_resolve_check.py
T
timmy 016d26eae7 checkin 2026-10-17 (volt): six tools -- unreadable input no longer reads as clean
Five tracebacked on input they could not read (rc 1, the code this corpus uses for
"a finding"); two -- citation_hash_class.py, citation_resolve_check.py -- printed a
clean zero census over a missing repo or bogus --rev and exited 0, because git() in
both discards the return code. Those two are the gates a reviewer runs to decide
whether a citation defect exists, so a green that never looked is the hazard.

queue_index.py / queue_claim_check.py: read errors -> rc 3 (the claim tool's docstring
already promised `3 queue unreadable` while its code raised UnicodeDecodeError).
citation_hash_class.py / citation_resolve_check.py: precondition(repo, rev) -> rc 2,
matching doc_census_check.py and landing_check.py. check_lint.py / census_row_delta.py:
named message, no traceback, rc 1 kept (filed: usage error and graded fault still
share that code).

175 insertions(+), 14 deletions(-). Selftests 54/54, 16/16, 37/37, 33/33, 19/19, 11/11;
sweep 11 tools/10 OK/0 fatal; --check rc 0 at 9 nested ids; 9 broken-input probes ->
0 tracebacks; copy-vs-copy A/B 7 SAME / 2 DIFFERS (both intended added cells).
run.md's earlier "7 SAME, 0 DIFFERS" claim was wrong and is corrected in the artifact.
2026-10-17 19:58:45 +00:00

429 lines
21 KiB
Python

#!/usr/bin/env python3
"""citation_resolve_check.py -- does every `<file>:<line>` citation in this
archive's prose actually resolve, and to which file?
WHY THIS EXISTS
This archive cites itself by line number. Its most-recurring review defect is
not a wrong count and not a wrong hash family -- it is a citation that does not
point where the sentence says it points. Logged in the ledger's own words:
WORK-QUEUE.md:8586 "the identifiers printed for Zephyrus's newest file are
the 11th's ... Filename and *Blocked on* text were right,
the identifiers wrong." Third instance of this class in
one week (reply 51 §3 -> :8467; reply 53 §0).
WORK-QUEUE.md:8003 cites to :2264/:2684/:6847 "read back and checked by hand",
because nothing checked them automatically.
WORK-QUEUE.md:8009 "The hygiene count confirmed at two revisions, not one"
-- the same read repeated by hand for want of a tool.
citation_hash_class.py settles a token's FAMILY. queue_claim_check.py settles
an item's CLOSURE. Neither settles whether a `path:line` lands on the line the
citation is offered as evidence for. That is the third axis, and this is it.
WHAT IT MEASURES (read-only; the only write path is --json)
Two citation forms, in one left-to-right pass per file, paragraph by paragraph.
EXPLICIT `<path>:<line>`, and `<path>:<line>-<line>` for a range. The file is
named AT the citation, so the verdict is a CLAIM:
RESOLVES path in the tree at --rev, 1 <= line <= line_count
STALE_LINE path in the tree, line > line_count -- the citation
points past the end of the file it names
EXTERNAL_PATH path not in this tree at --rev (a host path under
/root, or another repository's file). OUT OF SCOPE
and counted separately: the citation may be
perfectly true, this tool cannot read it, and saying
so is the honest verdict.
BARE `:<line>`, the troupe's inheritance idiom -- "the file last named".
This is NOT a claim and this tool refuses to make it one. It was
written to resolve the inherited file, and the first run FALSIFIED
that rule at scale (see below). Two candidates are offered for every
bare cite and each is resolved separately:
candidate 1 the nearest preceding path NAMED in the same
blank-line-delimited block (the idiom as written)
candidate 2 the file the citation is written IN
AS_NAMED only candidate 1 has that line
AS_SELF only candidate 2 has that line -- the named-file
rule mis-binds and the citation is still resolvable
AMBIGUOUS both have it, so the number does not decide
EXTERNAL_NAMED candidate 1 is not in this tree at all (a host
artifact); out of scope, same class as
EXTERNAL_PATH -- and ALSO the shape a mis-bind
takes when the last-named path is a host file
UNRESOLVED no candidate in this tree has the line and nothing
external was named: no outside reader can resolve
this citation at --rev. This is the defect claim.
Both candidates and both resolutions are printed for every row, so a
reader never has to trust the classifier.
WHY BARE CITES ARE NOT RESOLVED BY THE NAMED-FILE RULE -- measured, not argued
The rule was implemented as a verdict first, and the run falsified it. Three of
the largest inherited-"stale" groups were mis-binds, each verified by reading
the cited text at HEAD:
WORK-QUEUE.md:9804 ":9622's own code span" -- 9622 is a WORK-QUEUE.md line
(the queue row describing the tool); the rule had it bound
to tools/queue_claim_check.py, named 40 chars earlier.
WORK-QUEUE.md:8023 the hygiene table's `:6348` column -- WORK-QUEUE.md lines
in a row whose LAST column names the artifact the row is
ABOUT. A table row names a file for a different purpose
than the one the line number belongs to.
checkins/2026-10-15-merlin.md:739 "`:2264` / `:2684` / `:6847`" -- again
WORK-QUEUE.md lines, this time bound to the host file
root/crucible/q51/helm-selftest-q51.py named in an
earlier sentence of the same block (EXTERNAL_NAMED).
So "nearest named file" is a candidate; candidate 2 is what rescues the
residue. AS_SELF is the count of citations the named-file rule alone would have
reported as defects and that in fact resolve against the file they are in.
THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
EXPLICIT 410 citations: 113 RESOLVES / 0 STALE_LINE / 297 EXTERNAL_PATH.
Every explicit citation whose file is in this repo resolves.
BARE 1612 citations: 119 AS_NAMED / 716 AS_SELF / 51 AMBIGUOUS /
255 EXTERNAL_NAMED / 471 UNRESOLVED.
Only 119 of 1612 bare cites are resolvable by the file-naming
idiom the archive documents; 471 cannot be resolved by any reader
at HEAD without inferring the referent from context. That is the
number this tool exists to make checkable, and it is the honest
form of the ledger's own "a token quoted without its class".
mentions 6961 file mentions carrying no line -- these seed the scope.
Read every figure as a citation count, never as a defect count: 471 UNRESOLVED
is not 471 errors. The tool's own catch is the worked example below.
WORKED CATCH -- this tool's first finding is one of MY OWN artifacts
checkins/2026-10-15-volt.md:9 reads "the three open items belong to other lanes
(Q37 `:5278` the pipe's, `:8530` Mercury's, the career-lane apply queue `:7303`
Alexander's)". Read at HEAD: that file is 281 lines, so neither candidate has
line 5278; the block's last-named path is tools/queue_claim_check.py, which is
695 lines. Verdict UNRESOLVED, and it is correct -- the three numbers are
WORK-QUEUE.md lines (5278 is Q37's Status line, 8530 is Q53's, 7303 is the Q53
opening), and the citation does not say so. Any troupe reader infers it; a
stranger cannot check it. Reproduce with:
python3 citation_resolve_check.py --only unresolved --context | grep volt.md
NOT in scope, stated rather than implied: whether the cited LINE contains the
evidence the sentence claims. That needs the claim, not just the pointer. And a
citation REPRODUCED INSIDE QUOTES is still a citation to this scanner, which
does not parse quotation marks -- so a document that reports a defective
citation verbatim appears to contain it. Measured instance: the landing
check-in for this tool reproduces the defective citations it is reporting,
verbatim, and the scanner counts every one of them.
USAGE
python3 citation_resolve_check.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
[--json OUT] [--only stale|asself|unresolved|ambiguous|external|externalnamed|named|all]
[--context] [--selftest] # 11 fixtures
"""
import argparse
import collections
import json
import re
import subprocess
import sys
DEFAULT_REPO = "/root/troupe-standup"
# The extensions this archive actually cites by line. Deliberately a closed list:
# an open `\.[a-z]+` would swallow clock times and domain names.
EXT = (r"md|markdown|py|json|txt|js|mjs|cjs|sh|bash|yml|yaml|html|css|csv|tsv|"
r"jsonl|sql|db|sqlite|age|wav|mp3|mp4|gz|tgz|toml|cfg|ini|log|out|bak")
# Path branch first, so a named path always wins over the bare branch at the same
# offset. The bare branch requires the colon NOT to follow a digit or a letter --
# without that, `12:08:10Z`, `09:44:40Z` and `2026-10-11-0:0:2` are citations.
# The whole `:line` part is optional, so a bare file MENTION is visible and can
# seed the inheritance scope. (Getting this wrong the first time -- making the
# cite group mandatory -- hid every mention and made every bare cite
# unresolvable; the self-test fixture caught it before the tool was used.)
PAT = re.compile(
r"(?P<path>[A-Za-z0-9_][A-Za-z0-9_.@/+-]*\.(?:%s))"
r"(?:`?:(?P<line>[0-9]{1,7})(?:\s*[-\u2013,]\s*(?P<end>[0-9]{1,7}))?)?"
r"|(?<![\dA-Za-z]):(?P<bline>[0-9]{3,7})(?![0-9])" % EXT)
def git(repo, *a):
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
def precondition(repo, rev):
"""None when `repo` is a git work tree that carries `rev`, else why not.
Every reader below swallows git's exit status, so an unreadable repo or a
mistyped revision came back as an empty tree and the census printed
"citations: 0 / stale: 0" with rc 0 -- a green gate over an input it never
saw. The caller turns this into exit 2.
"""
p = subprocess.run(["git", "-C", repo, "rev-parse", "--verify", rev + "^{commit}"],
capture_output=True)
if p.returncode == 0:
return None
why = p.stderr.decode("utf-8", "replace").strip().splitlines()
return "cannot read %s at %s: %s" % (repo, rev, why[-1] if why else "git rev-parse failed")
class Store:
"""File text and line counts at one revision, read once per path."""
def __init__(self, repo, rev):
self.repo, self.rev = repo, rev
raw = git(repo, "ls-tree", "-r", "--name-only", rev).decode("utf-8", "replace")
self.tree = {p for p in raw.split("\n") if p}
self._text = {}
def text(self, path):
if path not in self._text:
b = subprocess.run(["git", "-C", self.repo, "show", f"{self.rev}:{path}"],
capture_output=True).stdout
self._text[path] = b.decode("utf-8", "replace")
return self._text[path]
def nlines(self, path):
t = self.text(path)
if t == "":
return 0
return t.count("\n") + (0 if t.endswith("\n") else 1)
def has(self, path, line):
return path is not None and path in self.tree and 1 <= line <= self.nlines(path)
def blocks(text):
"""Yield (first_line_no, block_text). Paragraphs are the inheritance scope:
the archive restarts its "the file named here" convention at a blank line."""
ln, start, buf = 1, 1, []
for raw in text.split("\n"):
if raw.strip() == "":
yield start, "\n".join(buf)
buf, start = [], ln + 1
else:
buf.append(raw)
ln += 1
if buf:
yield start, "\n".join(buf)
def classify_explicit(store, target, want):
if target not in store.tree:
return "EXTERNAL_PATH", None
have = store.nlines(target)
return ("RESOLVES" if 1 <= want <= have else "STALE_LINE"), have
def classify_bare(store, here, inherited, want):
"""Two candidates, each resolved separately; never collapsed into one verdict."""
named = store.has(inherited, want)
self_ok = store.has(here, want)
external = inherited is not None and inherited not in store.tree
if named and self_ok and inherited != here:
v = "AMBIGUOUS"
elif named:
v = "AS_NAMED"
elif self_ok:
v = "AS_SELF"
elif external:
v = "EXTERNAL_NAMED"
else:
v = "UNRESOLVED"
return v, named, self_ok
def scan(repo, rev, corpus, store):
rows, stats = [], collections.Counter()
for p in corpus:
if p not in store.tree:
stats["corpus_missing"] += 1
continue
text = store.text(p)
for blk_first, blk in blocks(text):
last = None
for m in PAT.finditer(blk):
abs_line = blk_first + blk.count("\n", 0, m.start())
ctx = blk[max(0, m.start() - 70):m.end() + 70].replace("\n", " ")
if m.group("path"):
last = m.group("path")
if m.group("line") is None: # a mention, no cite
stats["mentions"] += 1
continue
want = int(m.group("line"))
kind = "explicit"
verdict, have = classify_explicit(store, last, want)
named = self_ok = None
else:
kind = "bare"
want = int(m.group("bline"))
verdict, named, self_ok = classify_bare(store, p, last, want)
have = None
stats[kind] += 1
stats[verdict] += 1
rows.append({"file": p, "line": abs_line, "kind": kind,
"target": last, "want": want,
"end": int(m.group("end")) if m.group("end") else None,
"verdict": verdict, "have": have,
"named_has": named, "self_has": self_ok,
"context": ctx})
return rows, stats
# --------------------------------------------------------------- self-test
# Each fixture names its own CONTAINING file, because "candidate 2" is a property
# of where the citation is written, not of the text alone. Line counts at HEAD:
# WORK-QUEUE.md 9,856 / checkins/2026-10-15-volt.md 281 / the reply file 83.
QUEUE = "WORK-QUEUE.md"
VOLT = "checkins/2026-10-15-volt.md"
FIXTURES = [
# (containing file, corpus text, [(kind, verdict, target, want)...])
(QUEUE, "see checkins/2026-10-15-volt.md:12 for the table",
[("explicit", "RESOLVES", "checkins/2026-10-15-volt.md", 12)]),
(QUEUE, "WORK-QUEUE.md:999999 is past the end",
[("explicit", "STALE_LINE", "WORK-QUEUE.md", 999999)]),
(QUEUE, "a host path, not this repo: mercury-task.txt:24",
[("explicit", "EXTERNAL_PATH", "mercury-task.txt", 24)]),
# the idiom as written: the bare cite takes the file named in its own block,
# and when that IS the containing file it is not ambiguity
(QUEUE, "The queue is long.\nWORK-QUEUE.md is the file.\n"
"Its rows sit at :2264 and a bad one at :99999.",
[("bare", "AS_NAMED", "WORK-QUEUE.md", 2264),
("bare", "UNRESOLVED", "WORK-QUEUE.md", 99999)]),
# a blank line restarts the scope -> the named-file rule has nothing, and the
# containing-file candidate is what resolves it. This is the AS_SELF class.
(QUEUE, "WORK-QUEUE.md at head.\n\nand the same rows at :2264",
[("bare", "AS_SELF", None, 2264)]),
# two candidates, both with the line -> the number does not decide
(QUEUE, "checkins/2026-10-15-volt.md is a file.\nand the queue's own rows at :123",
[("bare", "AMBIGUOUS", "checkins/2026-10-15-volt.md", 123)]),
# a named candidate that is not in this tree is out of scope, not a defect
(QUEUE, "root/crucible/q51/helm-selftest-q51.py is his gate.\n"
"the queue's row at :99999",
[("bare", "EXTERNAL_NAMED", "root/crucible/q51/helm-selftest-q51.py", 99999)]),
# the tool's own worked catch, replayed as a fixture: the real block names
# tools/queue_claim_check.py (695 lines) and the containing check-in is 281
# lines, so neither candidate holds the queue's own line numbers
(VOLT, "the claim check, and now `tools/queue_claim_check.py` run here, leaves "
"the executable set empty: Q37 `:5278` the pipe's, `:8530` Mercury's",
[("bare", "UNRESOLVED", "tools/queue_claim_check.py", 5278),
("bare", "UNRESOLVED", "tools/queue_claim_check.py", 8530)]),
# STATED LIMITATION: a bare cite under 3 digits is out of scope by design --
# at 1-2 digits a line number is indistinguishable from a clock, a ratio or a
# version, and :08 inside 12:08:10Z is exactly what that floor excludes.
(QUEUE, "and the queue's own rows at :12", []),
# a clock is not a citation
(QUEUE, "read at 12:08:10Z and 09:44:40Z", []),
# a range is reported with its end line
(QUEUE, "recoveries/helm-pin-hash-convention.md:85-90",
[("explicit", "RESOLVES", "recoveries/helm-pin-hash-convention.md", 85)]),
]
def _expect(here, text, store):
got = []
for _first, blk in blocks(text):
last = None
for m in PAT.finditer(blk):
if m.group("path"):
last = m.group("path")
if m.group("line") is None:
continue
v, _ = classify_explicit(store, last, int(m.group("line")))
got.append(("explicit", v, last, int(m.group("line"))))
else:
want = int(m.group("bline"))
v, _n, _s = classify_bare(store, here, last, want)
got.append(("bare", v, last, want))
return got
def selftest(repo):
store = Store(repo, "HEAD")
n = bad = 0
for here, text, want in FIXTURES:
got = _expect(here, text, store)
n += 1
ok = got == want
bad += 0 if ok else 1
print(f"{'PASS' if ok else 'FAIL'} in {here[:26]}: {text[:44]!r}")
if not ok:
print(f" got {got}")
print(f" want {want}")
print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed")
return 0 if bad == 0 else 1
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--repo", default=DEFAULT_REPO)
ap.add_argument("--rev", default="HEAD")
ap.add_argument("--corpus", nargs="*")
ap.add_argument("--json")
ap.add_argument("--only", choices=["stale", "asself", "unresolved", "ambiguous",
"external", "externalnamed", "named", "all"],
default="stale")
ap.add_argument("--context", action="store_true")
ap.add_argument("--selftest", action="store_true")
a = ap.parse_args()
if a.selftest:
return selftest(a.repo)
problem = precondition(a.repo, a.rev)
if problem:
print(problem, file=sys.stderr)
return 2
store = Store(a.repo, a.rev)
corpus = a.corpus or sorted(p for p in store.tree if p.endswith(".md"))
rows, stats = scan(a.repo, a.rev, corpus, store)
print(f"corpus: {len(corpus)} markdown files at {a.rev} in {a.repo}")
print("--- EXPLICIT `<path>:<line>` -- a CLAIM (unit: citations) ---")
print(f" citations: {stats['explicit']}")
for v in ("RESOLVES", "STALE_LINE", "EXTERNAL_PATH"):
print(f" {v:15s} {stats[v]}")
print("--- BARE `:<line>` -- a CANDIDATE, never a claim (unit: citations) ---")
print(f" citations: {stats['bare']}")
for v in ("AS_NAMED", "AS_SELF", "AMBIGUOUS", "EXTERNAL_NAMED", "UNRESOLVED"):
print(f" {v:15s} {stats[v]}")
print(f" ({stats['AS_SELF']} of the bare cites resolve ONLY as the file they "
f"are written in -- the named-file rule alone would have called them "
f"defects; {stats['UNRESOLVED']} resolve as neither, so no reader can "
f"check them without inferring the referent)")
print(f"file mentions with no line: {stats['mentions']} (unit: occurrences) "
f"-- these seed the inheritance scope")
by_target = collections.Counter(r["target"] for r in rows
if r["kind"] == "explicit" and r["target"])
print(f"distinct explicitly cited targets: {len(by_target)} (unit: distinct paths)")
for t, c in by_target.most_common(8):
print(f" {c:6d} {t}"
f"{'' if t in store.tree else ' [not in tree at ' + a.rev + ']'}")
sel = {"stale": lambda r: r["kind"] == "explicit" and r["verdict"] == "STALE_LINE",
"external": lambda r: r["verdict"] == "EXTERNAL_PATH",
"externalnamed": lambda r: r["verdict"] == "EXTERNAL_NAMED",
"asself": lambda r: r["verdict"] == "AS_SELF",
"unresolved": lambda r: r["verdict"] == "UNRESOLVED",
"ambiguous": lambda r: r["verdict"] == "AMBIGUOUS",
"named": lambda r: r["verdict"] == "AS_NAMED",
"all": lambda r: True}[a.only]
show = [r for r in rows if sel(r)]
print(f"--- {a.only}: {len(show)} citations (showing first 300) ---")
for r in show[:300]:
if r["kind"] == "bare":
extra = (f" [named={r['target'] or '-'}:"
f"{'y' if r['named_has'] else 'n'} self="
f"{'y' if r['self_has'] else 'n'}]")
else:
extra = f" [have={r['have']}]"
print(f" {r['file']}:{r['line']} {r['kind']:8s} -> "
f"{r['target'] or '-'}:{r['want']} {r['verdict']}{extra}")
if a.context:
print(f" ...{r['context']}...")
if a.json:
with open(a.json, "w") as fh:
json.dump({"rev": a.rev, "repo": a.repo, "corpus": corpus,
"stats": dict(stats), "citations": rows}, fh, indent=1)
print(f"wrote {a.json}")
return 0
if __name__ == "__main__":
sys.exit(main())