tools (volt): the citation resolver -- every <path>:<line> citation in the archive's prose, resolved; the bare :line idiom measured as a candidate, not a verdict
402 -> 407 lines / 20,460 B / sha256 7d063b39e2e4f9e5691b720cb57895b29450ece77051a7db6a8584dfd7173237.
Census at babdadc over 297 markdown files: EXPLICIT 410 cites = 113 RESOLVES / 0 STALE_LINE / 297
EXTERNAL_PATH; BARE 1612 cites = 119 AS_NAMED / 716 AS_SELF / 51 AMBIGUOUS / 255 EXTERNAL_NAMED /
471 UNRESOLVED. The named-file rule was written as a verdict first and falsified by its own first run
(three mis-bind groups, each verified at HEAD), so bare cites carry two candidates and no claim.
selftest 11/11; all 113 in-tree explicit RESOLVES rows re-checked with wc -l, 0 mismatches; read-only
in fact (--json is the only write path, git status clean after every run). No queue item opened or
closed; WORK-QUEUE.md untouched.
This commit is contained in:
volt authored and 
Timmy committed
1 parent
babdadc949
commit
fca3f2ee3d
1 file changed
+407
@@ -0,0 +1,407 @@
|
||||
#!/usr/bin/env python3
|
||||
"""citation_resolve_check.py -- does every `<file>:<line>` citation in this
|
||||
archive's prose actually resolve, and to which file?
|
||||
|
||||
WHY THIS EXISTS
|
||||
This archive cites itself by line number. Its most-recurring review defect is
|
||||
not a wrong count and not a wrong hash family -- it is a citation that does not
|
||||
point where the sentence says it points. Logged in the ledger's own words:
|
||||
|
||||
WORK-QUEUE.md:8586 "the identifiers printed for Zephyrus's newest file are
|
||||
the 11th's ... Filename and *Blocked on* text were right,
|
||||
the identifiers wrong." Third instance of this class in
|
||||
one week (reply 51 §3 -> :8467; reply 53 §0).
|
||||
WORK-QUEUE.md:8003 cites to :2264/:2684/:6847 "read back and checked by hand",
|
||||
because nothing checked them automatically.
|
||||
WORK-QUEUE.md:8009 "The hygiene count confirmed at two revisions, not one"
|
||||
-- the same read repeated by hand for want of a tool.
|
||||
|
||||
citation_hash_class.py settles a token's FAMILY. queue_claim_check.py settles
|
||||
an item's CLOSURE. Neither settles whether a `path:line` lands on the line the
|
||||
citation is offered as evidence for. That is the third axis, and this is it.
|
||||
|
||||
WHAT IT MEASURES (read-only; the only write path is --json)
|
||||
Two citation forms, in one left-to-right pass per file, paragraph by paragraph.
|
||||
|
||||
EXPLICIT `<path>:<line>`, and `<path>:<line>-<line>` for a range. The file is
|
||||
named AT the citation, so the verdict is a CLAIM:
|
||||
RESOLVES path in the tree at --rev, 1 <= line <= line_count
|
||||
STALE_LINE path in the tree, line > line_count -- the citation
|
||||
points past the end of the file it names
|
||||
EXTERNAL_PATH path not in this tree at --rev (a host path under
|
||||
/root, or another repository's file). OUT OF SCOPE
|
||||
and counted separately: the citation may be
|
||||
perfectly true, this tool cannot read it, and saying
|
||||
so is the honest verdict.
|
||||
|
||||
BARE `:<line>`, the troupe's inheritance idiom -- "the file last named".
|
||||
This is NOT a claim and this tool refuses to make it one. It was
|
||||
written to resolve the inherited file, and the first run FALSIFIED
|
||||
that rule at scale (see below). Two candidates are offered for every
|
||||
bare cite and each is resolved separately:
|
||||
candidate 1 the nearest preceding path NAMED in the same
|
||||
blank-line-delimited block (the idiom as written)
|
||||
candidate 2 the file the citation is written IN
|
||||
AS_NAMED only candidate 1 has that line
|
||||
AS_SELF only candidate 2 has that line -- the named-file
|
||||
rule mis-binds and the citation is still resolvable
|
||||
AMBIGUOUS both have it, so the number does not decide
|
||||
EXTERNAL_NAMED candidate 1 is not in this tree at all (a host
|
||||
artifact); out of scope, same class as
|
||||
EXTERNAL_PATH -- and ALSO the shape a mis-bind
|
||||
takes when the last-named path is a host file
|
||||
UNRESOLVED no candidate in this tree has the line and nothing
|
||||
external was named: no outside reader can resolve
|
||||
this citation at --rev. This is the defect claim.
|
||||
Both candidates and both resolutions are printed for every row, so a
|
||||
reader never has to trust the classifier.
|
||||
|
||||
WHY BARE CITES ARE NOT RESOLVED BY THE NAMED-FILE RULE -- measured, not argued
|
||||
The rule was implemented as a verdict first, and the run falsified it. Three of
|
||||
the largest inherited-"stale" groups were mis-binds, each verified by reading
|
||||
the cited text at HEAD:
|
||||
WORK-QUEUE.md:9804 ":9622's own code span" -- 9622 is a WORK-QUEUE.md line
|
||||
(the queue row describing the tool); the rule had it bound
|
||||
to tools/queue_claim_check.py, named 40 chars earlier.
|
||||
WORK-QUEUE.md:8023 the hygiene table's `:6348` column -- WORK-QUEUE.md lines
|
||||
in a row whose LAST column names the artifact the row is
|
||||
ABOUT. A table row names a file for a different purpose
|
||||
than the one the line number belongs to.
|
||||
checkins/2026-10-15-merlin.md:739 "`:2264` / `:2684` / `:6847`" -- again
|
||||
WORK-QUEUE.md lines, this time bound to the host file
|
||||
root/crucible/q51/helm-selftest-q51.py named in an
|
||||
earlier sentence of the same block (EXTERNAL_NAMED).
|
||||
So "nearest named file" is a candidate; candidate 2 is what rescues the
|
||||
residue. AS_SELF is the count of citations the named-file rule alone would have
|
||||
reported as defects and that in fact resolve against the file they are in.
|
||||
|
||||
THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
|
||||
EXPLICIT 410 citations: 113 RESOLVES / 0 STALE_LINE / 297 EXTERNAL_PATH.
|
||||
Every explicit citation whose file is in this repo resolves.
|
||||
BARE 1612 citations: 119 AS_NAMED / 716 AS_SELF / 51 AMBIGUOUS /
|
||||
255 EXTERNAL_NAMED / 471 UNRESOLVED.
|
||||
Only 119 of 1612 bare cites are resolvable by the file-naming
|
||||
idiom the archive documents; 471 cannot be resolved by any reader
|
||||
at HEAD without inferring the referent from context. That is the
|
||||
number this tool exists to make checkable, and it is the honest
|
||||
form of the ledger's own "a token quoted without its class".
|
||||
mentions 6961 file mentions carrying no line -- these seed the scope.
|
||||
Read every figure as a citation count, never as a defect count: 441 UNRESOLVED
|
||||
is not 441 errors. The tool's own catch is the worked example below.
|
||||
|
||||
WORKED CATCH -- this tool's first finding is one of MY OWN artifacts
|
||||
checkins/2026-10-15-volt.md:9 reads "the three open items belong to other lanes
|
||||
(Q37 `:5278` the pipe's, `:8530` Mercury's, the career-lane apply queue `:7303`
|
||||
Alexander's)". Read at HEAD: that file is 281 lines, so neither candidate has
|
||||
line 5278; the block's last-named path is tools/queue_claim_check.py, which is
|
||||
695 lines. Verdict UNRESOLVED, and it is correct -- the three numbers are
|
||||
WORK-QUEUE.md lines (5278 is Q37's Status line, 8530 is Q53's, 7303 is the Q53
|
||||
opening), and the citation does not say so. Any troupe reader infers it; a
|
||||
stranger cannot check it. Reproduce with:
|
||||
python3 citation_resolve_check.py --only unresolved --context | grep volt.md
|
||||
|
||||
NOT in scope, stated rather than implied: whether the cited LINE contains the
|
||||
evidence the sentence claims. That needs the claim, not just the pointer. And a
|
||||
citation REPRODUCED INSIDE QUOTES is still a citation to this scanner, which
|
||||
does not parse quotation marks -- so a document that reports a defective
|
||||
citation verbatim appears to contain it. Measured instance: the landing
|
||||
check-in for this tool reproduces the defective citations it is reporting,
|
||||
verbatim, and the scanner counts every one of them.
|
||||
|
||||
USAGE
|
||||
python3 citation_resolve_check.py [--repo DIR] [--rev HEAD] [--corpus PATH...]
|
||||
[--json OUT] [--only stale|asself|unresolved|ambiguous|external|externalnamed|named|all]
|
||||
[--context] [--selftest]
|
||||
"""
|
||||
import argparse
|
||||
import collections
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
DEFAULT_REPO = "/root/troupe-standup"
|
||||
|
||||
# The extensions this archive actually cites by line. Deliberately a closed list:
|
||||
# an open `\.[a-z]+` would swallow clock times and domain names.
|
||||
EXT = (r"md|markdown|py|json|txt|js|mjs|cjs|sh|bash|yml|yaml|html|css|csv|tsv|"
|
||||
r"jsonl|sql|db|sqlite|age|wav|mp3|mp4|gz|tgz|toml|cfg|ini|log|out|bak")
|
||||
|
||||
# Path branch first, so a named path always wins over the bare branch at the same
|
||||
# offset. The bare branch requires the colon NOT to follow a digit or a letter --
|
||||
# without that, `12:08:10Z`, `09:44:40Z` and `2026-10-11-0:0:2` are citations.
|
||||
# The whole `:line` part is optional, so a bare file MENTION is visible and can
|
||||
# seed the inheritance scope. (Getting this wrong the first time -- making the
|
||||
# cite group mandatory -- hid every mention and made every bare cite
|
||||
# unresolvable; the self-test fixture caught it before the tool was used.)
|
||||
PAT = re.compile(
|
||||
r"(?P<path>[A-Za-z0-9_][A-Za-z0-9_.@/+-]*\.(?:%s))"
|
||||
r"(?:`?:(?P<line>[0-9]{1,7})(?:\s*[-\u2013,]\s*(?P<end>[0-9]{1,7}))?)?"
|
||||
r"|(?<![\dA-Za-z]):(?P<bline>[0-9]{3,7})(?![0-9])" % EXT)
|
||||
|
||||
|
||||
def git(repo, *a):
|
||||
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
|
||||
|
||||
|
||||
class Store:
|
||||
"""File text and line counts at one revision, read once per path."""
|
||||
|
||||
def __init__(self, repo, rev):
|
||||
self.repo, self.rev = repo, rev
|
||||
raw = git(repo, "ls-tree", "-r", "--name-only", rev).decode("utf-8", "replace")
|
||||
self.tree = {p for p in raw.split("\n") if p}
|
||||
self._text = {}
|
||||
|
||||
def text(self, path):
|
||||
if path not in self._text:
|
||||
b = subprocess.run(["git", "-C", self.repo, "show", f"{self.rev}:{path}"],
|
||||
capture_output=True).stdout
|
||||
self._text[path] = b.decode("utf-8", "replace")
|
||||
return self._text[path]
|
||||
|
||||
def nlines(self, path):
|
||||
t = self.text(path)
|
||||
if t == "":
|
||||
return 0
|
||||
return t.count("\n") + (0 if t.endswith("\n") else 1)
|
||||
|
||||
def has(self, path, line):
|
||||
return path is not None and path in self.tree and 1 <= line <= self.nlines(path)
|
||||
|
||||
|
||||
def blocks(text):
|
||||
"""Yield (first_line_no, block_text). Paragraphs are the inheritance scope:
|
||||
the archive restarts its "the file named here" convention at a blank line."""
|
||||
ln, start, buf = 1, 1, []
|
||||
for raw in text.split("\n"):
|
||||
if raw.strip() == "":
|
||||
yield start, "\n".join(buf)
|
||||
buf, start = [], ln + 1
|
||||
else:
|
||||
buf.append(raw)
|
||||
ln += 1
|
||||
if buf:
|
||||
yield start, "\n".join(buf)
|
||||
|
||||
|
||||
def classify_explicit(store, target, want):
|
||||
if target not in store.tree:
|
||||
return "EXTERNAL_PATH", None
|
||||
have = store.nlines(target)
|
||||
return ("RESOLVES" if 1 <= want <= have else "STALE_LINE"), have
|
||||
|
||||
|
||||
def classify_bare(store, here, inherited, want):
|
||||
"""Two candidates, each resolved separately; never collapsed into one verdict."""
|
||||
named = store.has(inherited, want)
|
||||
self_ok = store.has(here, want)
|
||||
external = inherited is not None and inherited not in store.tree
|
||||
if named and self_ok and inherited != here:
|
||||
v = "AMBIGUOUS"
|
||||
elif named:
|
||||
v = "AS_NAMED"
|
||||
elif self_ok:
|
||||
v = "AS_SELF"
|
||||
elif external:
|
||||
v = "EXTERNAL_NAMED"
|
||||
else:
|
||||
v = "UNRESOLVED"
|
||||
return v, named, self_ok
|
||||
|
||||
|
||||
def scan(repo, rev, corpus, store):
|
||||
rows, stats = [], collections.Counter()
|
||||
for p in corpus:
|
||||
if p not in store.tree:
|
||||
stats["corpus_missing"] += 1
|
||||
continue
|
||||
text = store.text(p)
|
||||
for blk_first, blk in blocks(text):
|
||||
last = None
|
||||
for m in PAT.finditer(blk):
|
||||
abs_line = blk_first + blk.count("\n", 0, m.start())
|
||||
ctx = blk[max(0, m.start() - 70):m.end() + 70].replace("\n", " ")
|
||||
if m.group("path"):
|
||||
last = m.group("path")
|
||||
if m.group("line") is None: # a mention, no cite
|
||||
stats["mentions"] += 1
|
||||
continue
|
||||
want = int(m.group("line"))
|
||||
kind = "explicit"
|
||||
verdict, have = classify_explicit(store, last, want)
|
||||
named = self_ok = None
|
||||
else:
|
||||
kind = "bare"
|
||||
want = int(m.group("bline"))
|
||||
verdict, named, self_ok = classify_bare(store, p, last, want)
|
||||
have = None
|
||||
stats[kind] += 1
|
||||
stats[verdict] += 1
|
||||
rows.append({"file": p, "line": abs_line, "kind": kind,
|
||||
"target": last, "want": want,
|
||||
"end": int(m.group("end")) if m.group("end") else None,
|
||||
"verdict": verdict, "have": have,
|
||||
"named_has": named, "self_has": self_ok,
|
||||
"context": ctx})
|
||||
return rows, stats
|
||||
|
||||
|
||||
# --------------------------------------------------------------- self-test
|
||||
# Each fixture names its own CONTAINING file, because "candidate 2" is a property
|
||||
# of where the citation is written, not of the text alone. Line counts at HEAD:
|
||||
# WORK-QUEUE.md 9,856 / checkins/2026-10-15-volt.md 281 / the reply file 83.
|
||||
QUEUE = "WORK-QUEUE.md"
|
||||
VOLT = "checkins/2026-10-15-volt.md"
|
||||
FIXTURES = [
|
||||
# (containing file, corpus text, [(kind, verdict, target, want)...])
|
||||
(QUEUE, "see checkins/2026-10-15-volt.md:12 for the table",
|
||||
[("explicit", "RESOLVES", "checkins/2026-10-15-volt.md", 12)]),
|
||||
(QUEUE, "WORK-QUEUE.md:999999 is past the end",
|
||||
[("explicit", "STALE_LINE", "WORK-QUEUE.md", 999999)]),
|
||||
(QUEUE, "a host path, not this repo: mercury-task.txt:24",
|
||||
[("explicit", "EXTERNAL_PATH", "mercury-task.txt", 24)]),
|
||||
# the idiom as written: the bare cite takes the file named in its own block,
|
||||
# and when that IS the containing file it is not ambiguity
|
||||
(QUEUE, "The queue is long.\nWORK-QUEUE.md is the file.\n"
|
||||
"Its rows sit at :2264 and a bad one at :99999.",
|
||||
[("bare", "AS_NAMED", "WORK-QUEUE.md", 2264),
|
||||
("bare", "UNRESOLVED", "WORK-QUEUE.md", 99999)]),
|
||||
# a blank line restarts the scope -> the named-file rule has nothing, and the
|
||||
# containing-file candidate is what resolves it. This is the AS_SELF class.
|
||||
(QUEUE, "WORK-QUEUE.md at head.\n\nand the same rows at :2264",
|
||||
[("bare", "AS_SELF", None, 2264)]),
|
||||
# two candidates, both with the line -> the number does not decide
|
||||
(QUEUE, "checkins/2026-10-15-volt.md is a file.\nand the queue's own rows at :123",
|
||||
[("bare", "AMBIGUOUS", "checkins/2026-10-15-volt.md", 123)]),
|
||||
# a named candidate that is not in this tree is out of scope, not a defect
|
||||
(QUEUE, "root/crucible/q51/helm-selftest-q51.py is his gate.\n"
|
||||
"the queue's row at :99999",
|
||||
[("bare", "EXTERNAL_NAMED", "root/crucible/q51/helm-selftest-q51.py", 99999)]),
|
||||
# the tool's own worked catch, replayed as a fixture: the real block names
|
||||
# tools/queue_claim_check.py (695 lines) and the containing check-in is 281
|
||||
# lines, so neither candidate holds the queue's own line numbers
|
||||
(VOLT, "the claim check, and now `tools/queue_claim_check.py` run here, leaves "
|
||||
"the executable set empty: Q37 `:5278` the pipe's, `:8530` Mercury's",
|
||||
[("bare", "UNRESOLVED", "tools/queue_claim_check.py", 5278),
|
||||
("bare", "UNRESOLVED", "tools/queue_claim_check.py", 8530)]),
|
||||
# STATED LIMITATION: a bare cite under 3 digits is out of scope by design --
|
||||
# at 1-2 digits a line number is indistinguishable from a clock, a ratio or a
|
||||
# version, and :08 inside 12:08:10Z is exactly what that floor excludes.
|
||||
(QUEUE, "and the queue's own rows at :12", []),
|
||||
# a clock is not a citation
|
||||
(QUEUE, "read at 12:08:10Z and 09:44:40Z", []),
|
||||
# a range is reported with its end line
|
||||
(QUEUE, "recoveries/helm-pin-hash-convention.md:85-90",
|
||||
[("explicit", "RESOLVES", "recoveries/helm-pin-hash-convention.md", 85)]),
|
||||
]
|
||||
|
||||
|
||||
def _expect(here, text, store):
|
||||
got = []
|
||||
for _first, blk in blocks(text):
|
||||
last = None
|
||||
for m in PAT.finditer(blk):
|
||||
if m.group("path"):
|
||||
last = m.group("path")
|
||||
if m.group("line") is None:
|
||||
continue
|
||||
v, _ = classify_explicit(store, last, int(m.group("line")))
|
||||
got.append(("explicit", v, last, int(m.group("line"))))
|
||||
else:
|
||||
want = int(m.group("bline"))
|
||||
v, _n, _s = classify_bare(store, here, last, want)
|
||||
got.append(("bare", v, last, want))
|
||||
return got
|
||||
|
||||
|
||||
def selftest(repo):
|
||||
store = Store(repo, "HEAD")
|
||||
n = bad = 0
|
||||
for here, text, want in FIXTURES:
|
||||
got = _expect(here, text, store)
|
||||
n += 1
|
||||
ok = got == want
|
||||
bad += 0 if ok else 1
|
||||
print(f"{'PASS' if ok else 'FAIL'} in {here[:26]}: {text[:44]!r}")
|
||||
if not ok:
|
||||
print(f" got {got}")
|
||||
print(f" want {want}")
|
||||
print(f"self-test: {n} fixtures, {n - bad} passed, {bad} failed")
|
||||
return 0 if bad == 0 else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__,
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--repo", default=DEFAULT_REPO)
|
||||
ap.add_argument("--rev", default="HEAD")
|
||||
ap.add_argument("--corpus", nargs="*")
|
||||
ap.add_argument("--json")
|
||||
ap.add_argument("--only", choices=["stale", "asself", "unresolved", "ambiguous",
|
||||
"external", "externalnamed", "named", "all"],
|
||||
default="stale")
|
||||
ap.add_argument("--context", action="store_true")
|
||||
ap.add_argument("--selftest", action="store_true")
|
||||
a = ap.parse_args()
|
||||
|
||||
if a.selftest:
|
||||
return selftest(a.repo)
|
||||
|
||||
store = Store(a.repo, a.rev)
|
||||
corpus = a.corpus or sorted(p for p in store.tree if p.endswith(".md"))
|
||||
rows, stats = scan(a.repo, a.rev, corpus, store)
|
||||
|
||||
print(f"corpus: {len(corpus)} markdown files at {a.rev} in {a.repo}")
|
||||
print("--- EXPLICIT `<path>:<line>` -- a CLAIM (unit: citations) ---")
|
||||
print(f" citations: {stats['explicit']}")
|
||||
for v in ("RESOLVES", "STALE_LINE", "EXTERNAL_PATH"):
|
||||
print(f" {v:15s} {stats[v]}")
|
||||
print("--- BARE `:<line>` -- a CANDIDATE, never a claim (unit: citations) ---")
|
||||
print(f" citations: {stats['bare']}")
|
||||
for v in ("AS_NAMED", "AS_SELF", "AMBIGUOUS", "EXTERNAL_NAMED", "UNRESOLVED"):
|
||||
print(f" {v:15s} {stats[v]}")
|
||||
print(f" ({stats['AS_SELF']} of the bare cites resolve ONLY as the file they "
|
||||
f"are written in -- the named-file rule alone would have called them "
|
||||
f"defects; {stats['UNRESOLVED']} resolve as neither, so no reader can "
|
||||
f"check them without inferring the referent)")
|
||||
print(f"file mentions with no line: {stats['mentions']} (unit: occurrences) "
|
||||
f"-- these seed the inheritance scope")
|
||||
by_target = collections.Counter(r["target"] for r in rows
|
||||
if r["kind"] == "explicit" and r["target"])
|
||||
print(f"distinct explicitly cited targets: {len(by_target)} (unit: distinct paths)")
|
||||
for t, c in by_target.most_common(8):
|
||||
print(f" {c:6d} {t}"
|
||||
f"{'' if t in store.tree else ' [not in tree at ' + a.rev + ']'}")
|
||||
|
||||
sel = {"stale": lambda r: r["kind"] == "explicit" and r["verdict"] == "STALE_LINE",
|
||||
"external": lambda r: r["verdict"] == "EXTERNAL_PATH",
|
||||
"externalnamed": lambda r: r["verdict"] == "EXTERNAL_NAMED",
|
||||
"asself": lambda r: r["verdict"] == "AS_SELF",
|
||||
"unresolved": lambda r: r["verdict"] == "UNRESOLVED",
|
||||
"ambiguous": lambda r: r["verdict"] == "AMBIGUOUS",
|
||||
"named": lambda r: r["verdict"] == "AS_NAMED",
|
||||
"all": lambda r: True}[a.only]
|
||||
show = [r for r in rows if sel(r)]
|
||||
print(f"--- {a.only}: {len(show)} citations (showing first 300) ---")
|
||||
for r in show[:300]:
|
||||
if r["kind"] == "bare":
|
||||
extra = (f" [named={r['target'] or '-'}:"
|
||||
f"{'y' if r['named_has'] else 'n'} self="
|
||||
f"{'y' if r['self_has'] else 'n'}]")
|
||||
else:
|
||||
extra = f" [have={r['have']}]"
|
||||
print(f" {r['file']}:{r['line']} {r['kind']:8s} -> "
|
||||
f"{r['target'] or '-'}:{r['want']} {r['verdict']}{extra}")
|
||||
if a.context:
|
||||
print(f" ...{r['context']}...")
|
||||
if a.json:
|
||||
with open(a.json, "w") as fh:
|
||||
json.dump({"rev": a.rev, "repo": a.repo, "corpus": corpus,
|
||||
"stats": dict(stats), "citations": rows}, fh, indent=1)
|
||||
print(f"wrote {a.json}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
Reference in new issue
Block a user