volt
26056d2a21
`--fail` moves the verdict into the exit code: rc 0 ran and graded no fault, rc 1 ran and graded a STALE claim or an INTERNAL SPLIT, rc 2 could not grade (usage, a --tools filter matching nothing tracked, unreadable input). Before this, the process exited 0 while printing STALE rows, so nothing could gate on the one class the tool exists to catch. Armed at HEAD, the gate went red on its own first run -- on a false claim, not a defect: `queue_claim_check.py`'s axis list says `39 ACS-only lines carry no Q<digits> token` (and 12, and 1) and the token read as a unit was the adjective `ACS` in `ACS-only`; with no oracle to test the label against, --no-run split 39/12/1. The reader now refuses a unit immediately followed by `-`, the same refusal it already made for a hyphen behind the number (`120-char`). Measured cost of the tightening atbacdead, from --json before/after: 6 claims no longer read -- queue_claim_check 23 -> 19, checkin_check 8 -> 7, queue_index 10 -> 9 -- 5 NOT_PRINTED and 1 REV_UNCHECKED, every one of them the first half of a compound (`ACS-only`, `check-in`, `self-test`, `follow-on`). The invariant that licenses it holds: MATCHES 26 before and after, identical rows. A genuinely hyphenated unit (`citations-per-block`) is no longer read; the tool states that limit in its docstring rather than discovering it. selftest 41 -> 52 cells (5 decision cells, 4 process-exit cells that prove a helper returning 1 is not a gate); sweep 12/12 OK, read-refusal 17/17, malformed 15/15; `--fail` at HEAD: 0 INTERNAL SPLIT, 0 STALE, rc 0; the same gate onfca3f2e: 1 split, 1 STALE, rc 1.
1168 lines
59 KiB
Python
1168 lines
59 KiB
Python
#!/usr/bin/env python3
|
|
"""doc_census_check.py -- does a tool's own docstring census match the census
|
|
that tool PRINTS?
|
|
|
|
WHY THIS EXISTS
|
|
This archive's three scanners each carry a census table in their module
|
|
docstring: counts that a reader cites, that no command checks. The docstring
|
|
is prose; prose is not executed. On 2026-10-16 the control plane returned
|
|
exactly this defect against a sibling scanner, in its own words:
|
|
|
|
"The scanner's own docstring carries a number that does not match the
|
|
census it prints -- the exact class the tool exists to catch."
|
|
|
|
tools/citation_resolve_check.py said `471 UNRESOLVED` on one line and
|
|
`441 UNRESOLVED` nine lines later, over a census the tool prints as 471.
|
|
Nothing was checking the docstring against the output beside it. That is this
|
|
tool's whole job.
|
|
|
|
WHAT IT MEASURES (read-only: this tool only writes --json)
|
|
Two independent verdicts per tool, and they do not share evidence.
|
|
|
|
1. INTERNAL SPLIT -- no oracle, no run. Within one docstring, every numeric
|
|
claim of the form `<value> <unit>` is collected; two claims of ONE unit
|
|
that a single revision scope covers, carrying two different values, is a
|
|
split. This is the verdict that catches the 441/471 instance, and it is
|
|
why the tool is useful with no subprocess at all.
|
|
|
|
2. ORACLE -- the tool is RUN, at the revision its own docstring names, and
|
|
each claim is compared against the value that run prints for that unit.
|
|
MATCHES claim == printed value
|
|
STALE claim != printed value, and the printed unit is not
|
|
declared clone-local -- a defect claim
|
|
CLONE_LOCAL claim != printed value but the tool's own output marks
|
|
that unit clone-local ("blobs in THIS clone's local object
|
|
store"), so drift is the documented expectation, not a
|
|
defect. Reported, never counted as one.
|
|
AMBIGUOUS the tool prints that unit with 2+ values at that rev
|
|
(e.g. `citations` prints once per section), so there is no
|
|
single oracle value; both are printed and no claim is made
|
|
NOT_PRINTED the unit does not appear in this tool's output at all
|
|
|
|
A claim is `<digits> <unitword>` with the unit word ADJACENT, which is the
|
|
archive's own notation. A bare number is not a claim and is not checked.
|
|
|
|
TWO ORACLE LEGS, AND WHY THE SECOND ONE EXISTS
|
|
Leg 1 is the tool's own run at the block's revision, exactly as above. Leg 2
|
|
is the SAME TOOL'S `--selftest`, added 2026-10-16 after the control plane
|
|
returned a docstring claim of `13 cells` against a selftest that prints `18
|
|
cells` -- the defect class this tool exists to catch, sitting inside its own
|
|
default scope and still read as NOT_PRINTED, because no leg ever passed
|
|
`--selftest` and a cell count is printed by nothing else. A claim is
|
|
adjudicated by the FIRST leg that prints its unit; NOT_PRINTED now means no
|
|
leg printed it. A tool that takes no `--selftest` simply has one leg -- a fact
|
|
about the tool, not a defect in it.
|
|
|
|
A leg that exits non-zero having printed nothing is reported RUN_FAILED, never
|
|
read as an empty census, and it adjudicates nothing either: the block's claims
|
|
are still read off any surviving leg -- leg 2 -- and are left unadjudicated
|
|
only when no leg survives, because a failed run is not a reason to stop
|
|
looking. A revision that will not materialise fails the same way rather than
|
|
producing a census at all: an empty queue is a census of zeros, and reading it
|
|
as that revision's census is the phantom this rule refuses.
|
|
|
|
A HEX RUN INTRODUCED AS A DIGEST IS NOT A REVISION. A block that quotes a
|
|
truncated sha256 in its prose is a size, not a place: read as a rev it files
|
|
the block under a revision that cannot exist, fails that leg, and then
|
|
adjudicates every claim in the block again beside the real ones. Two signals
|
|
are read, both local to the token -- a digest word in front of it (`sha256
|
|
<hex>`, `hash <hex>`), or an ellipsis behind it (`<hex>…`), which is how this
|
|
archive writes a truncated digest. See `block_revs`.
|
|
|
|
The run leg is `<script> --repo R --rev REV`. A tool that takes no `--rev`
|
|
answers it with an argparse diagnosis and prints no census at all, so the leg
|
|
is re-run with the rejected flags DROPPED -- the exact tokens argparse named,
|
|
and only those -- and is tagged `no-rev`; a recipe's leg may carry the same tag
|
|
for the same reason. Such a tool measures the tree it is POINTED AT, so a claim
|
|
in a block that names a revision is reported REV_UNCHECKED -- never counted as
|
|
verified against that revision, and never counted as a defect -- while a block
|
|
that names no revision is checked normally, since the default is what the flag
|
|
means there.
|
|
|
|
THIS TOOL DOES NOT RUN ITSELF. The default scope is every `tools/*.py` in the
|
|
tree, and this tool is one of them. Running it on itself spawns it, which
|
|
spawns it again, without a floor -- measured, not reasoned, in the check-in of
|
|
2026-10-16. The self target is therefore excluded from the run leg and every
|
|
one of its claims is reported SELF_NOT_RUN. It is deliberately still listed in
|
|
the scanned set, so the exclusion is a printed row and not a silence.
|
|
|
|
SCOPE, STATED RATHER THAN IMPLIED
|
|
Only module docstrings of .py tools are scanned: a docstring is where a census
|
|
table lives. Only revisions NAMED IN THE BLOCK are used as oracles -- a block
|
|
naming no revision is checked against the tool's default (HEAD) and tagged, so
|
|
a stale rev is never silently attributed. A tool whose subject is not `--repo`
|
|
+ `--rev` is driven through a per-tool recipe declared below, with a temp file
|
|
that is removed: `queue_claim_check.py` takes a materialised QUEUE file and so
|
|
its leg IS the revision, while `append_only_owners.py` takes a tree scan that
|
|
no flag can pin to a revision, so its leg is tagged `no-rev`. A tool no recipe
|
|
covers, and that answers `--repo ... --rev ...` with an argparse diagnosis, is
|
|
re-run with the rejected flags dropped and tagged the same way -- measured at
|
|
HEAD on 2026-10-16 against `check_lint.py`, which takes neither flag and whose
|
|
default census is a real one.
|
|
|
|
WHAT THIS DOES NOT CHECK, and the measured size of the gap. A claim is only
|
|
adjudicated when the unit it names is a unit the tool PRINTS. Over the three
|
|
scanners at d355395 the run read ninety-two claims (with the pre-2026-10-18
|
|
separator rule, which read punctuation and list markers as claims): twelve
|
|
were verified against the run they point at, eight were in units the tool
|
|
prints with two values (SECTION_SPLIT_BY_TOOL -- no oracle), and seventy-one
|
|
were in units the tool never prints at all. So the oracle's reach is narrow
|
|
and its precision is the reason: it reported exactly one bad claim on that
|
|
corpus, and that one was the returned defect. A wider rule was tried first and
|
|
falsified by its own first run -- num_agreement_check.py flagged a hundred and
|
|
seventeen restatements across the archive, too coarse to be read -- so the
|
|
narrow rule is the one that ships. Two claim forms are read (`<value> <unit>` and
|
|
`<unit>: <value>`); a claim written in neither form is not seen, and that is a
|
|
stated gap, not a silent one.
|
|
|
|
NO CENSUS OF THIS TOOL'S OWN VERDICTS APPEARS IN THIS DOCSTRING, deliberately:
|
|
this scanner scans itself, so a count of its own output written here would
|
|
have to agree with the output of the run that the same commit changes. That is
|
|
the limit citation_hash_class.py records in the same words -- a commit cannot
|
|
measure the file it introduces -- and here it is avoided rather than
|
|
discovered.
|
|
|
|
FAILING ON THE CLASS IT CATCHES (`--fail`)
|
|
Without it the process exits 0 while printing STALE rows, so no gate could
|
|
act on the one class this tool exists to catch without parsing its text.
|
|
`--fail` makes the exit code carry the verdict. The cost of the old silence
|
|
is measured twice, and the two are NOT the same measurement:
|
|
|
|
- STRUCTURAL -- reproducible right here, no archive needed. A stub tree
|
|
whose docstring says `99 widgets` beside a printed `widgets: 7` exits 0
|
|
without `--fail` and 1 with it (`gate e2e` cells in `--selftest`).
|
|
|
|
- HISTORICAL -- true only of the pre-tightening separator (`ac0e211^`).
|
|
That build at rev 4ce1aed, `--tools tools/checkin_check.py`, exits 0
|
|
with two STALE verdicts on `checkin_check.py:59`: `Never claimed 8` and
|
|
`never claimed 1`, both `printed 2 [selftest]` -- the digits in "Never 1,
|
|
never a traceback". Reproduced 2026-10-18 from this host. The tightened
|
|
separator stops reading those digits as a census: the same rev now
|
|
adjudicates 7 claims, 0 STALE. So the historical run is cited with its
|
|
build and never as current behaviour -- a claim that says "rc 0 with two
|
|
STALE" without naming the build goes stale the moment the reader is
|
|
tightened, which is the failure this tool grades other files for.
|
|
|
|
The code means:
|
|
|
|
0 ran, and graded no fault (every claim adjudicated agrees)
|
|
1 ran, and graded a fault (an INTERNAL SPLIT or a STALE claim)
|
|
2 could not grade (usage, a --tools filter that matches
|
|
nothing tracked, unreadable input)
|
|
|
|
rc 2 keeps its meaning from the tools/ contract (6ee8fce) and is decided
|
|
before the gate runs, so "could not grade" never reads as "graded clean" or
|
|
as "graded a fault". A LEG FAILURE (RUN_FAILED) is reported in its own column
|
|
and does not fail the gate, for the reason counts() states: a leg that
|
|
produced no census adjudicated no claim. A gate that counted it would fail
|
|
runs it never checked.
|
|
|
|
USAGE
|
|
python3 doc_census_check.py [--repo DIR] [--rev HEAD] [--tools NAME...]
|
|
[--json OUT] [--no-run] [--fail] [--selftest] # selftest: 52 cells, one per claim class
|
|
"""
|
|
import argparse
|
|
import ast
|
|
import collections
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
|
|
DEFAULT_REPO = "/root/troupe-standup"
|
|
|
|
# A claim, in either of the two forms this archive writes:
|
|
# `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations"
|
|
# `<unit> [: =] <value>` -- "ACS = 72", "overlap = 20" (markdown bold allowed)
|
|
# A bare number is not a claim. Deliberately not "any word": `471 cannot be
|
|
# resolved` is prose, and a checker that flags prose is a checker nobody reads.
|
|
#
|
|
# THE SEPARATOR IS SPACE, OR A CLOSING DELIMITER AND A SPACE -- tightened
|
|
# 2026-10-18 after the census was read off every tool at HEAD (140 claims) and
|
|
# the separator vocabulary was printed: `''` (11 claims, a hex run read as
|
|
# digits+word: `83acffe0` -> 83 `acffe0`), `'. '` (5, an ordinal or a sentence
|
|
# end: `1. INTERNAL`, `615. Cite`), `'-'` (5, a compound adjective: `120-char`,
|
|
# `1-based`), `' '` (4, an aligned list marker: `0 ran`), and the punctuation
|
|
# spills `'; '` `' "'` `'+ '` `') '`. Only 95 of the 140 were a count and its
|
|
# unit. The loose class fired two STALE verdicts in the full run at 4ce1aed
|
|
# against `checkin_check.py:59` -- "Never 1, never a traceback", where the digit
|
|
# is a code, not a census -- which is a false defect claim of exactly the kind
|
|
# this archive grades lanes for. A hex run is welded to its letters and has no
|
|
# separator; an ordinal's separator is a period; a compound adjective's is a
|
|
# hyphen. None of the three is `<value> <unit>`.
|
|
#
|
|
# A HYPHEN AFTER THE UNIT IS THE SAME REFUSAL, made 2026-10-18 after the
|
|
# --fail gate went red at HEAD on a false claim: `queue_claim_check.py`'s axis
|
|
# list says `39 ACS-only lines carry no Q<digits> token`, `12 ACS-only lines
|
|
# are indented`, `1 ACS-only line carries a BQ exclusion phrase` -- three
|
|
# sub-counts of ONE population of 52, and the token read as a unit was the
|
|
# adjective `ACS` in `ACS-only`. With no oracle to test the label against, a
|
|
# `--no-run` split fired on 39/12/1. A hyphen immediately behind the unit word
|
|
# marks a compound adjective, not a unit noun, exactly as a hyphen behind the
|
|
# number does (`120-char`). The cost is stated: a genuinely hyphenated unit
|
|
# (`citations-per-block`) is no longer read, and the invariant that licenses
|
|
# this tightening is that the 26 MATCHES do not move -- measured before and
|
|
# after, same 26 rows.
|
|
CLAIM = re.compile(
|
|
r"(?<![0-9A-Za-z_.])(?P<num>[0-9]+(?:,[0-9]{3})*)(?P<sep>\s{1,2}|[)\]}\"'`*]\s)"
|
|
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Z_]{2,})\b(?!-)")
|
|
CLAIM_EQ = re.compile(
|
|
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*\**\s*(?P<num>[0-9]+(?:,[0-9]{3})*)")
|
|
REV_LONG = re.compile(r"\b[0-9a-f]{7,40}\b")
|
|
|
|
# A hex run introduced as a DIGEST is not a revision, and reading it as one costs
|
|
# more than a wrong label: the block is filed under a revision that cannot exist,
|
|
# its leg fails, and then every claim in the block is adjudicated once per
|
|
# spurious rev beside the real ones. Measured 2026-10-16 at HEAD on
|
|
# `tools/queue_claim_check.py`, whose UNITS block quotes the queue's size as
|
|
# "/ sha256 `fbe51445a89f81ab…`": three claims emitted FOUR times, once per
|
|
# REV_LONG token, and two `git show <sha256-prefix>:WORK-QUEUE.md` leg failures.
|
|
# Two signals, both local to the token: a digest word in front of it (sha256,
|
|
# sha1, md5, digest, hash), or an ellipsis behind it -- the archive's own way of
|
|
# writing a truncated digest. A revision cited in the ordinary way carries
|
|
# neither, and a FULL sha256 is already invisible to REV_LONG: the run has no
|
|
# word boundary to bite on.
|
|
DIGEST_BEFORE = re.compile(
|
|
r"(?:sha[-\s]?256|sha[-\s]?1|md5|digest|hash)\s*[:=]?\s*[`'\"]?\s*$", re.I)
|
|
DIGEST_AFTER = re.compile(r"^[`'\"]?\s*(?:…|\.\.\.)")
|
|
|
|
|
|
def block_revs(text):
|
|
"""The revisions a block names -- hex runs read as digests excluded."""
|
|
out = set()
|
|
for m in REV_LONG.finditer(text):
|
|
if DIGEST_BEFORE.search(text[:m.start()]) or \
|
|
DIGEST_AFTER.match(text[m.end():]):
|
|
continue
|
|
out.add(m.group(0))
|
|
return sorted(out)
|
|
|
|
# Function words and units this tool does not adjudicate. `at` in "2026-10-15 at
|
|
# rev 9188dd3" is not a census quantity; neither is `B` (bytes differ by
|
|
# revision by design, and "1,201,550 B" is a size, not a count). Stated so the
|
|
# exclusion is a rule rather than a habit.
|
|
STOPWORDS = {
|
|
"at", "the", "of", "in", "is", "are", "was", "and", "or", "to", "for", "on",
|
|
"by", "as", "it", "its", "that", "this", "these", "those", "with", "from",
|
|
"no", "not", "one", "two", "three", "all", "any", "be", "been", "has",
|
|
"have", "had", "his", "her", "our", "your", "their", "but", "if", "then",
|
|
"than", "so", "up", "out", "off", "over", "into", "run", "runs", "may",
|
|
"can", "same", "each", "both", "per", "via", "b", "kb", "mb", "gb",
|
|
}
|
|
|
|
# How to run each tool at a revision. Any tool not listed here is run as
|
|
# `<script> --repo R --rev REV`. A recipe returns `(argv, tag)` and the tag says
|
|
# whether the recipe could materialise the revision the block names.
|
|
#
|
|
# queue_claim_check.py takes no --rev: its subject is a FILE, and the file is
|
|
# materialised at the revision and handed to it -- so the leg IS that rev.
|
|
# A revision that will NOT materialise is a raised failure, never an empty
|
|
# file: an empty queue is a census of zeros, and a census of zeros reads as
|
|
# that revision's census. Measured 2026-10-16 -- `git show
|
|
# c3b4a290dc64bd24:WORK-QUEUE.md` (a sha256 prefix REV_LONG read as a rev)
|
|
# exits 128, and the leg it fed printed `0 opening records` at rc 0.
|
|
# append_only_owners.py takes a positional AUDIT.json and nothing else: its
|
|
# input is a tree scan, which its sibling produces over the WORKING TREE
|
|
# (`append_only_audit.py` has no --rev either). This scanner is read-only, so
|
|
# it will not add a git worktree to the repository it is only allowed to
|
|
# read: the leg is tagged no-rev and every claim in a block that names a
|
|
# revision is reported REV_UNCHECKED, never adjudicated against a tree it
|
|
# did not stand on.
|
|
def _recipe_queue(repo, rev, tmpdir):
|
|
dst = os.path.join(tmpdir, f"WORK-QUEUE.{rev}.md")
|
|
p = subprocess.run(["git", "-C", repo, "show", f"{rev}:WORK-QUEUE.md"],
|
|
capture_output=True)
|
|
if p.returncode != 0:
|
|
raise RuntimeError("git show %s:WORK-QUEUE.md -> rc=%d: %s"
|
|
% (rev, p.returncode,
|
|
p.stderr.decode("utf-8", "replace").strip()[:120]))
|
|
with open(dst, "wb") as fh:
|
|
fh.write(p.stdout)
|
|
return ["--queue", dst, "--census"], ""
|
|
|
|
|
|
def _recipe_owners(repo, rev, tmpdir):
|
|
dst = os.path.join(tmpdir, f"append-only-audit.{rev}.json")
|
|
subprocess.run([sys.executable,
|
|
os.path.join(repo, "tools", "append_only_audit.py"),
|
|
"--repo", repo, "--json", dst], capture_output=True)
|
|
return [dst], "no-rev"
|
|
|
|
|
|
RECIPES = {"queue_claim_check.py": _recipe_queue,
|
|
"append_only_owners.py": _recipe_owners}
|
|
|
|
|
|
def git(repo, *a):
|
|
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
|
|
|
|
|
|
def docstring_span(path, text):
|
|
try:
|
|
tree = ast.parse(text)
|
|
except SyntaxError:
|
|
return None
|
|
if not tree.body or not isinstance(tree.body[0], ast.Expr):
|
|
return None
|
|
node = tree.body[0]
|
|
if not isinstance(node.value, ast.Constant) or not isinstance(node.value.value, str):
|
|
return None
|
|
return node.lineno, node.end_lineno
|
|
|
|
|
|
def blocks(lines, lo, hi):
|
|
"""Blank-line-delimited paragraphs -- the same scope the sibling scanners
|
|
use (citation_resolve_check.py for inheritance, num_agreement_check.py for
|
|
quantity agreement), so one paragraph is one scope in all three."""
|
|
start, buf = lo, []
|
|
for ln in range(lo, hi + 1):
|
|
raw = lines[ln - 1]
|
|
if raw.strip() == "":
|
|
if buf:
|
|
yield start, buf
|
|
buf, start = [], ln + 1
|
|
else:
|
|
buf.append((ln, raw))
|
|
if buf:
|
|
yield start, buf
|
|
|
|
|
|
# ------------------------------------------------- printed census -> oracle
|
|
def parse_printed(stdout):
|
|
"""stdout -> {unit: {value: [line, ...]}} plus the set of clone-local units.
|
|
|
|
Four line shapes are read, all of them the scanners' own:
|
|
` RESOLVES 113` indented label then count (family rows)
|
|
` citations: 410` label then count
|
|
`corpus: 307 files at ...` count then unit
|
|
`overlap A&B = 20 ; ...` inline assignments, split on the separators
|
|
A unit whose printed line (or the line above it, for a wrapped phrase) says
|
|
"clone" is clone-local: its count is expected to move with the clone.
|
|
"""
|
|
units = collections.defaultdict(lambda: collections.defaultdict(list))
|
|
clone = set()
|
|
lines = stdout.split("\n")
|
|
|
|
def add(unit, value, ln, cloneline):
|
|
u = unit.strip().lower()
|
|
if len(u) < 3 or not re.fullmatch(r"[a-z_][a-z0-9_]*", u) or u in STOPWORDS:
|
|
return
|
|
units[u][value].append(ln)
|
|
if "clone" in cloneline.lower():
|
|
clone.add(u)
|
|
|
|
for i, raw in enumerate(lines, 1):
|
|
ctx = raw + " " + (lines[i - 2] if i >= 2 else "")
|
|
m = re.match(r"^\s{2,}([A-Za-z_][A-Za-z0-9_]*)\s+([0-9][0-9,]*)\s*$", raw)
|
|
if m: # family row: label count
|
|
add(m.group(1), int(m.group(2).replace(",", "")), i, ctx)
|
|
continue
|
|
m = re.match(r"^\s*([A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*([0-9][0-9,]*)\b", raw)
|
|
if m: # label: count
|
|
add(m.group(1), int(m.group(2).replace(",", "")), i, ctx)
|
|
for m in re.finditer(r"([0-9][0-9,]*)\s+(?:[a-z]+ )?([A-Za-z_][A-Za-z0-9_]*)\b", raw):
|
|
add(m.group(2), int(m.group(1).replace(",", "")), i, ctx)
|
|
for seg in re.split(r"[;]", raw): # inline `X = N`
|
|
m = re.match(r"^\s*(.*?)\s*[:=]\s*([0-9][0-9,]*)\s*$", seg)
|
|
if not m or not m.group(1):
|
|
continue
|
|
words = re.findall(r"[A-Za-z_][A-Za-z0-9_]*", m.group(1))
|
|
if not words:
|
|
continue
|
|
val = int(m.group(2).replace(",", ""))
|
|
add("_".join(w.lower() for w in words), val, i, ctx) # whole phrase
|
|
add(words[-1], val, i, ctx) # the nearest label: `overlap A&B`
|
|
# the scanners' own unit declaration: `... 6961 (unit: occurrences)`.
|
|
# The unit is named AFTER its number, which no other rule reads.
|
|
for m in re.finditer(r"\(\s*unit\s*:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\)", raw):
|
|
left = raw[:m.start()]
|
|
nums = re.findall(r"([0-9][0-9,]*)\s*$", left.rstrip())
|
|
if nums:
|
|
add(m.group(1), int(nums[-1].replace(",", "")), i, ctx)
|
|
return {k: dict(v) for k, v in units.items()}, clone
|
|
|
|
|
|
def ordinal_at(raw, pos):
|
|
"""True when `pos` is the number that OPENS this line and 2+ spaces follow.
|
|
|
|
That is a list marker -- the exit-code block's `0 ran; nothing to report`,
|
|
`1 ran, and graded ...`, `2 could not grade` -- and the word after it is
|
|
prose, not a unit. The same shape mid-line (`WORK-QUEUE.md:8003 cites`) is
|
|
a citation's line number, not a count, and is excluded by the same rule.
|
|
"""
|
|
m = re.match(r"[ \t]*([0-9][0-9,]*)([ \t]{2,})", raw)
|
|
return bool(m) and m.start(1) == pos
|
|
|
|
|
|
def claims_in(blk):
|
|
"""[(line, value, unit, token, offset)] for one block.
|
|
|
|
Both claim forms are read and a claim found by both is reported once. The
|
|
offset is measured on the block text so a RESTATEMENT (far apart) can be
|
|
told from an ENUMERATION of neighbouring sub-counts, the same
|
|
discrimination num_agreement_check.py applies.
|
|
"""
|
|
out, off = [], 0
|
|
for ln, raw in blk:
|
|
found = {}
|
|
for m in CLAIM.finditer(raw):
|
|
u = m.group("unit")
|
|
if len(u) < 3 or u.lower() in STOPWORDS or re.fullmatch(r"[qQ]\d+", u):
|
|
continue
|
|
if ordinal_at(raw, m.start()):
|
|
continue
|
|
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
|
|
for m in CLAIM_EQ.finditer(raw):
|
|
u = m.group("unit")
|
|
if (len(u) < 3 or u.lower() in STOPWORDS
|
|
or re.fullmatch(r"[qQ]\d+", u) or m.start() in found):
|
|
continue
|
|
if m.start() and re.match(r"[0-9A-Za-z_.]", raw[m.start() - 1]):
|
|
continue # `WORK-QUEUE.md:8586` is a citation
|
|
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
|
|
for pos in sorted(found):
|
|
val, u, tok = found[pos]
|
|
out.append((ln, val, u, tok, off + pos))
|
|
off += len(raw) + 1
|
|
return out
|
|
|
|
|
|
def is_label(unit):
|
|
"""An uppercase census label -- the vocabulary a split is checked in when
|
|
the tool has not been run (no oracle available)."""
|
|
return len(unit) >= 3 and re.fullmatch(r"[A-Z][A-Z_]*", unit) is not None
|
|
|
|
|
|
def keys_for(unit, printed):
|
|
"""Candidate oracle keys for a claimed unit word."""
|
|
cands = [unit, unit.lower(), unit.lower().rstrip("s"), unit.upper()]
|
|
seen, out = set(), []
|
|
for c in cands:
|
|
if c.lower() in printed and c.lower() not in seen:
|
|
seen.add(c.lower())
|
|
out.append(c.lower())
|
|
return out
|
|
|
|
|
|
def oracle(printed, clone, unit, value, tag=""):
|
|
"""(verdict, detail) for one claim; `printed` is one revision's map.
|
|
|
|
`tag` names the leg a verdict came from when it is not leg 1 ("no-rev",
|
|
"selftest"): a reader can then tell which census the number was read off.
|
|
"""
|
|
keys = keys_for(unit, printed)
|
|
vals = set()
|
|
for k in keys:
|
|
vals |= set(printed[k])
|
|
if not vals:
|
|
return "NOT_PRINTED", ""
|
|
if len(vals) > 1:
|
|
return "AMBIGUOUS", "printed: " + ", ".join(str(v) for v in sorted(vals))
|
|
printed_val = vals.pop()
|
|
if printed_val == value:
|
|
return "MATCHES", ""
|
|
if any(k in clone for k in keys):
|
|
return "CLONE_LOCAL", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
|
|
return "STALE", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
|
|
|
|
|
|
def adjudicate(legs, unit, value, block_revs):
|
|
"""The first leg that prints the unit adjudicates the claim.
|
|
|
|
`legs` is [(printed, clone, tag)] in priority order: leg 1 (the run) first,
|
|
then the selftest leg. NEITHER a `no-rev` leg NOR the selftest leg is a
|
|
verifier for a block that names a revision: one measured a different tree,
|
|
the other measured the tool's own fixtures. Such a claim is reported
|
|
REV_UNCHECKED -- visible, counted as neither a match nor a defect.
|
|
"""
|
|
for printed, clone, tag in legs:
|
|
if not printed or not keys_for(unit, printed):
|
|
continue
|
|
if block_revs and tag == "no-rev":
|
|
return "REV_UNCHECKED", ("block names %s; the tool takes no --rev, so this "
|
|
"leg measured the working tree"
|
|
% ",".join(block_revs))
|
|
if block_revs and tag == "selftest":
|
|
return "REV_UNCHECKED", ("block names %s; the selftest leg prints this "
|
|
"tool's own fixtures, not a census at that "
|
|
"revision" % ",".join(block_revs))
|
|
return oracle(printed, clone, unit, value, tag)
|
|
return "NOT_PRINTED", ""
|
|
|
|
|
|
# --------------------------------------------------------------- self-test
|
|
# Synthetic stdout, so the parsers and the verdicts are executed, not asserted.
|
|
FAKE_STDOUT = """\
|
|
corpus: 297 markdown files at babdadc in /root/troupe-standup
|
|
--- EXPLICIT `<path>:<line>` -- a CLAIM (unit: citations) ---
|
|
citations: 410
|
|
RESOLVES 113
|
|
STALE_LINE 0
|
|
--- BARE `:<line>` -- a CANDIDATE, never a claim (unit: citations) ---
|
|
citations: 1612
|
|
UNRESOLVED 471
|
|
file mentions with no line: 6961 (unit: occurrences) -- these seed the scope
|
|
index: 871 blobs (unit: blobs in THIS clone's local object store), 680 commits
|
|
overlap A&B = 20 ; ACS-only = 52
|
|
"""
|
|
FAKE_DOC = """\
|
|
THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
|
|
BARE 1612 citations: 255 EXTERNAL_NAMED / 471 UNRESOLVED.
|
|
Read every figure as a citation count, never as a defect count: 441 UNRESOLVED
|
|
is not 441 errors.
|
|
|
|
A SECOND BLOCK, no revision named, quoting 471 UNRESOLVED and nothing else.
|
|
"""
|
|
|
|
# Leg 2's shape: a selftest summary line, and nothing else. The cell count is
|
|
# printed here and nowhere else, which is why a claim about it needs this leg.
|
|
FAKE_SELFTEST = """\
|
|
c1 a NEW byte PASS
|
|
selftest: 18 cells, 18 passed, 0 failed
|
|
"""
|
|
|
|
|
|
def _fixture_checks():
|
|
printed, clone = parse_printed(FAKE_STDOUT)
|
|
ok = True
|
|
cells, bad = [0], [0]
|
|
|
|
def check(name, got, want):
|
|
nonlocal ok
|
|
good = got == want
|
|
cells[0] += 1
|
|
bad[0] += 0 if good else 1
|
|
ok = ok and good
|
|
print(f"{'PASS' if good else 'FAIL'} {name}")
|
|
if not good:
|
|
print(f" got {got}")
|
|
print(f" want {want}")
|
|
|
|
check("parse: UNRESOLVED single", sorted(printed.get("unresolved", {})), [471])
|
|
check("parse: citations ambiguous", sorted(printed.get("citations", {})), [410, 1612])
|
|
check("parse: count-then-unit", sorted(printed.get("occurrences", {})), [6961])
|
|
check("parse: one word skipped", sorted(printed.get("files", {})), [297])
|
|
check("parse: clone-local marked", "blobs" in clone, True)
|
|
# the phrase rule: `ACS-only = 52` must NOT register as the unit `acs`
|
|
check("parse: inline phrase key", sorted(printed.get("acs_only", {})), [52])
|
|
|
|
lines = FAKE_DOC.split("\n")
|
|
got, first_block, revs_seen = [], [], set()
|
|
for first, blk in blocks(lines, 1, len(lines)):
|
|
revs = block_revs("\n".join(r for _, r in blk))
|
|
revs_seen |= set(revs)
|
|
for ln, val, unit, _tok, _pos in claims_in(blk):
|
|
got.append((unit, val))
|
|
if first == 1:
|
|
first_block.append((unit, val))
|
|
# the fixture that matters: one unit, one rev scope, two values -> a split
|
|
check("claims: both UNRESOLVED values in one block",
|
|
[uv for uv in first_block if uv[0] == "UNRESOLVED"],
|
|
[("UNRESOLVED", 471), ("UNRESOLVED", 441)])
|
|
check("claims: `471 cannot be resolved` is not a claim",
|
|
any(u in ("cannot", "resolved") for u, _ in got), False)
|
|
# the tightened separator, one cell per form it now refuses, plus the
|
|
# positive control -- a rule that refuses everything proves nothing
|
|
def claims_of(text):
|
|
return [(u, v) for _ln, v, u, _t, _p in claims_in([(1, text)])]
|
|
|
|
check("claims: a welded digit-and-letter run is not a claim",
|
|
claims_of("sha 83acffe0 / rev 9188dd3 at HEAD"), [])
|
|
check("claims: an ordinal is not a claim",
|
|
claims_of("1. INTERNAL SPLIT -- no oracle, no run"), [])
|
|
check("claims: a list marker is not a claim",
|
|
claims_of("0 ran; nothing to report"), [])
|
|
check("claims: a sentence end is not a claim",
|
|
claims_of("read as UTF-8. Never 1, never a traceback"), [])
|
|
check("claims: a compound adjective is not a claim",
|
|
claims_of("a 120-char window and a 1-based index"), [])
|
|
# the same refusal on the unit side, which is where the 2026-10-18 false
|
|
# split came from: `ACS-only` is a compound adjective, not a unit
|
|
check("claims: a hyphenated unit word is a compound adjective, not a claim",
|
|
claims_of("39 ACS-only lines carry no Q<digits> token"), [])
|
|
check("claims: the same unit unhyphenated is still a claim",
|
|
claims_of("39 ACS lines carry no Q<digits> token"), [("ACS", 39)])
|
|
check("claims: the tightening keeps a count and its unit",
|
|
claims_of("471 UNRESOLVED and 1,612 citations"),
|
|
[("UNRESOLVED", 471), ("citations", 1612)])
|
|
check("claims: the block's rev is read", "babdadc" in revs_seen, True)
|
|
# the digest rule, one signal per cell, and the negative control between
|
|
# them -- the text of each is the text of the defect, measured at HEAD
|
|
check("revs: `sha256 <hex>` is a digest, not the block's revision",
|
|
block_revs("queue 1,170,483 B / sha256 `c3b4a290dc64bd24`"), [])
|
|
check("revs: an ellipsis-truncated hex run is a digest, not a revision",
|
|
block_revs("at `28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`)"),
|
|
["28ca8e5"])
|
|
check("revs: an ordinarily cited revision is still the block's revision",
|
|
block_revs("Run 37 read (`29a4872`, queue 1,170,483 B) and `28ca8e5`"),
|
|
["28ca8e5", "29a4872"])
|
|
check("oracle: 471 matches", oracle(printed, clone, "UNRESOLVED", 471),
|
|
("MATCHES", ""))
|
|
check("oracle: 441 is stale", oracle(printed, clone, "UNRESOLVED", 441),
|
|
("STALE", "printed 471"))
|
|
check("oracle: ambiguous unit makes no claim",
|
|
oracle(printed, clone, "citations", 1612),
|
|
("AMBIGUOUS", "printed: 410, 1612"))
|
|
check("oracle: clone-local is not a defect",
|
|
oracle(printed, clone, "blobs", 1234), ("CLONE_LOCAL", "printed 871"))
|
|
check("oracle: unprinted unit", oracle(printed, clone, "wombats", 3),
|
|
("NOT_PRINTED", ""))
|
|
|
|
# ---- leg 2 (--selftest) and the leg rules. The first check is the defect
|
|
# the leg was added for: a docstring cell count the run leg cannot see.
|
|
s2, s2c = parse_printed(FAKE_SELFTEST)
|
|
check("selftest leg: the cell count is a printed unit", sorted(s2.get("cells", {})), [18])
|
|
legs = [(printed, clone, "")] + [(s2, s2c, "selftest")]
|
|
check("adjudicate: a cell count leg 1 cannot see is STALE, not NOT_PRINTED",
|
|
adjudicate(legs, "cells", 13, []), ("STALE", "printed 18 [selftest]"))
|
|
check("adjudicate: the corrected cell count matches",
|
|
adjudicate(legs, "cells", 18, []), ("MATCHES", ""))
|
|
check("adjudicate: leg 1 wins when both legs print the unit",
|
|
adjudicate([(s2, s2c, ""), ({**s2, "cells": {99: [1]}}, s2c, "selftest")],
|
|
"cells", 99, []), ("STALE", "printed 18"))
|
|
check("adjudicate: a no-rev leg does not verify a block that names a revision",
|
|
adjudicate([(s2, s2c, "no-rev")], "cells", 18, ["f5403b2"]),
|
|
("REV_UNCHECKED", "block names f5403b2; the tool takes no --rev, so this "
|
|
"leg measured the working tree"))
|
|
check("adjudicate: the selftest leg is not a verifier at a named revision",
|
|
adjudicate([(s2, s2c, "selftest")], "cells", 18, ["f5403b2"]),
|
|
("REV_UNCHECKED", "block names f5403b2; the selftest leg prints this "
|
|
"tool's own fixtures, not a census at that revision"))
|
|
check("adjudicate: a no-rev leg IS an oracle for a block naming no revision",
|
|
adjudicate([(s2, s2c, "no-rev")], "cells", 13, []),
|
|
("STALE", "printed 18 [no-rev]"))
|
|
check("adjudicate: no leg prints the unit -> NOT_PRINTED",
|
|
adjudicate([(printed, clone, "")], "wombats", 3, []), ("NOT_PRINTED", ""))
|
|
|
|
# ---- the recipe contract, exercised rather than asserted: run_tool takes a
|
|
# recipe's (argv, tag) and the tag must survive to the leg, because the
|
|
# tag is what decides whether a claim may be adjudicated at all. A stub
|
|
# stands in for a tool so this stays hermetic and instant; the real
|
|
# owners recipe costs a 107 s tree scan and is exercised by the run.
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
|
|
stub = os.path.join(td, "stub.py")
|
|
with open(stub, "w") as fh:
|
|
fh.write("print('cells: 7')\n")
|
|
for want_tag in ("no-rev", ""):
|
|
RECIPES["stub.py"] = (lambda t: lambda repo, rev, tmp: (["--probe"], t))(want_tag)
|
|
try:
|
|
out, rc, tag = run_tool(td, stub, "HEAD", td)
|
|
finally:
|
|
RECIPES.pop("stub.py", None)
|
|
check(f"recipe: run_tool carries a recipe's tag {want_tag!r} to the leg",
|
|
(rc, tag, "cells: 7" in out), (0, want_tag, True))
|
|
|
|
def _boom(repo_, rev_, tmp_):
|
|
raise RuntimeError("boom")
|
|
|
|
RECIPES["stub.py"] = _boom
|
|
raised = ""
|
|
try:
|
|
try:
|
|
run_tool(td, stub, "HEAD", td)
|
|
except RuntimeError as exc:
|
|
raised = str(exc)
|
|
finally:
|
|
RECIPES.pop("stub.py", None)
|
|
check("recipe: an unmaterialisable revision raises, never an empty census",
|
|
raised, "boom")
|
|
|
|
# A tool that rejects `--repo`/`--rev` but has a perfectly good default
|
|
# census. The retry must drop exactly the tokens argparse named: the
|
|
# first cut re-ran it as `<script> --repo R`, which was rejected again,
|
|
# and a working tool was reported RUN_FAILED. Both halves are pinned --
|
|
# the retry happens, and it happens only for an argparse rejection.
|
|
apstub = os.path.join(td, "stub_argparse.py")
|
|
with open(apstub, "w") as fh:
|
|
fh.write("import argparse\n"
|
|
"ap = argparse.ArgumentParser()\n"
|
|
'ap.add_argument("--selftest", action="store_true")\n'
|
|
"ap.parse_args()\n"
|
|
"print('cells: 7')\n")
|
|
out, rc, tag = run_tool(td, apstub, "HEAD", td)
|
|
check("run_tool: argparse rejection retried with the rejected flags dropped",
|
|
(rc, tag, "cells: 7" in out), (0, "no-rev", True))
|
|
|
|
rcstub = os.path.join(td, "stub_rc2.py")
|
|
with open(rcstub, "w") as fh:
|
|
fh.write("import sys\n"
|
|
"sys.stderr.write('boom\\n')\n"
|
|
"sys.exit(2)\n")
|
|
out, rc, tag = run_tool(td, rcstub, "HEAD", td)
|
|
check("run_tool: a non-argparse failure is not re-tagged no-rev",
|
|
(rc, tag, "rc=2" in out), (2, "", True))
|
|
# ---- a failed run leg does not silence its block. This path lives in
|
|
# check_tool and needs a subprocess, so it is exercised end to end with
|
|
# a stub tool rather than asserted: the stub rejects every census
|
|
# invocation with an argparse diagnosis and answers only `--selftest`.
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
|
|
stub = os.path.join(td, "stub_failing_run.py")
|
|
with open(stub, "w") as fh:
|
|
fh.write("'''stub: a run leg that fails, a selftest that answers.\n"
|
|
"\n7 cells\n'''\n"
|
|
"import sys\n"
|
|
'if "--selftest" in sys.argv:\n'
|
|
' print("selftest: 7 cells, 7 passed, 0 failed")\n'
|
|
"else:\n"
|
|
' sys.stderr.write("usage: stub_failing_run.py [-h] [--selftest]\\n")\n'
|
|
" sys.exit(2)\n")
|
|
rep = check_tool(td, stub, "stub_failing_run.py", {}, run=True, tmpdir=td)
|
|
check("check_tool: a failed run leg is reported, and leg 2 still adjudicates",
|
|
([f["verdict"] for f in rep["failures"]],
|
|
[(v["unit"], v["claim"], v["verdict"]) for v in rep["verdicts"]]),
|
|
(["RUN_FAILED"], [("cells", 7, "MATCHES")]))
|
|
|
|
# ---- the banner's arithmetic, which is what a reader actually reads: a
|
|
# RUN_FAILED leg and a STALE claim are counted in two columns, so four
|
|
# failed runs cannot read as four bad claims.
|
|
synthetic = {"tool": "t", "splits": [], "unchecked": {}, "selftest_leg": True,
|
|
"failures": [{"verdict": "RUN_FAILED"}],
|
|
"verdicts": [{"verdict": "MATCHES"}, {"verdict": "STALE"}]}
|
|
check("counts: a leg failure is not a claim defect", counts(synthetic)[:4],
|
|
(0, 1, 1, 2))
|
|
check("counts: an absent leg 2 is noted",
|
|
counts({**synthetic, "selftest_leg": False})[4],
|
|
["no --selftest: leg 2 absent"])
|
|
|
|
# --tools: a filter that matches nothing must not read as a clean scan.
|
|
check("tools: a bare basename is reported missing, not scanned as zero",
|
|
resolve_tools(["checkin_check.py"], {"tools/checkin_check.py"}),
|
|
(["checkin_check.py"], ["checkin_check.py"]))
|
|
check("tools: no filter means every tracked tool and nothing missing",
|
|
resolve_tools(None, {"tools/b.py", "tools/a.py", "README.md"}),
|
|
(["tools/a.py", "tools/b.py"], []))
|
|
|
|
# ---- the gate. Before `--fail` this tool exited 0 while printing STALE
|
|
# rows -- the two historical STALEs at 4ce1aed require the
|
|
# pre-tightening reader, `ac0e211^`, so they are cited with that build
|
|
# and not as current behaviour (see FAILING ON THE CLASS IT CATCHES) --
|
|
# so the one class it exists to catch could not fail anything. The
|
|
# first five cells pin the decision; the last four pin the PROCESS exit
|
|
# code, because a helper returning 1 proves nothing about a gate.
|
|
stale_rep = {"tool": "t", "splits": [], "unchecked": {}, "failures": [],
|
|
"verdicts": [{"verdict": "STALE"}], "selftest_leg": True}
|
|
split_rep = {**stale_rep, "splits": [{"unit": "UNRESOLVED", "rows": []}],
|
|
"verdicts": [{"verdict": "MATCHES"}]}
|
|
# a leg failure alone: reported, never spent as a defect
|
|
clean_rep = {**stale_rep, "failures": [{"verdict": "RUN_FAILED"}],
|
|
"verdicts": [{"verdict": "MATCHES"},
|
|
{"verdict": "NOT_PRINTED"},
|
|
{"verdict": "AMBIGUOUS"},
|
|
{"verdict": "REV_UNCHECKED"}]}
|
|
check("gate: off is rc 0 even with a STALE claim", gate_rc([stale_rep], False), 0)
|
|
check("gate: on and a STALE claim -> rc 1", gate_rc([stale_rep], True), 1)
|
|
check("gate: on and an INTERNAL SPLIT alone -> rc 1", gate_rc([split_rep], True), 1)
|
|
check("gate: on and a leg failure alone -> rc 0 (reported, not spent)",
|
|
gate_rc([clean_rep], True), 0)
|
|
check("gate: the two columns sum across tools", gate_faults([stale_rep, split_rep]),
|
|
(1, 1))
|
|
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-gate-") as td:
|
|
os.makedirs(os.path.join(td, "tools"))
|
|
for name, claim in (("stub_stale.py", 99), ("stub_ok.py", 7)):
|
|
with open(os.path.join(td, "tools", name), "w") as fh:
|
|
fh.write(f'"""A stub whose docstring says {claim} widgets."""\n'
|
|
"print('widgets: 7')\n")
|
|
genv = dict(os.environ, GIT_AUTHOR_NAME="t", GIT_AUTHOR_EMAIL="t@t",
|
|
GIT_COMMITTER_NAME="t", GIT_COMMITTER_EMAIL="t@t")
|
|
for sub in (["init", "-q"], ["add", "-A"], ["commit", "-qm", "fixture"]):
|
|
subprocess.run(["git", "-C", td] + sub, capture_output=True, env=genv)
|
|
|
|
def run_gate(*extra):
|
|
return subprocess.run([sys.executable, os.path.abspath(__file__),
|
|
"--repo", td, "--rev", "HEAD"] + list(extra),
|
|
capture_output=True, env=genv).returncode
|
|
|
|
check("gate e2e: a STALE claim fails the process (rc 1)", run_gate("--fail"), 1)
|
|
check("gate e2e: the same tree without --fail is rc 0 (the old silence)",
|
|
run_gate(), 0)
|
|
check("gate e2e: a matching claim passes the gate (rc 0)",
|
|
run_gate("--tools", "tools/stub_ok.py", "--fail"), 0)
|
|
check("gate e2e: a --tools filter matching nothing is rc 2, not 1",
|
|
run_gate("--tools", "tools/nope.py", "--fail"), 2)
|
|
|
|
print(f"selftest: {cells[0]} cells, {cells[0] - bad[0]} passed, {bad[0]} failed")
|
|
return 0 if ok else 1
|
|
|
|
|
|
# ------------------------------------------------------------------- driver
|
|
def run_tool(repo, script, rev, tmpdir, timeout=600):
|
|
"""(stdout, rc, tag) for the tool's own run at `rev`.
|
|
|
|
`tag` is "" when the leg stood on the revision it was asked for and "no-rev"
|
|
when it did not: either the tool rejected `--rev` and was re-run without it,
|
|
or a recipe in RECIPES declared that it could not materialise the revision.
|
|
Such a leg measures the tree it is pointed at, so a claim in a block that
|
|
names a revision is reported REV_UNCHECKED rather than adjudicated (see
|
|
check_tool). stdout is returned; stderr is only folded in when there is no
|
|
stdout at all, so an argparse diagnosis is visible as a failure instead of
|
|
parsed as a census.
|
|
"""
|
|
recipe = RECIPES.get(os.path.basename(script), None)
|
|
if recipe is None:
|
|
argv, tag = [sys.executable, script, "--repo", repo, "--rev", rev], ""
|
|
else:
|
|
args, tag = recipe(repo, rev, tmpdir)
|
|
argv = [sys.executable, script] + args
|
|
p = subprocess.run(argv, capture_output=True, timeout=timeout)
|
|
out = p.stdout.decode("utf-8", "replace")
|
|
err = p.stderr.decode("utf-8", "replace")
|
|
# A tool that takes neither `--repo` nor `--rev` answers with an argparse
|
|
# diagnosis naming the flags it rejected, and NOTHING else: measured
|
|
# 2026-10-16, `check_lint.py --repo R --rev HEAD` exits 2 with stdout empty
|
|
# and `unrecognized arguments: --repo R --rev HEAD`, and the first cut of
|
|
# this rule re-ran it as `<script> --repo R` -- still rejected, still rc 2,
|
|
# so a tool with a perfectly good default census was reported RUN_FAILED.
|
|
# The retry therefore drops the tokens argparse itself named, and only
|
|
# those; a tool whose values collide with a dropped flag is a stated
|
|
# limitation, not a silent one.
|
|
m = re.search(r"unrecognized arguments:\s*(.+)$", err, re.M)
|
|
if tag != "no-rev" and p.returncode != 0 and m:
|
|
drop = set(m.group(1).split())
|
|
argv = [x for x in argv if x not in drop]
|
|
p = subprocess.run(argv, capture_output=True, timeout=timeout)
|
|
out = p.stdout.decode("utf-8", "replace")
|
|
err = p.stderr.decode("utf-8", "replace")
|
|
tag = "no-rev"
|
|
if not out.strip():
|
|
out = "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
|
|
return out, p.returncode, tag
|
|
|
|
|
|
def run_selftest(script, timeout=300):
|
|
"""Leg 2: the tool's own `--selftest`, or None if it takes no such flag.
|
|
|
|
A cell count is printed here and nowhere else, which is the whole reason
|
|
this leg exists. `None` means the tool has no selftest to offer -- reported
|
|
as one leg, not as a failure.
|
|
"""
|
|
try:
|
|
p = subprocess.run([sys.executable, script, "--selftest"],
|
|
capture_output=True, timeout=timeout)
|
|
except subprocess.TimeoutExpired:
|
|
return "<--selftest timed out>"
|
|
out = p.stdout.decode("utf-8", "replace")
|
|
err = p.stderr.decode("utf-8", "replace")
|
|
if "unrecognized arguments" in err and "--selftest" in err:
|
|
return None
|
|
if not out.strip():
|
|
return "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
|
|
return out
|
|
|
|
|
|
def unit_scope(unit, legs, no_run):
|
|
"""Is a split in this unit checkable, and why not if it is not?
|
|
|
|
With the oracle legs in hand a unit is checkable when some leg prints it
|
|
with exactly ONE value: the docstring and the output then talk about one
|
|
number. A unit printed with 2+ values is SECTION-SPLIT BY THE TOOL'S OWN
|
|
DESIGN (`citations` prints once under EXPLICIT and once under BARE), so a
|
|
docstring that mirrors both is correct and is reported as such, not flagged.
|
|
A unit no leg prints is not a census unit here. Without a run, only
|
|
uppercase census labels are checked.
|
|
"""
|
|
if no_run or not legs:
|
|
return ("LABEL_ONLY" if is_label(unit) else "UNCHECKED_NO_ORACLE")
|
|
saw_split = False
|
|
for printed, clone, _tag in legs:
|
|
vals = set()
|
|
for k in keys_for(unit, printed or {}):
|
|
vals |= set((printed or {})[k])
|
|
if not vals:
|
|
continue
|
|
if len(vals) > 1:
|
|
saw_split = True
|
|
continue
|
|
return "CHECKABLE"
|
|
if saw_split:
|
|
return "SECTION_SPLIT_BY_TOOL"
|
|
return "NOT_A_CENSUS_UNIT"
|
|
|
|
|
|
def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240):
|
|
text = open(script, encoding="utf-8", errors="replace").read()
|
|
span = docstring_span(script, text)
|
|
if span is None:
|
|
return None
|
|
lines = text.split("\n")
|
|
lo, hi = span
|
|
blk_claims = []
|
|
for first, blk in blocks(lines, lo, hi):
|
|
btext = "\n".join(raw for _, raw in blk)
|
|
revs = block_revs(btext)
|
|
cl = claims_in(blk)
|
|
if cl:
|
|
blk_claims.append((first, revs, cl, btext))
|
|
|
|
# ---- the legs first. Leg 1: the tool's own run, once per revision the
|
|
# docstring names. Leg 2: its --selftest, once. The tool under test is
|
|
# never this tool: running this scanner on itself spawns it, which
|
|
# spawns it again, without a floor.
|
|
is_self = os.path.basename(script) == os.path.basename(os.path.abspath(__file__))
|
|
printed_by_rev, verdicts, failures = {}, [], []
|
|
selftest_leg = None
|
|
if run and not is_self:
|
|
for first, revs, cl, _b in blk_claims:
|
|
for rev in (revs or ["HEAD"]):
|
|
key = (name, rev)
|
|
if key not in cache:
|
|
try:
|
|
out, rc, tag = run_tool(repo, script, rev, tmpdir)
|
|
cache[key] = (parse_printed(out), rc, out, tag)
|
|
except Exception as exc: # noqa: BLE001
|
|
cache[key] = (None, None, f"<run failed: {exc}>", "")
|
|
printed_by_rev.setdefault((first, rev), cache[key])
|
|
skey = (name, "<selftest>")
|
|
if skey not in cache:
|
|
try:
|
|
sout = run_selftest(script)
|
|
except Exception as exc: # noqa: BLE001
|
|
sout = f"<selftest failed: {exc}>"
|
|
cache[skey] = (parse_printed(sout) if sout else None,
|
|
0 if sout else None, sout or "", "selftest")
|
|
selftest_leg = cache[skey]
|
|
|
|
def legs_for(first, rev):
|
|
"""Priority-ordered [(printed, clone, tag)] for one block scope."""
|
|
out = []
|
|
entry = printed_by_rev.get((first, rev))
|
|
if entry and entry[0]:
|
|
out.append((entry[0][0], entry[0][1], entry[3]))
|
|
if selftest_leg and selftest_leg[0]:
|
|
out.append((selftest_leg[0][0], selftest_leg[0][1], "selftest"))
|
|
return out
|
|
|
|
# 1. INTERNAL SPLIT: one unit, one revision scope, two values, far enough
|
|
# apart to be a restatement rather than a neighbouring sub-count.
|
|
splits, unchecked = [], collections.Counter()
|
|
by_unit = collections.defaultdict(list)
|
|
for first, revs, cl, _b in blk_claims:
|
|
scope = tuple(revs)
|
|
for ln, val, unit, _t, pos in cl:
|
|
by_unit[(unit.upper() if unit.isupper() else unit.lower(), scope)].append(
|
|
(first, ln, val, pos))
|
|
for (unit, scope), items in sorted(by_unit.items()):
|
|
vals = {v for _, _, v, _ in items}
|
|
if len(vals) < 2:
|
|
continue
|
|
# a restatement is a pair far apart; an enumeration is neighbours
|
|
far = any(abs(b[3] - a[3]) >= isolation
|
|
for a, b in [(x, y) for x in items for y in items]
|
|
if a[1] != b[1] and a[2] != b[2])
|
|
if not far:
|
|
continue
|
|
rev_for_oracle = (items[0][0], scope[0] if len(scope) == 1 else "HEAD")
|
|
why = ("SELF_NOT_RUN" if is_self and run
|
|
else unit_scope(unit, legs_for(*rev_for_oracle), not run))
|
|
if why != "CHECKABLE" and why != "LABEL_ONLY":
|
|
unchecked[why] += 1
|
|
continue
|
|
splits.append({"unit": unit, "revs": list(scope),
|
|
"values": sorted(vals), "how": why,
|
|
"rows": [{"line": ln, "value": v, "block": b}
|
|
for b, ln, v, _ in items]})
|
|
|
|
# 2. ORACLE: every claim compared against what the tool prints, leg 1 first.
|
|
if run:
|
|
for first, revs, cl, _b in blk_claims:
|
|
for rev in (revs or ["HEAD"]):
|
|
if is_self:
|
|
for ln, val, unit, tok, _pos in cl:
|
|
verdicts.append({"file": name, "block": first, "line": ln,
|
|
"rev": rev, "unit": unit, "claim": val,
|
|
"token": tok, "verdict": "SELF_NOT_RUN",
|
|
"detail": "the tool under test is this tool"})
|
|
continue
|
|
pr, rc, out, _tag = printed_by_rev[(first, rev)]
|
|
printed, clone = pr if pr else (None, None)
|
|
legs = legs_for(first, rev)
|
|
# A leg with no census -- raised, or exited non-zero having
|
|
# printed nothing -- adjudicates nothing either: drop it and let
|
|
# whichever leg survives answer instead. Only when none survives
|
|
# is the block left unadjudicated.
|
|
no_census = pr is None or (rc not in (0, None)
|
|
and not any(printed.values()))
|
|
if pr is None:
|
|
failures.append({"file": name, "block": first, "rev": rev,
|
|
"unit": "-", "claim": None,
|
|
"verdict": "RUN_FAILED", "detail": str(out)[:90]})
|
|
elif no_census:
|
|
failures.append({"file": name, "block": first, "rev": rev,
|
|
"unit": "-", "claim": None,
|
|
"verdict": "RUN_FAILED",
|
|
"detail": "rc=%s, nothing printed: %s"
|
|
% (rc, str(out)[:80])})
|
|
if no_census:
|
|
legs = [l for l in legs if l[2] == "selftest"]
|
|
if not legs:
|
|
continue
|
|
for ln, val, unit, tok, _pos in cl:
|
|
v, detail = adjudicate(legs, unit, val, revs)
|
|
verdicts.append({"file": name, "block": first, "line": ln,
|
|
"rev": rev, "unit": unit, "claim": val,
|
|
"token": tok, "verdict": v, "detail": detail})
|
|
return {"tool": name, "splits": splits, "unchecked": dict(unchecked),
|
|
"failures": failures, "verdicts": verdicts,
|
|
"self": is_self and run, "selftest_leg": (not run) or bool(selftest_leg and selftest_leg[0]),
|
|
"blocks_with_claims": len(blk_claims)}
|
|
|
|
|
|
def counts(rep):
|
|
"""(splits, claim_defects, leg_failures, claims, notes) -- the banner's rows.
|
|
|
|
A LEG FAILURE IS NOT A CLAIM DEFECT. RUN_FAILED says no census came back and
|
|
such a leg adjudicates nothing (see check_tool); STALE says a claim was read
|
|
against a census and disagreed. The first live run of the two-leg tool mixed
|
|
them in one `N STALE/failed` column, so four failed runs read as four bad
|
|
claims -- measured 2026-10-16 at HEAD, check_lint.py 2 and
|
|
queue_claim_check.py 2, none of them a claim disagreement. Two columns, so
|
|
the distinction survives the summary a reader actually reads.
|
|
"""
|
|
stale = [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
|
|
notes = []
|
|
if rep.get("self"):
|
|
notes.append("self: no leg run (SELF_NOT_RUN)")
|
|
elif not rep.get("selftest_leg"):
|
|
notes.append("no --selftest: leg 2 absent")
|
|
return (len(rep["splits"]), len(stale), len(rep["failures"]),
|
|
len(rep["verdicts"]), notes)
|
|
|
|
|
|
def gate_faults(reports):
|
|
"""(splits, stale) -- the two things `--fail` refuses to let pass.
|
|
|
|
The class this tool exists to catch is a docstring claim that disagrees
|
|
with the census printed beside it (STALE), and two claims of one unit that
|
|
disagree with each other (INTERNAL SPLIT). A LEG FAILURE IS NOT IN THIS
|
|
COUNT, for the reason counts() gives: a leg that produced no census
|
|
adjudicated no claim, so counting it would fail a run on a claim never
|
|
checked. rc 2, decided in main(), covers "could not grade" separately.
|
|
"""
|
|
splits = stale = 0
|
|
for rep in reports:
|
|
splits += len(rep.get("splits") or [])
|
|
stale += sum(1 for v in (rep.get("verdicts") or [])
|
|
if v.get("verdict") == "STALE")
|
|
return splits, stale
|
|
|
|
|
|
def gate_rc(reports, fail):
|
|
"""0 when the gate is off or nothing was graded faulty; 1 when it is on and
|
|
a defect claim was graded. Never 2: "could not grade" is main()'s call,
|
|
made before any report exists, and this function cannot see it."""
|
|
if not fail:
|
|
return 0
|
|
splits, stale = gate_faults(reports)
|
|
return 1 if (splits or stale) else 0
|
|
|
|
|
|
def resolve_tools(requested, tree):
|
|
"""(tools, missing). A requested name that is not tracked at the rev is
|
|
REPORTED, never silently dropped: `--tools checkin_check.py` (a bare
|
|
basename instead of a repo-relative path) scanned zero tools and exited 0,
|
|
which reads exactly like a clean pass. Measured 2026-10-18 at 8932d06."""
|
|
if not requested:
|
|
return sorted(p for p in tree
|
|
if p.startswith("tools/") and p.endswith(".py")), []
|
|
return list(requested), [t for t in requested if t not in tree]
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser(description=__doc__,
|
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--repo", default=DEFAULT_REPO)
|
|
ap.add_argument("--rev", default="HEAD")
|
|
ap.add_argument("--tools", nargs="*")
|
|
ap.add_argument("--json")
|
|
ap.add_argument("--no-run", action="store_true",
|
|
help="internal-split verdicts only; no subprocess")
|
|
ap.add_argument("--fail", action="store_true",
|
|
help="rc 1 if any tool reports an INTERNAL SPLIT or a STALE "
|
|
"claim; leg failures are reported, never spent as "
|
|
"defects (rc 2 stays 'could not grade')")
|
|
ap.add_argument("--selftest", action="store_true")
|
|
a = ap.parse_args()
|
|
|
|
if a.selftest:
|
|
return _fixture_checks()
|
|
|
|
store = git(a.repo, "ls-tree", "-r", "--name-only", a.rev)
|
|
tree = {p for p in store.decode("utf-8", "replace").split("\n") if p}
|
|
tools, missing = resolve_tools(a.tools, tree)
|
|
if not tools:
|
|
print(f"no tools found at {a.rev}", file=sys.stderr)
|
|
return 2
|
|
if missing:
|
|
print(f"cannot grade: --tools names nothing tracked at {a.rev}: "
|
|
f"{', '.join(missing)}", file=sys.stderr)
|
|
print(" names are repo-relative, e.g. --tools tools/checkin_check.py; "
|
|
"a filter that matches nothing is not a clean scan", file=sys.stderr)
|
|
return 2
|
|
|
|
cache, reports = {}, []
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-") as tmpdir:
|
|
for t in tools:
|
|
src = subprocess.run(["git", "-C", a.repo, "show", f"{a.rev}:{t}"],
|
|
capture_output=True).stdout
|
|
local = os.path.join(tmpdir, os.path.basename(t))
|
|
with open(local, "wb") as fh:
|
|
fh.write(src)
|
|
try:
|
|
rep = check_tool(a.repo, local, t, cache, run=not a.no_run,
|
|
tmpdir=tmpdir)
|
|
except subprocess.TimeoutExpired:
|
|
rep = {"tool": t, "splits": [], "blocks_with_claims": 0,
|
|
"verdicts": [{"file": t, "verdict": "TIMEOUT", "rev": "-",
|
|
"unit": "-", "claim": None, "block": 0}]}
|
|
if rep:
|
|
reports.append(rep)
|
|
|
|
print(f"tools scanned at {a.rev}: {len(reports)} "
|
|
f"({'internal-split only, label vocabulary' if a.no_run else 'docstring vs printed census'})")
|
|
tot = collections.Counter()
|
|
for rep in reports:
|
|
bad = rep["failures"] + [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
|
|
nsplits, nstale, nfail, nclaims, notes = counts(rep)
|
|
print(f"\n{rep['tool']}: {nsplits} INTERNAL SPLIT, "
|
|
f"{nstale} STALE claim(s), {nfail} leg failure(s), "
|
|
f"{nclaims} claims checked"
|
|
+ (" [" + "; ".join(notes) + "]" if notes else ""))
|
|
for s in rep["splits"]:
|
|
vals = ", ".join(f"{r['value']}@{r['line']}" for r in s["rows"])
|
|
scope = ",".join(s["revs"]) or "no rev named"
|
|
print(f" SPLIT {s['unit']} [{scope}] ({s['how']}): {vals}")
|
|
for k, c in sorted(rep["unchecked"].items()):
|
|
print(f" (not flagged: {c} split(s) in units {k})")
|
|
for v in bad:
|
|
print(f" {v['verdict']:10s} {v['file']}:{v.get('line', '-')} "
|
|
f"{v['unit']} claimed {v['claim']} at {v['rev']} -- {v['detail']}")
|
|
for k in sorted({v["verdict"] for v in rep["verdicts"]}):
|
|
if k in ("REV_UNCHECKED", "SELF_NOT_RUN", "CLONE_LOCAL"):
|
|
n = sum(1 for v in rep["verdicts"] if v["verdict"] == k)
|
|
print(f" ({n} {k}: reported, never counted as a defect)")
|
|
for k, c in collections.Counter(v["verdict"] for v in rep["verdicts"]).items():
|
|
tot[k] += c
|
|
print("\n--- verdict totals (unit: claims) ---")
|
|
for k in sorted(tot):
|
|
print(f" {k:12s} {tot[k]}")
|
|
if a.json:
|
|
with open(a.json, "w") as fh:
|
|
json.dump({"rev": a.rev, "repo": a.repo, "reports": reports}, fh, indent=1)
|
|
print(f"wrote {a.json}")
|
|
if a.fail:
|
|
nsplit, nstale = gate_faults(reports)
|
|
if nsplit or nstale:
|
|
print(f"\nFAIL: {nsplit} INTERNAL SPLIT + {nstale} STALE claim(s) "
|
|
f"at {a.rev}"
|
|
+ (" (internal-split only: --no-run)" if a.no_run else ""))
|
|
return 1
|
|
print(f"\nOK: 0 INTERNAL SPLIT, 0 STALE claim(s) at {a.rev}"
|
|
+ (" (internal-split only: --no-run)" if a.no_run else ""))
|
|
return gate_rc(reports, a.fail)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|