volt
fcfb323180
A split enters the graded list only via CHECKABLE (a leg ran and printed the
unit with one value). Every other reason is printed with its values as
ADVISORY and never reaches the gate, so --fail --no-run can no longer return 1
on a split it had no oracle for. Ruling: reply 22, 2026-10-18.
Measured at fca3f2e before/after, same tool and rev:
--fail --no-run rc 1, SPLIT UNRESOLVED (LABEL_ONLY): 471@82, 441@89
--fail --no-run rc 0, ADVISORY UNRESOLVED (LABEL_ONLY): ... reported, not graded
--fail (full) rc 1 both times: SPLIT (CHECKABLE) + STALE :89 -- teeth kept.
Selftest 52 -> 60 cells (6 mode cells, 2 process-exit cells); sweep 12/12.
1254 lines
64 KiB
Python
1254 lines
64 KiB
Python
#!/usr/bin/env python3
|
|
"""doc_census_check.py -- does a tool's own docstring census match the census
|
|
that tool PRINTS?
|
|
|
|
WHY THIS EXISTS
|
|
This archive's three scanners each carry a census table in their module
|
|
docstring: counts that a reader cites, that no command checks. The docstring
|
|
is prose; prose is not executed. On 2026-10-16 the control plane returned
|
|
exactly this defect against a sibling scanner, in its own words:
|
|
|
|
"The scanner's own docstring carries a number that does not match the
|
|
census it prints -- the exact class the tool exists to catch."
|
|
|
|
tools/citation_resolve_check.py said `471 UNRESOLVED` on one line and
|
|
`441 UNRESOLVED` nine lines later, over a census the tool prints as 471.
|
|
Nothing was checking the docstring against the output beside it. That is this
|
|
tool's whole job.
|
|
|
|
WHAT IT MEASURES (read-only: this tool only writes --json)
|
|
Two independent verdicts per tool, and they do not share evidence.
|
|
|
|
1. INTERNAL SPLIT -- one unit, one scope, two values. Within one docstring,
|
|
every numeric claim of the form `<value> <unit>` is collected; two claims
|
|
of ONE unit that a single revision scope covers, carrying two different
|
|
values, is a split. This catches the 441/471 instance. A split is GRADED
|
|
only when a leg verified the unit (`CHECKABLE`: the run printed it with
|
|
exactly one value); every other split -- no oracle (`LABEL_ONLY`,
|
|
`--no-run`), a unit no leg prints (`NOT_A_CENSUS_UNIT`), a unit the tool
|
|
prints per section (`SECTION_SPLIT_BY_TOOL`) -- is printed with its values
|
|
as ADVISORY and never reaches the gate. 2026-10-18: the old build failed
|
|
`--fail --no-run` on the 471/441 split at fca3f2e (rc 1) while the full run
|
|
could not see either value, so rc 1 meant "could not check this".
|
|
|
|
2. ORACLE -- the tool is RUN, at the revision its own docstring names, and
|
|
each claim is compared against the value that run prints for that unit.
|
|
MATCHES claim == printed value
|
|
STALE claim != printed value, and the printed unit is not
|
|
declared clone-local -- a defect claim
|
|
CLONE_LOCAL claim != printed value but the tool's own output marks
|
|
that unit clone-local ("blobs in THIS clone's local object
|
|
store"), so drift is the documented expectation, not a
|
|
defect. Reported, never counted as one.
|
|
AMBIGUOUS the tool prints that unit with 2+ values at that rev
|
|
(e.g. `citations` prints once per section), so there is no
|
|
single oracle value; both are printed and no claim is made
|
|
NOT_PRINTED the unit does not appear in this tool's output at all
|
|
|
|
A claim is `<digits> <unitword>` with the unit word ADJACENT, which is the
|
|
archive's own notation. A bare number is not a claim and is not checked.
|
|
|
|
TWO ORACLE LEGS, AND WHY THE SECOND ONE EXISTS
|
|
Leg 1 is the tool's own run at the block's revision, exactly as above. Leg 2
|
|
is the SAME TOOL'S `--selftest`, added 2026-10-16 after the control plane
|
|
returned a docstring claim of `13 cells` against a selftest that prints `18
|
|
cells` -- the defect class this tool exists to catch, sitting inside its own
|
|
default scope and still read as NOT_PRINTED, because no leg ever passed
|
|
`--selftest` and a cell count is printed by nothing else. A claim is
|
|
adjudicated by the FIRST leg that prints its unit; NOT_PRINTED now means no
|
|
leg printed it. A tool that takes no `--selftest` simply has one leg -- a fact
|
|
about the tool, not a defect in it.
|
|
|
|
A leg that exits non-zero having printed nothing is reported RUN_FAILED, never
|
|
read as an empty census, and it adjudicates nothing either: the block's claims
|
|
are still read off any surviving leg -- leg 2 -- and are left unadjudicated
|
|
only when no leg survives, because a failed run is not a reason to stop
|
|
looking. A revision that will not materialise fails the same way rather than
|
|
producing a census at all: an empty queue is a census of zeros, and reading it
|
|
as that revision's census is the phantom this rule refuses.
|
|
|
|
A HEX RUN INTRODUCED AS A DIGEST IS NOT A REVISION. A block that quotes a
|
|
truncated sha256 in its prose is a size, not a place: read as a rev it files
|
|
the block under a revision that cannot exist, fails that leg, and then
|
|
adjudicates every claim in the block again beside the real ones. Two signals
|
|
are read, both local to the token -- a digest word in front of it (`sha256
|
|
<hex>`, `hash <hex>`), or an ellipsis behind it (`<hex>…`), which is how this
|
|
archive writes a truncated digest. See `block_revs`.
|
|
|
|
The run leg is `<script> --repo R --rev REV`. A tool that takes no `--rev`
|
|
answers it with an argparse diagnosis and prints no census at all, so the leg
|
|
is re-run with the rejected flags DROPPED -- the exact tokens argparse named,
|
|
and only those -- and is tagged `no-rev`; a recipe's leg may carry the same tag
|
|
for the same reason. Such a tool measures the tree it is POINTED AT, so a claim
|
|
in a block that names a revision is reported REV_UNCHECKED -- never counted as
|
|
verified against that revision, and never counted as a defect -- while a block
|
|
that names no revision is checked normally, since the default is what the flag
|
|
means there.
|
|
|
|
THIS TOOL DOES NOT RUN ITSELF. The default scope is every `tools/*.py` in the
|
|
tree, and this tool is one of them. Running it on itself spawns it, which
|
|
spawns it again, without a floor -- measured, not reasoned, in the check-in of
|
|
2026-10-16. The self target is therefore excluded from the run leg and every
|
|
one of its claims is reported SELF_NOT_RUN. It is deliberately still listed in
|
|
the scanned set, so the exclusion is a printed row and not a silence.
|
|
|
|
SCOPE, STATED RATHER THAN IMPLIED
|
|
Only module docstrings of .py tools are scanned: a docstring is where a census
|
|
table lives. Only revisions NAMED IN THE BLOCK are used as oracles -- a block
|
|
naming no revision is checked against the tool's default (HEAD) and tagged, so
|
|
a stale rev is never silently attributed. A tool whose subject is not `--repo`
|
|
+ `--rev` is driven through a per-tool recipe declared below, with a temp file
|
|
that is removed: `queue_claim_check.py` takes a materialised QUEUE file and so
|
|
its leg IS the revision, while `append_only_owners.py` takes a tree scan that
|
|
no flag can pin to a revision, so its leg is tagged `no-rev`. A tool no recipe
|
|
covers, and that answers `--repo ... --rev ...` with an argparse diagnosis, is
|
|
re-run with the rejected flags dropped and tagged the same way -- measured at
|
|
HEAD on 2026-10-16 against `check_lint.py`, which takes neither flag and whose
|
|
default census is a real one.
|
|
|
|
WHAT THIS DOES NOT CHECK, and the measured size of the gap. A claim is only
|
|
adjudicated when the unit it names is a unit the tool PRINTS. Over the three
|
|
scanners at d355395 the run read ninety-two claims (with the pre-2026-10-18
|
|
separator rule, which read punctuation and list markers as claims): twelve
|
|
were verified against the run they point at, eight were in units the tool
|
|
prints with two values (SECTION_SPLIT_BY_TOOL -- no oracle), and seventy-one
|
|
were in units the tool never prints at all. So the oracle's reach is narrow
|
|
and its precision is the reason: it reported exactly one bad claim on that
|
|
corpus, and that one was the returned defect. A wider rule was tried first and
|
|
falsified by its own first run -- num_agreement_check.py flagged a hundred and
|
|
seventeen restatements across the archive, too coarse to be read -- so the
|
|
narrow rule is the one that ships. Two claim forms are read (`<value> <unit>` and
|
|
`<unit>: <value>`); a claim written in neither form is not seen, and that is a
|
|
stated gap, not a silent one.
|
|
|
|
NO CENSUS OF THIS TOOL'S OWN VERDICTS APPEARS IN THIS DOCSTRING, deliberately:
|
|
this scanner scans itself, so a count of its own output written here would
|
|
have to agree with the output of the run that the same commit changes. That is
|
|
the limit citation_hash_class.py records in the same words -- a commit cannot
|
|
measure the file it introduces -- and here it is avoided rather than
|
|
discovered.
|
|
|
|
FAILING ON THE CLASS IT CATCHES (`--fail`)
|
|
Without it the process exits 0 while printing STALE rows, so no gate could
|
|
act on the one class this tool exists to catch without parsing its text.
|
|
`--fail` makes the exit code carry the verdict. The cost of the old silence
|
|
is measured twice, and the two are NOT the same measurement:
|
|
|
|
- STRUCTURAL -- reproducible right here, no archive needed. A stub tree
|
|
whose docstring says `99 widgets` beside a printed `widgets: 7` exits 0
|
|
without `--fail` and 1 with it (`gate e2e` cells in `--selftest`).
|
|
|
|
- HISTORICAL -- true only of the pre-tightening separator (`ac0e211^`).
|
|
That build at rev 4ce1aed, `--tools tools/checkin_check.py`, exits 0
|
|
with two STALE verdicts on `checkin_check.py:59`: `Never claimed 8` and
|
|
`never claimed 1`, both `printed 2 [selftest]` -- the digits in "Never 1,
|
|
never a traceback". Reproduced 2026-10-18 from this host. The tightened
|
|
separator stops reading those digits as a census: the same rev now
|
|
adjudicates 7 claims, 0 STALE. So the historical run is cited with its
|
|
build and never as current behaviour -- a claim that says "rc 0 with two
|
|
STALE" without naming the build goes stale the moment the reader is
|
|
tightened, which is the failure this tool grades other files for.
|
|
|
|
The code means:
|
|
|
|
0 ran, and graded no fault (every claim adjudicated agrees)
|
|
1 ran, and graded a fault (a VERIFIED INTERNAL SPLIT, or STALE)
|
|
2 could not grade (usage, a --tools filter that matches
|
|
nothing tracked, unreadable input)
|
|
|
|
THE TWO MODES GRADE THE SAME CLASS. `--no-run` runs nothing, so it has no
|
|
oracle and cannot prove any split is a census unit: it prints every split as
|
|
ADVISORY with its values, and `--fail --no-run` exits 0. That is the ruling
|
|
of reply 22 (2026-10-18), and it fixes a real asymmetry: `--fail --no-run`
|
|
returned 1 on `citation_resolve_check.py` at fca3f2e (471@82 / 441@89) while
|
|
the same units were advisory under a full run. Measured after the repair at
|
|
the same rev, one mode apart: `--fail --no-run` rc 0 with the pair printed as
|
|
ADVISORY; `--fail` (full) rc 1 on the same split, now CHECKABLE because the
|
|
run prints `UNRESOLVED 471` -- plus the STALE claim at :89. The teeth are
|
|
kept where they can be verified, which is the only place they mean anything.
|
|
|
|
rc 2 keeps its meaning from the tools/ contract (6ee8fce) and is decided
|
|
before the gate runs, so "could not grade" never reads as "graded clean" or
|
|
as "graded a fault". A LEG FAILURE (RUN_FAILED) is reported in its own column
|
|
and does not fail the gate, for the reason counts() states: a leg that
|
|
produced no census adjudicated no claim. A gate that counted it would fail
|
|
runs it never checked.
|
|
|
|
USAGE
|
|
python3 doc_census_check.py [--repo DIR] [--rev HEAD] [--tools NAME...]
|
|
[--json OUT] [--no-run] [--fail] [--selftest] # selftest: 60 cells, one per claim class
|
|
"""
|
|
import argparse
|
|
import ast
|
|
import collections
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
|
|
DEFAULT_REPO = "/root/troupe-standup"
|
|
|
|
# A claim, in either of the two forms this archive writes:
|
|
# `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations"
|
|
# `<unit> [: =] <value>` -- "ACS = 72", "overlap = 20" (markdown bold allowed)
|
|
# A bare number is not a claim. Deliberately not "any word": `471 cannot be
|
|
# resolved` is prose, and a checker that flags prose is a checker nobody reads.
|
|
#
|
|
# THE SEPARATOR IS SPACE, OR A CLOSING DELIMITER AND A SPACE -- tightened
|
|
# 2026-10-18 after the census was read off every tool at HEAD (140 claims) and
|
|
# the separator vocabulary was printed: `''` (11 claims, a hex run read as
|
|
# digits+word: `83acffe0` -> 83 `acffe0`), `'. '` (5, an ordinal or a sentence
|
|
# end: `1. INTERNAL`, `615. Cite`), `'-'` (5, a compound adjective: `120-char`,
|
|
# `1-based`), `' '` (4, an aligned list marker: `0 ran`), and the punctuation
|
|
# spills `'; '` `' "'` `'+ '` `') '`. Only 95 of the 140 were a count and its
|
|
# unit. The loose class fired two STALE verdicts in the full run at 4ce1aed
|
|
# against `checkin_check.py:59` -- "Never 1, never a traceback", where the digit
|
|
# is a code, not a census -- which is a false defect claim of exactly the kind
|
|
# this archive grades lanes for. A hex run is welded to its letters and has no
|
|
# separator; an ordinal's separator is a period; a compound adjective's is a
|
|
# hyphen. None of the three is `<value> <unit>`.
|
|
#
|
|
# A HYPHEN AFTER THE UNIT IS THE SAME REFUSAL, made 2026-10-18 after the
|
|
# --fail gate went red at HEAD on a false claim: `queue_claim_check.py`'s axis
|
|
# list says `39 ACS-only lines carry no Q<digits> token`, `12 ACS-only lines
|
|
# are indented`, `1 ACS-only line carries a BQ exclusion phrase` -- three
|
|
# sub-counts of ONE population of 52, and the token read as a unit was the
|
|
# adjective `ACS` in `ACS-only`. With no oracle to test the label against, a
|
|
# `--no-run` split fired on 39/12/1. A hyphen immediately behind the unit word
|
|
# marks a compound adjective, not a unit noun, exactly as a hyphen behind the
|
|
# number does (`120-char`). The cost is stated: a genuinely hyphenated unit
|
|
# (`citations-per-block`) is no longer read, and the invariant that licenses
|
|
# this tightening is that the 26 MATCHES do not move -- measured before and
|
|
# after, same 26 rows.
|
|
CLAIM = re.compile(
|
|
r"(?<![0-9A-Za-z_.])(?P<num>[0-9]+(?:,[0-9]{3})*)(?P<sep>\s{1,2}|[)\]}\"'`*]\s)"
|
|
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Z_]{2,})\b(?!-)")
|
|
CLAIM_EQ = re.compile(
|
|
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*\**\s*(?P<num>[0-9]+(?:,[0-9]{3})*)")
|
|
REV_LONG = re.compile(r"\b[0-9a-f]{7,40}\b")
|
|
|
|
# A hex run introduced as a DIGEST is not a revision, and reading it as one costs
|
|
# more than a wrong label: the block is filed under a revision that cannot exist,
|
|
# its leg fails, and then every claim in the block is adjudicated once per
|
|
# spurious rev beside the real ones. Measured 2026-10-16 at HEAD on
|
|
# `tools/queue_claim_check.py`, whose UNITS block quotes the queue's size as
|
|
# "/ sha256 `fbe51445a89f81ab…`": three claims emitted FOUR times, once per
|
|
# REV_LONG token, and two `git show <sha256-prefix>:WORK-QUEUE.md` leg failures.
|
|
# Two signals, both local to the token: a digest word in front of it (sha256,
|
|
# sha1, md5, digest, hash), or an ellipsis behind it -- the archive's own way of
|
|
# writing a truncated digest. A revision cited in the ordinary way carries
|
|
# neither, and a FULL sha256 is already invisible to REV_LONG: the run has no
|
|
# word boundary to bite on.
|
|
DIGEST_BEFORE = re.compile(
|
|
r"(?:sha[-\s]?256|sha[-\s]?1|md5|digest|hash)\s*[:=]?\s*[`'\"]?\s*$", re.I)
|
|
DIGEST_AFTER = re.compile(r"^[`'\"]?\s*(?:…|\.\.\.)")
|
|
|
|
|
|
def block_revs(text):
|
|
"""The revisions a block names -- hex runs read as digests excluded."""
|
|
out = set()
|
|
for m in REV_LONG.finditer(text):
|
|
if DIGEST_BEFORE.search(text[:m.start()]) or \
|
|
DIGEST_AFTER.match(text[m.end():]):
|
|
continue
|
|
out.add(m.group(0))
|
|
return sorted(out)
|
|
|
|
# Function words and units this tool does not adjudicate. `at` in "2026-10-15 at
|
|
# rev 9188dd3" is not a census quantity; neither is `B` (bytes differ by
|
|
# revision by design, and "1,201,550 B" is a size, not a count). Stated so the
|
|
# exclusion is a rule rather than a habit.
|
|
STOPWORDS = {
|
|
"at", "the", "of", "in", "is", "are", "was", "and", "or", "to", "for", "on",
|
|
"by", "as", "it", "its", "that", "this", "these", "those", "with", "from",
|
|
"no", "not", "one", "two", "three", "all", "any", "be", "been", "has",
|
|
"have", "had", "his", "her", "our", "your", "their", "but", "if", "then",
|
|
"than", "so", "up", "out", "off", "over", "into", "run", "runs", "may",
|
|
"can", "same", "each", "both", "per", "via", "b", "kb", "mb", "gb",
|
|
}
|
|
|
|
# How to run each tool at a revision. Any tool not listed here is run as
|
|
# `<script> --repo R --rev REV`. A recipe returns `(argv, tag)` and the tag says
|
|
# whether the recipe could materialise the revision the block names.
|
|
#
|
|
# queue_claim_check.py takes no --rev: its subject is a FILE, and the file is
|
|
# materialised at the revision and handed to it -- so the leg IS that rev.
|
|
# A revision that will NOT materialise is a raised failure, never an empty
|
|
# file: an empty queue is a census of zeros, and a census of zeros reads as
|
|
# that revision's census. Measured 2026-10-16 -- `git show
|
|
# c3b4a290dc64bd24:WORK-QUEUE.md` (a sha256 prefix REV_LONG read as a rev)
|
|
# exits 128, and the leg it fed printed `0 opening records` at rc 0.
|
|
# append_only_owners.py takes a positional AUDIT.json and nothing else: its
|
|
# input is a tree scan, which its sibling produces over the WORKING TREE
|
|
# (`append_only_audit.py` has no --rev either). This scanner is read-only, so
|
|
# it will not add a git worktree to the repository it is only allowed to
|
|
# read: the leg is tagged no-rev and every claim in a block that names a
|
|
# revision is reported REV_UNCHECKED, never adjudicated against a tree it
|
|
# did not stand on.
|
|
def _recipe_queue(repo, rev, tmpdir):
|
|
dst = os.path.join(tmpdir, f"WORK-QUEUE.{rev}.md")
|
|
p = subprocess.run(["git", "-C", repo, "show", f"{rev}:WORK-QUEUE.md"],
|
|
capture_output=True)
|
|
if p.returncode != 0:
|
|
raise RuntimeError("git show %s:WORK-QUEUE.md -> rc=%d: %s"
|
|
% (rev, p.returncode,
|
|
p.stderr.decode("utf-8", "replace").strip()[:120]))
|
|
with open(dst, "wb") as fh:
|
|
fh.write(p.stdout)
|
|
return ["--queue", dst, "--census"], ""
|
|
|
|
|
|
def _recipe_owners(repo, rev, tmpdir):
|
|
dst = os.path.join(tmpdir, f"append-only-audit.{rev}.json")
|
|
subprocess.run([sys.executable,
|
|
os.path.join(repo, "tools", "append_only_audit.py"),
|
|
"--repo", repo, "--json", dst], capture_output=True)
|
|
return [dst], "no-rev"
|
|
|
|
|
|
RECIPES = {"queue_claim_check.py": _recipe_queue,
|
|
"append_only_owners.py": _recipe_owners}
|
|
|
|
|
|
def git(repo, *a):
|
|
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
|
|
|
|
|
|
def docstring_span(path, text):
|
|
try:
|
|
tree = ast.parse(text)
|
|
except SyntaxError:
|
|
return None
|
|
if not tree.body or not isinstance(tree.body[0], ast.Expr):
|
|
return None
|
|
node = tree.body[0]
|
|
if not isinstance(node.value, ast.Constant) or not isinstance(node.value.value, str):
|
|
return None
|
|
return node.lineno, node.end_lineno
|
|
|
|
|
|
def blocks(lines, lo, hi):
|
|
"""Blank-line-delimited paragraphs -- the same scope the sibling scanners
|
|
use (citation_resolve_check.py for inheritance, num_agreement_check.py for
|
|
quantity agreement), so one paragraph is one scope in all three."""
|
|
start, buf = lo, []
|
|
for ln in range(lo, hi + 1):
|
|
raw = lines[ln - 1]
|
|
if raw.strip() == "":
|
|
if buf:
|
|
yield start, buf
|
|
buf, start = [], ln + 1
|
|
else:
|
|
buf.append((ln, raw))
|
|
if buf:
|
|
yield start, buf
|
|
|
|
|
|
# ------------------------------------------------- printed census -> oracle
|
|
def parse_printed(stdout):
|
|
"""stdout -> {unit: {value: [line, ...]}} plus the set of clone-local units.
|
|
|
|
Four line shapes are read, all of them the scanners' own:
|
|
` RESOLVES 113` indented label then count (family rows)
|
|
` citations: 410` label then count
|
|
`corpus: 307 files at ...` count then unit
|
|
`overlap A&B = 20 ; ...` inline assignments, split on the separators
|
|
A unit whose printed line (or the line above it, for a wrapped phrase) says
|
|
"clone" is clone-local: its count is expected to move with the clone.
|
|
"""
|
|
units = collections.defaultdict(lambda: collections.defaultdict(list))
|
|
clone = set()
|
|
lines = stdout.split("\n")
|
|
|
|
def add(unit, value, ln, cloneline):
|
|
u = unit.strip().lower()
|
|
if len(u) < 3 or not re.fullmatch(r"[a-z_][a-z0-9_]*", u) or u in STOPWORDS:
|
|
return
|
|
units[u][value].append(ln)
|
|
if "clone" in cloneline.lower():
|
|
clone.add(u)
|
|
|
|
for i, raw in enumerate(lines, 1):
|
|
ctx = raw + " " + (lines[i - 2] if i >= 2 else "")
|
|
m = re.match(r"^\s{2,}([A-Za-z_][A-Za-z0-9_]*)\s+([0-9][0-9,]*)\s*$", raw)
|
|
if m: # family row: label count
|
|
add(m.group(1), int(m.group(2).replace(",", "")), i, ctx)
|
|
continue
|
|
m = re.match(r"^\s*([A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*([0-9][0-9,]*)\b", raw)
|
|
if m: # label: count
|
|
add(m.group(1), int(m.group(2).replace(",", "")), i, ctx)
|
|
for m in re.finditer(r"([0-9][0-9,]*)\s+(?:[a-z]+ )?([A-Za-z_][A-Za-z0-9_]*)\b", raw):
|
|
add(m.group(2), int(m.group(1).replace(",", "")), i, ctx)
|
|
for seg in re.split(r"[;]", raw): # inline `X = N`
|
|
m = re.match(r"^\s*(.*?)\s*[:=]\s*([0-9][0-9,]*)\s*$", seg)
|
|
if not m or not m.group(1):
|
|
continue
|
|
words = re.findall(r"[A-Za-z_][A-Za-z0-9_]*", m.group(1))
|
|
if not words:
|
|
continue
|
|
val = int(m.group(2).replace(",", ""))
|
|
add("_".join(w.lower() for w in words), val, i, ctx) # whole phrase
|
|
add(words[-1], val, i, ctx) # the nearest label: `overlap A&B`
|
|
# the scanners' own unit declaration: `... 6961 (unit: occurrences)`.
|
|
# The unit is named AFTER its number, which no other rule reads.
|
|
for m in re.finditer(r"\(\s*unit\s*:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\)", raw):
|
|
left = raw[:m.start()]
|
|
nums = re.findall(r"([0-9][0-9,]*)\s*$", left.rstrip())
|
|
if nums:
|
|
add(m.group(1), int(nums[-1].replace(",", "")), i, ctx)
|
|
return {k: dict(v) for k, v in units.items()}, clone
|
|
|
|
|
|
def ordinal_at(raw, pos):
|
|
"""True when `pos` is the number that OPENS this line and 2+ spaces follow.
|
|
|
|
That is a list marker -- the exit-code block's `0 ran; nothing to report`,
|
|
`1 ran, and graded ...`, `2 could not grade` -- and the word after it is
|
|
prose, not a unit. The same shape mid-line (`WORK-QUEUE.md:8003 cites`) is
|
|
a citation's line number, not a count, and is excluded by the same rule.
|
|
"""
|
|
m = re.match(r"[ \t]*([0-9][0-9,]*)([ \t]{2,})", raw)
|
|
return bool(m) and m.start(1) == pos
|
|
|
|
|
|
def claims_in(blk):
|
|
"""[(line, value, unit, token, offset)] for one block.
|
|
|
|
Both claim forms are read and a claim found by both is reported once. The
|
|
offset is measured on the block text so a RESTATEMENT (far apart) can be
|
|
told from an ENUMERATION of neighbouring sub-counts, the same
|
|
discrimination num_agreement_check.py applies.
|
|
"""
|
|
out, off = [], 0
|
|
for ln, raw in blk:
|
|
found = {}
|
|
for m in CLAIM.finditer(raw):
|
|
u = m.group("unit")
|
|
if len(u) < 3 or u.lower() in STOPWORDS or re.fullmatch(r"[qQ]\d+", u):
|
|
continue
|
|
if ordinal_at(raw, m.start()):
|
|
continue
|
|
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
|
|
for m in CLAIM_EQ.finditer(raw):
|
|
u = m.group("unit")
|
|
if (len(u) < 3 or u.lower() in STOPWORDS
|
|
or re.fullmatch(r"[qQ]\d+", u) or m.start() in found):
|
|
continue
|
|
if m.start() and re.match(r"[0-9A-Za-z_.]", raw[m.start() - 1]):
|
|
continue # `WORK-QUEUE.md:8586` is a citation
|
|
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
|
|
for pos in sorted(found):
|
|
val, u, tok = found[pos]
|
|
out.append((ln, val, u, tok, off + pos))
|
|
off += len(raw) + 1
|
|
return out
|
|
|
|
|
|
def is_label(unit):
|
|
"""An uppercase census label -- the vocabulary a split is checked in when
|
|
the tool has not been run (no oracle available)."""
|
|
return len(unit) >= 3 and re.fullmatch(r"[A-Z][A-Z_]*", unit) is not None
|
|
|
|
|
|
def keys_for(unit, printed):
|
|
"""Candidate oracle keys for a claimed unit word."""
|
|
cands = [unit, unit.lower(), unit.lower().rstrip("s"), unit.upper()]
|
|
seen, out = set(), []
|
|
for c in cands:
|
|
if c.lower() in printed and c.lower() not in seen:
|
|
seen.add(c.lower())
|
|
out.append(c.lower())
|
|
return out
|
|
|
|
|
|
def oracle(printed, clone, unit, value, tag=""):
|
|
"""(verdict, detail) for one claim; `printed` is one revision's map.
|
|
|
|
`tag` names the leg a verdict came from when it is not leg 1 ("no-rev",
|
|
"selftest"): a reader can then tell which census the number was read off.
|
|
"""
|
|
keys = keys_for(unit, printed)
|
|
vals = set()
|
|
for k in keys:
|
|
vals |= set(printed[k])
|
|
if not vals:
|
|
return "NOT_PRINTED", ""
|
|
if len(vals) > 1:
|
|
return "AMBIGUOUS", "printed: " + ", ".join(str(v) for v in sorted(vals))
|
|
printed_val = vals.pop()
|
|
if printed_val == value:
|
|
return "MATCHES", ""
|
|
if any(k in clone for k in keys):
|
|
return "CLONE_LOCAL", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
|
|
return "STALE", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
|
|
|
|
|
|
def adjudicate(legs, unit, value, block_revs):
|
|
"""The first leg that prints the unit adjudicates the claim.
|
|
|
|
`legs` is [(printed, clone, tag)] in priority order: leg 1 (the run) first,
|
|
then the selftest leg. NEITHER a `no-rev` leg NOR the selftest leg is a
|
|
verifier for a block that names a revision: one measured a different tree,
|
|
the other measured the tool's own fixtures. Such a claim is reported
|
|
REV_UNCHECKED -- visible, counted as neither a match nor a defect.
|
|
"""
|
|
for printed, clone, tag in legs:
|
|
if not printed or not keys_for(unit, printed):
|
|
continue
|
|
if block_revs and tag == "no-rev":
|
|
return "REV_UNCHECKED", ("block names %s; the tool takes no --rev, so this "
|
|
"leg measured the working tree"
|
|
% ",".join(block_revs))
|
|
if block_revs and tag == "selftest":
|
|
return "REV_UNCHECKED", ("block names %s; the selftest leg prints this "
|
|
"tool's own fixtures, not a census at that "
|
|
"revision" % ",".join(block_revs))
|
|
return oracle(printed, clone, unit, value, tag)
|
|
return "NOT_PRINTED", ""
|
|
|
|
|
|
# --------------------------------------------------------------- self-test
|
|
# Synthetic stdout, so the parsers and the verdicts are executed, not asserted.
|
|
FAKE_STDOUT = """\
|
|
corpus: 297 markdown files at babdadc in /root/troupe-standup
|
|
--- EXPLICIT `<path>:<line>` -- a CLAIM (unit: citations) ---
|
|
citations: 410
|
|
RESOLVES 113
|
|
STALE_LINE 0
|
|
--- BARE `:<line>` -- a CANDIDATE, never a claim (unit: citations) ---
|
|
citations: 1612
|
|
UNRESOLVED 471
|
|
file mentions with no line: 6961 (unit: occurrences) -- these seed the scope
|
|
index: 871 blobs (unit: blobs in THIS clone's local object store), 680 commits
|
|
overlap A&B = 20 ; ACS-only = 52
|
|
"""
|
|
FAKE_DOC = """\
|
|
THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
|
|
BARE 1612 citations: 255 EXTERNAL_NAMED / 471 UNRESOLVED.
|
|
Read every figure as a citation count, never as a defect count: 441 UNRESOLVED
|
|
is not 441 errors.
|
|
|
|
A SECOND BLOCK, no revision named, quoting 471 UNRESOLVED and nothing else.
|
|
"""
|
|
|
|
# Leg 2's shape: a selftest summary line, and nothing else. The cell count is
|
|
# printed here and nowhere else, which is why a claim about it needs this leg.
|
|
FAKE_SELFTEST = """\
|
|
c1 a NEW byte PASS
|
|
selftest: 18 cells, 18 passed, 0 failed
|
|
"""
|
|
|
|
|
|
def _fixture_checks():
|
|
printed, clone = parse_printed(FAKE_STDOUT)
|
|
ok = True
|
|
cells, bad = [0], [0]
|
|
|
|
def check(name, got, want):
|
|
nonlocal ok
|
|
good = got == want
|
|
cells[0] += 1
|
|
bad[0] += 0 if good else 1
|
|
ok = ok and good
|
|
print(f"{'PASS' if good else 'FAIL'} {name}")
|
|
if not good:
|
|
print(f" got {got}")
|
|
print(f" want {want}")
|
|
|
|
check("parse: UNRESOLVED single", sorted(printed.get("unresolved", {})), [471])
|
|
check("parse: citations ambiguous", sorted(printed.get("citations", {})), [410, 1612])
|
|
check("parse: count-then-unit", sorted(printed.get("occurrences", {})), [6961])
|
|
check("parse: one word skipped", sorted(printed.get("files", {})), [297])
|
|
check("parse: clone-local marked", "blobs" in clone, True)
|
|
# the phrase rule: `ACS-only = 52` must NOT register as the unit `acs`
|
|
check("parse: inline phrase key", sorted(printed.get("acs_only", {})), [52])
|
|
|
|
lines = FAKE_DOC.split("\n")
|
|
got, first_block, revs_seen = [], [], set()
|
|
for first, blk in blocks(lines, 1, len(lines)):
|
|
revs = block_revs("\n".join(r for _, r in blk))
|
|
revs_seen |= set(revs)
|
|
for ln, val, unit, _tok, _pos in claims_in(blk):
|
|
got.append((unit, val))
|
|
if first == 1:
|
|
first_block.append((unit, val))
|
|
# the fixture that matters: one unit, one rev scope, two values -> a split
|
|
check("claims: both UNRESOLVED values in one block",
|
|
[uv for uv in first_block if uv[0] == "UNRESOLVED"],
|
|
[("UNRESOLVED", 471), ("UNRESOLVED", 441)])
|
|
check("claims: `471 cannot be resolved` is not a claim",
|
|
any(u in ("cannot", "resolved") for u, _ in got), False)
|
|
# the tightened separator, one cell per form it now refuses, plus the
|
|
# positive control -- a rule that refuses everything proves nothing
|
|
def claims_of(text):
|
|
return [(u, v) for _ln, v, u, _t, _p in claims_in([(1, text)])]
|
|
|
|
check("claims: a welded digit-and-letter run is not a claim",
|
|
claims_of("sha 83acffe0 / rev 9188dd3 at HEAD"), [])
|
|
check("claims: an ordinal is not a claim",
|
|
claims_of("1. INTERNAL SPLIT -- no oracle, no run"), [])
|
|
check("claims: a list marker is not a claim",
|
|
claims_of("0 ran; nothing to report"), [])
|
|
check("claims: a sentence end is not a claim",
|
|
claims_of("read as UTF-8. Never 1, never a traceback"), [])
|
|
check("claims: a compound adjective is not a claim",
|
|
claims_of("a 120-char window and a 1-based index"), [])
|
|
# the same refusal on the unit side, which is where the 2026-10-18 false
|
|
# split came from: `ACS-only` is a compound adjective, not a unit
|
|
check("claims: a hyphenated unit word is a compound adjective, not a claim",
|
|
claims_of("39 ACS-only lines carry no Q<digits> token"), [])
|
|
check("claims: the same unit unhyphenated is still a claim",
|
|
claims_of("39 ACS lines carry no Q<digits> token"), [("ACS", 39)])
|
|
check("claims: the tightening keeps a count and its unit",
|
|
claims_of("471 UNRESOLVED and 1,612 citations"),
|
|
[("UNRESOLVED", 471), ("citations", 1612)])
|
|
check("claims: the block's rev is read", "babdadc" in revs_seen, True)
|
|
# the digest rule, one signal per cell, and the negative control between
|
|
# them -- the text of each is the text of the defect, measured at HEAD
|
|
check("revs: `sha256 <hex>` is a digest, not the block's revision",
|
|
block_revs("queue 1,170,483 B / sha256 `c3b4a290dc64bd24`"), [])
|
|
check("revs: an ellipsis-truncated hex run is a digest, not a revision",
|
|
block_revs("at `28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`)"),
|
|
["28ca8e5"])
|
|
check("revs: an ordinarily cited revision is still the block's revision",
|
|
block_revs("Run 37 read (`29a4872`, queue 1,170,483 B) and `28ca8e5`"),
|
|
["28ca8e5", "29a4872"])
|
|
check("oracle: 471 matches", oracle(printed, clone, "UNRESOLVED", 471),
|
|
("MATCHES", ""))
|
|
check("oracle: 441 is stale", oracle(printed, clone, "UNRESOLVED", 441),
|
|
("STALE", "printed 471"))
|
|
check("oracle: ambiguous unit makes no claim",
|
|
oracle(printed, clone, "citations", 1612),
|
|
("AMBIGUOUS", "printed: 410, 1612"))
|
|
check("oracle: clone-local is not a defect",
|
|
oracle(printed, clone, "blobs", 1234), ("CLONE_LOCAL", "printed 871"))
|
|
check("oracle: unprinted unit", oracle(printed, clone, "wombats", 3),
|
|
("NOT_PRINTED", ""))
|
|
|
|
# ---- leg 2 (--selftest) and the leg rules. The first check is the defect
|
|
# the leg was added for: a docstring cell count the run leg cannot see.
|
|
s2, s2c = parse_printed(FAKE_SELFTEST)
|
|
check("selftest leg: the cell count is a printed unit", sorted(s2.get("cells", {})), [18])
|
|
legs = [(printed, clone, "")] + [(s2, s2c, "selftest")]
|
|
check("adjudicate: a cell count leg 1 cannot see is STALE, not NOT_PRINTED",
|
|
adjudicate(legs, "cells", 13, []), ("STALE", "printed 18 [selftest]"))
|
|
check("adjudicate: the corrected cell count matches",
|
|
adjudicate(legs, "cells", 18, []), ("MATCHES", ""))
|
|
check("adjudicate: leg 1 wins when both legs print the unit",
|
|
adjudicate([(s2, s2c, ""), ({**s2, "cells": {99: [1]}}, s2c, "selftest")],
|
|
"cells", 99, []), ("STALE", "printed 18"))
|
|
check("adjudicate: a no-rev leg does not verify a block that names a revision",
|
|
adjudicate([(s2, s2c, "no-rev")], "cells", 18, ["f5403b2"]),
|
|
("REV_UNCHECKED", "block names f5403b2; the tool takes no --rev, so this "
|
|
"leg measured the working tree"))
|
|
check("adjudicate: the selftest leg is not a verifier at a named revision",
|
|
adjudicate([(s2, s2c, "selftest")], "cells", 18, ["f5403b2"]),
|
|
("REV_UNCHECKED", "block names f5403b2; the selftest leg prints this "
|
|
"tool's own fixtures, not a census at that revision"))
|
|
check("adjudicate: a no-rev leg IS an oracle for a block naming no revision",
|
|
adjudicate([(s2, s2c, "no-rev")], "cells", 13, []),
|
|
("STALE", "printed 18 [no-rev]"))
|
|
check("adjudicate: no leg prints the unit -> NOT_PRINTED",
|
|
adjudicate([(printed, clone, "")], "wombats", 3, []), ("NOT_PRINTED", ""))
|
|
|
|
# ---- the recipe contract, exercised rather than asserted: run_tool takes a
|
|
# recipe's (argv, tag) and the tag must survive to the leg, because the
|
|
# tag is what decides whether a claim may be adjudicated at all. A stub
|
|
# stands in for a tool so this stays hermetic and instant; the real
|
|
# owners recipe costs a 107 s tree scan and is exercised by the run.
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
|
|
stub = os.path.join(td, "stub.py")
|
|
with open(stub, "w") as fh:
|
|
fh.write("print('cells: 7')\n")
|
|
for want_tag in ("no-rev", ""):
|
|
RECIPES["stub.py"] = (lambda t: lambda repo, rev, tmp: (["--probe"], t))(want_tag)
|
|
try:
|
|
out, rc, tag = run_tool(td, stub, "HEAD", td)
|
|
finally:
|
|
RECIPES.pop("stub.py", None)
|
|
check(f"recipe: run_tool carries a recipe's tag {want_tag!r} to the leg",
|
|
(rc, tag, "cells: 7" in out), (0, want_tag, True))
|
|
|
|
def _boom(repo_, rev_, tmp_):
|
|
raise RuntimeError("boom")
|
|
|
|
RECIPES["stub.py"] = _boom
|
|
raised = ""
|
|
try:
|
|
try:
|
|
run_tool(td, stub, "HEAD", td)
|
|
except RuntimeError as exc:
|
|
raised = str(exc)
|
|
finally:
|
|
RECIPES.pop("stub.py", None)
|
|
check("recipe: an unmaterialisable revision raises, never an empty census",
|
|
raised, "boom")
|
|
|
|
# A tool that rejects `--repo`/`--rev` but has a perfectly good default
|
|
# census. The retry must drop exactly the tokens argparse named: the
|
|
# first cut re-ran it as `<script> --repo R`, which was rejected again,
|
|
# and a working tool was reported RUN_FAILED. Both halves are pinned --
|
|
# the retry happens, and it happens only for an argparse rejection.
|
|
apstub = os.path.join(td, "stub_argparse.py")
|
|
with open(apstub, "w") as fh:
|
|
fh.write("import argparse\n"
|
|
"ap = argparse.ArgumentParser()\n"
|
|
'ap.add_argument("--selftest", action="store_true")\n'
|
|
"ap.parse_args()\n"
|
|
"print('cells: 7')\n")
|
|
out, rc, tag = run_tool(td, apstub, "HEAD", td)
|
|
check("run_tool: argparse rejection retried with the rejected flags dropped",
|
|
(rc, tag, "cells: 7" in out), (0, "no-rev", True))
|
|
|
|
rcstub = os.path.join(td, "stub_rc2.py")
|
|
with open(rcstub, "w") as fh:
|
|
fh.write("import sys\n"
|
|
"sys.stderr.write('boom\\n')\n"
|
|
"sys.exit(2)\n")
|
|
out, rc, tag = run_tool(td, rcstub, "HEAD", td)
|
|
check("run_tool: a non-argparse failure is not re-tagged no-rev",
|
|
(rc, tag, "rc=2" in out), (2, "", True))
|
|
# ---- a failed run leg does not silence its block. This path lives in
|
|
# check_tool and needs a subprocess, so it is exercised end to end with
|
|
# a stub tool rather than asserted: the stub rejects every census
|
|
# invocation with an argparse diagnosis and answers only `--selftest`.
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
|
|
stub = os.path.join(td, "stub_failing_run.py")
|
|
with open(stub, "w") as fh:
|
|
fh.write("'''stub: a run leg that fails, a selftest that answers.\n"
|
|
"\n7 cells\n'''\n"
|
|
"import sys\n"
|
|
'if "--selftest" in sys.argv:\n'
|
|
' print("selftest: 7 cells, 7 passed, 0 failed")\n'
|
|
"else:\n"
|
|
' sys.stderr.write("usage: stub_failing_run.py [-h] [--selftest]\\n")\n'
|
|
" sys.exit(2)\n")
|
|
rep = check_tool(td, stub, "stub_failing_run.py", {}, run=True, tmpdir=td)
|
|
check("check_tool: a failed run leg is reported, and leg 2 still adjudicates",
|
|
([f["verdict"] for f in rep["failures"]],
|
|
[(v["unit"], v["claim"], v["verdict"]) for v in rep["verdicts"]]),
|
|
(["RUN_FAILED"], [("cells", 7, "MATCHES")]))
|
|
|
|
# ---- the banner's arithmetic, which is what a reader actually reads: a
|
|
# RUN_FAILED leg and a STALE claim are counted in two columns, so four
|
|
# failed runs cannot read as four bad claims.
|
|
synthetic = {"tool": "t", "splits": [], "unchecked": {}, "selftest_leg": True,
|
|
"failures": [{"verdict": "RUN_FAILED"}],
|
|
"verdicts": [{"verdict": "MATCHES"}, {"verdict": "STALE"}]}
|
|
check("counts: a leg failure is not a claim defect", counts(synthetic)[:4],
|
|
(0, 1, 1, 2))
|
|
check("counts: an absent leg 2 is noted",
|
|
counts({**synthetic, "selftest_leg": False})[4],
|
|
["no --selftest: leg 2 absent"])
|
|
|
|
# --tools: a filter that matches nothing must not read as a clean scan.
|
|
check("tools: a bare basename is reported missing, not scanned as zero",
|
|
resolve_tools(["checkin_check.py"], {"tools/checkin_check.py"}),
|
|
(["checkin_check.py"], ["checkin_check.py"]))
|
|
check("tools: no filter means every tracked tool and nothing missing",
|
|
resolve_tools(None, {"tools/b.py", "tools/a.py", "README.md"}),
|
|
(["tools/a.py", "tools/b.py"], []))
|
|
|
|
# ---- the gate. Before `--fail` this tool exited 0 while printing STALE
|
|
# rows -- the two historical STALEs at 4ce1aed require the
|
|
# pre-tightening reader, `ac0e211^`, so they are cited with that build
|
|
# and not as current behaviour (see FAILING ON THE CLASS IT CATCHES) --
|
|
# so the one class it exists to catch could not fail anything. The
|
|
# first five cells pin the decision; the last four pin the PROCESS exit
|
|
# code, because a helper returning 1 proves nothing about a gate.
|
|
stale_rep = {"tool": "t", "splits": [], "unchecked": {}, "failures": [],
|
|
"verdicts": [{"verdict": "STALE"}], "selftest_leg": True}
|
|
split_rep = {**stale_rep, "splits": [{"unit": "UNRESOLVED", "rows": []}],
|
|
"verdicts": [{"verdict": "MATCHES"}]}
|
|
# a leg failure alone: reported, never spent as a defect
|
|
clean_rep = {**stale_rep, "failures": [{"verdict": "RUN_FAILED"}],
|
|
"verdicts": [{"verdict": "MATCHES"},
|
|
{"verdict": "NOT_PRINTED"},
|
|
{"verdict": "AMBIGUOUS"},
|
|
{"verdict": "REV_UNCHECKED"}]}
|
|
check("gate: off is rc 0 even with a STALE claim", gate_rc([stale_rep], False), 0)
|
|
check("gate: on and a STALE claim -> rc 1", gate_rc([stale_rep], True), 1)
|
|
check("gate: on and an INTERNAL SPLIT alone -> rc 1", gate_rc([split_rep], True), 1)
|
|
check("gate: on and a leg failure alone -> rc 0 (reported, not spent)",
|
|
gate_rc([clean_rep], True), 0)
|
|
check("gate: the two columns sum across tools", gate_faults([stale_rep, split_rep]),
|
|
(1, 1))
|
|
|
|
# ---- the mode symmetry (reply 22, 2026-10-18). A split is graded only when
|
|
# a leg verified its unit; without a run there is no oracle, so the
|
|
# same docstring that fails a full run must NOT fail `--no-run`. Both
|
|
# halves are pinned: the advisory side, and the teeth that survive.
|
|
check("mode: --no-run can only label a split, never verify it",
|
|
unit_scope("UNRESOLVED", [], True), "LABEL_ONLY")
|
|
check("mode: a unit no leg prints is not a census unit",
|
|
unit_scope("WOMBATS", [({"widgets": {7: [1]}}, {}, "")], False),
|
|
"NOT_A_CENSUS_UNIT")
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-mode-") as td:
|
|
split_stub = os.path.join(td, "stub_split.py")
|
|
with open(split_stub, "w") as fh:
|
|
fh.write('"""A stub whose docstring splits one unit.\n'
|
|
"\n471 UNRESOLVED at the head.\n" + "pad " * 80 + "\n"
|
|
'441 UNRESOLVED at the tail.\n"""\n'
|
|
"print('UNRESOLVED: 471')\n")
|
|
rep_norun = check_tool(td, split_stub, "stub_split.py", {}, run=False,
|
|
tmpdir=td)
|
|
rep_run = check_tool(td, split_stub, "stub_split.py", {}, run=True,
|
|
tmpdir=td)
|
|
check("mode: --no-run prints the split, graded as none",
|
|
([s["unit"] for s in rep_norun["splits"]],
|
|
[(s["unit"], s["how"]) for s in rep_norun["advisory"]]),
|
|
([], [("UNRESOLVED", "LABEL_ONLY")]))
|
|
check("mode: the same split is CHECKABLE once a leg prints the unit",
|
|
[(s["unit"], s["how"]) for s in rep_run["splits"]],
|
|
[("UNRESOLVED", "CHECKABLE")])
|
|
check("mode: --fail cannot fail on a split it could not verify",
|
|
gate_rc([rep_norun], True), 0)
|
|
check("mode: and still fails on the verified one", gate_rc([rep_run], True), 1)
|
|
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-gate-") as td:
|
|
os.makedirs(os.path.join(td, "tools"))
|
|
for name, claim in (("stub_stale.py", 99), ("stub_ok.py", 7)):
|
|
with open(os.path.join(td, "tools", name), "w") as fh:
|
|
fh.write(f'"""A stub whose docstring says {claim} widgets."""\n'
|
|
"print('widgets: 7')\n")
|
|
with open(os.path.join(td, "tools", "stub_split.py"), "w") as fh:
|
|
fh.write('"""A stub whose docstring splits one unit.\n'
|
|
"\n471 UNRESOLVED at the head.\n" + "pad " * 80 + "\n"
|
|
'441 UNRESOLVED at the tail.\n"""\n'
|
|
"print('UNRESOLVED: 471')\n")
|
|
genv = dict(os.environ, GIT_AUTHOR_NAME="t", GIT_AUTHOR_EMAIL="t@t",
|
|
GIT_COMMITTER_NAME="t", GIT_COMMITTER_EMAIL="t@t")
|
|
for sub in (["init", "-q"], ["add", "-A"], ["commit", "-qm", "fixture"]):
|
|
subprocess.run(["git", "-C", td] + sub, capture_output=True, env=genv)
|
|
|
|
def run_gate(*extra):
|
|
return subprocess.run([sys.executable, os.path.abspath(__file__),
|
|
"--repo", td, "--rev", "HEAD"] + list(extra),
|
|
capture_output=True, env=genv).returncode
|
|
|
|
check("gate e2e: a STALE claim fails the process (rc 1)", run_gate("--fail"), 1)
|
|
check("gate e2e: the same tree without --fail is rc 0 (the old silence)",
|
|
run_gate(), 0)
|
|
check("gate e2e: a matching claim passes the gate (rc 0)",
|
|
run_gate("--tools", "tools/stub_ok.py", "--fail"), 0)
|
|
check("gate e2e: a --tools filter matching nothing is rc 2, not 1",
|
|
run_gate("--tools", "tools/nope.py", "--fail"), 2)
|
|
# the mode pair, as PROCESS exit codes: same tree, same split, the two
|
|
# modes must not disagree about what they verified
|
|
check("gate e2e: --fail --no-run on a split is rc 0 (no oracle, advisory)",
|
|
run_gate("--tools", "tools/stub_split.py", "--no-run", "--fail"), 0)
|
|
check("gate e2e: --fail with the run grades that same split (rc 1)",
|
|
run_gate("--tools", "tools/stub_split.py", "--fail"), 1)
|
|
|
|
print(f"selftest: {cells[0]} cells, {cells[0] - bad[0]} passed, {bad[0]} failed")
|
|
return 0 if ok else 1
|
|
|
|
|
|
# ------------------------------------------------------------------- driver
|
|
def run_tool(repo, script, rev, tmpdir, timeout=600):
|
|
"""(stdout, rc, tag) for the tool's own run at `rev`.
|
|
|
|
`tag` is "" when the leg stood on the revision it was asked for and "no-rev"
|
|
when it did not: either the tool rejected `--rev` and was re-run without it,
|
|
or a recipe in RECIPES declared that it could not materialise the revision.
|
|
Such a leg measures the tree it is pointed at, so a claim in a block that
|
|
names a revision is reported REV_UNCHECKED rather than adjudicated (see
|
|
check_tool). stdout is returned; stderr is only folded in when there is no
|
|
stdout at all, so an argparse diagnosis is visible as a failure instead of
|
|
parsed as a census.
|
|
"""
|
|
recipe = RECIPES.get(os.path.basename(script), None)
|
|
if recipe is None:
|
|
argv, tag = [sys.executable, script, "--repo", repo, "--rev", rev], ""
|
|
else:
|
|
args, tag = recipe(repo, rev, tmpdir)
|
|
argv = [sys.executable, script] + args
|
|
p = subprocess.run(argv, capture_output=True, timeout=timeout)
|
|
out = p.stdout.decode("utf-8", "replace")
|
|
err = p.stderr.decode("utf-8", "replace")
|
|
# A tool that takes neither `--repo` nor `--rev` answers with an argparse
|
|
# diagnosis naming the flags it rejected, and NOTHING else: measured
|
|
# 2026-10-16, `check_lint.py --repo R --rev HEAD` exits 2 with stdout empty
|
|
# and `unrecognized arguments: --repo R --rev HEAD`, and the first cut of
|
|
# this rule re-ran it as `<script> --repo R` -- still rejected, still rc 2,
|
|
# so a tool with a perfectly good default census was reported RUN_FAILED.
|
|
# The retry therefore drops the tokens argparse itself named, and only
|
|
# those; a tool whose values collide with a dropped flag is a stated
|
|
# limitation, not a silent one.
|
|
m = re.search(r"unrecognized arguments:\s*(.+)$", err, re.M)
|
|
if tag != "no-rev" and p.returncode != 0 and m:
|
|
drop = set(m.group(1).split())
|
|
argv = [x for x in argv if x not in drop]
|
|
p = subprocess.run(argv, capture_output=True, timeout=timeout)
|
|
out = p.stdout.decode("utf-8", "replace")
|
|
err = p.stderr.decode("utf-8", "replace")
|
|
tag = "no-rev"
|
|
if not out.strip():
|
|
out = "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
|
|
return out, p.returncode, tag
|
|
|
|
|
|
def run_selftest(script, timeout=300):
|
|
"""Leg 2: the tool's own `--selftest`, or None if it takes no such flag.
|
|
|
|
A cell count is printed here and nowhere else, which is the whole reason
|
|
this leg exists. `None` means the tool has no selftest to offer -- reported
|
|
as one leg, not as a failure.
|
|
"""
|
|
try:
|
|
p = subprocess.run([sys.executable, script, "--selftest"],
|
|
capture_output=True, timeout=timeout)
|
|
except subprocess.TimeoutExpired:
|
|
return "<--selftest timed out>"
|
|
out = p.stdout.decode("utf-8", "replace")
|
|
err = p.stderr.decode("utf-8", "replace")
|
|
if "unrecognized arguments" in err and "--selftest" in err:
|
|
return None
|
|
if not out.strip():
|
|
return "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
|
|
return out
|
|
|
|
|
|
def unit_scope(unit, legs, no_run):
|
|
"""Is a split in this unit checkable, and why not if it is not?
|
|
|
|
With the oracle legs in hand a unit is checkable when some leg prints it
|
|
with exactly ONE value: the docstring and the output then talk about one
|
|
number. A unit printed with 2+ values is SECTION-SPLIT BY THE TOOL'S OWN
|
|
DESIGN (`citations` prints once under EXPLICIT and once under BARE), so a
|
|
docstring that mirrors both is correct and is reported as such, not flagged.
|
|
A unit no leg prints is not a census unit here. Without a run, only
|
|
uppercase census labels are checked.
|
|
"""
|
|
if no_run or not legs:
|
|
return ("LABEL_ONLY" if is_label(unit) else "UNCHECKED_NO_ORACLE")
|
|
saw_split = False
|
|
for printed, clone, _tag in legs:
|
|
vals = set()
|
|
for k in keys_for(unit, printed or {}):
|
|
vals |= set((printed or {})[k])
|
|
if not vals:
|
|
continue
|
|
if len(vals) > 1:
|
|
saw_split = True
|
|
continue
|
|
return "CHECKABLE"
|
|
if saw_split:
|
|
return "SECTION_SPLIT_BY_TOOL"
|
|
return "NOT_A_CENSUS_UNIT"
|
|
|
|
|
|
def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240):
|
|
text = open(script, encoding="utf-8", errors="replace").read()
|
|
span = docstring_span(script, text)
|
|
if span is None:
|
|
return None
|
|
lines = text.split("\n")
|
|
lo, hi = span
|
|
blk_claims = []
|
|
for first, blk in blocks(lines, lo, hi):
|
|
btext = "\n".join(raw for _, raw in blk)
|
|
revs = block_revs(btext)
|
|
cl = claims_in(blk)
|
|
if cl:
|
|
blk_claims.append((first, revs, cl, btext))
|
|
|
|
# ---- the legs first. Leg 1: the tool's own run, once per revision the
|
|
# docstring names. Leg 2: its --selftest, once. The tool under test is
|
|
# never this tool: running this scanner on itself spawns it, which
|
|
# spawns it again, without a floor.
|
|
is_self = os.path.basename(script) == os.path.basename(os.path.abspath(__file__))
|
|
printed_by_rev, verdicts, failures = {}, [], []
|
|
selftest_leg = None
|
|
if run and not is_self:
|
|
for first, revs, cl, _b in blk_claims:
|
|
for rev in (revs or ["HEAD"]):
|
|
key = (name, rev)
|
|
if key not in cache:
|
|
try:
|
|
out, rc, tag = run_tool(repo, script, rev, tmpdir)
|
|
cache[key] = (parse_printed(out), rc, out, tag)
|
|
except Exception as exc: # noqa: BLE001
|
|
cache[key] = (None, None, f"<run failed: {exc}>", "")
|
|
printed_by_rev.setdefault((first, rev), cache[key])
|
|
skey = (name, "<selftest>")
|
|
if skey not in cache:
|
|
try:
|
|
sout = run_selftest(script)
|
|
except Exception as exc: # noqa: BLE001
|
|
sout = f"<selftest failed: {exc}>"
|
|
cache[skey] = (parse_printed(sout) if sout else None,
|
|
0 if sout else None, sout or "", "selftest")
|
|
selftest_leg = cache[skey]
|
|
|
|
def legs_for(first, rev):
|
|
"""Priority-ordered [(printed, clone, tag)] for one block scope."""
|
|
out = []
|
|
entry = printed_by_rev.get((first, rev))
|
|
if entry and entry[0]:
|
|
out.append((entry[0][0], entry[0][1], entry[3]))
|
|
if selftest_leg and selftest_leg[0]:
|
|
out.append((selftest_leg[0][0], selftest_leg[0][1], "selftest"))
|
|
return out
|
|
|
|
# 1. INTERNAL SPLIT: one unit, one revision scope, two values, far enough
|
|
# apart to be a restatement rather than a neighbouring sub-count.
|
|
splits, advisory, unchecked = [], [], collections.Counter()
|
|
by_unit = collections.defaultdict(list)
|
|
for first, revs, cl, _b in blk_claims:
|
|
scope = tuple(revs)
|
|
for ln, val, unit, _t, pos in cl:
|
|
by_unit[(unit.upper() if unit.isupper() else unit.lower(), scope)].append(
|
|
(first, ln, val, pos))
|
|
for (unit, scope), items in sorted(by_unit.items()):
|
|
vals = {v for _, _, v, _ in items}
|
|
if len(vals) < 2:
|
|
continue
|
|
# a restatement is a pair far apart; an enumeration is neighbours
|
|
far = any(abs(b[3] - a[3]) >= isolation
|
|
for a, b in [(x, y) for x in items for y in items]
|
|
if a[1] != b[1] and a[2] != b[2])
|
|
if not far:
|
|
continue
|
|
rev_for_oracle = (items[0][0], scope[0] if len(scope) == 1 else "HEAD")
|
|
why = ("SELF_NOT_RUN" if is_self and run
|
|
else unit_scope(unit, legs_for(*rev_for_oracle), not run))
|
|
row = {"unit": unit, "revs": list(scope), "values": sorted(vals),
|
|
"how": why, "rows": [{"line": ln, "value": v, "block": b}
|
|
for b, ln, v, _ in items]}
|
|
# THE ONLY DOOR INTO `splits`. A split is graded only when a leg ran and
|
|
# printed its unit with exactly one value (CHECKABLE): then the docstring
|
|
# and the output are talking about one number and a disagreement is a
|
|
# defect the gate can verify. Every other reason -- no oracle at all
|
|
# (LABEL_ONLY, --no-run), a unit no leg prints (NOT_A_CENSUS_UNIT), a
|
|
# unit the tool itself prints per section (SECTION_SPLIT_BY_TOOL), this
|
|
# tool reading itself (SELF_NOT_RUN) -- is reported with its values and
|
|
# never reaches the gate. Decided 2026-10-18 (reply 22) after
|
|
# `--fail --no-run` returned 1 on the 471/441 split at fca3f2e while the
|
|
# full run returned 0 for the same units it could not see: rc 1 must
|
|
# never mean "I could not check this".
|
|
if why != "CHECKABLE":
|
|
unchecked[why] += 1
|
|
advisory.append(row)
|
|
continue
|
|
splits.append(row)
|
|
|
|
# 2. ORACLE: every claim compared against what the tool prints, leg 1 first.
|
|
if run:
|
|
for first, revs, cl, _b in blk_claims:
|
|
for rev in (revs or ["HEAD"]):
|
|
if is_self:
|
|
for ln, val, unit, tok, _pos in cl:
|
|
verdicts.append({"file": name, "block": first, "line": ln,
|
|
"rev": rev, "unit": unit, "claim": val,
|
|
"token": tok, "verdict": "SELF_NOT_RUN",
|
|
"detail": "the tool under test is this tool"})
|
|
continue
|
|
pr, rc, out, _tag = printed_by_rev[(first, rev)]
|
|
printed, clone = pr if pr else (None, None)
|
|
legs = legs_for(first, rev)
|
|
# A leg with no census -- raised, or exited non-zero having
|
|
# printed nothing -- adjudicates nothing either: drop it and let
|
|
# whichever leg survives answer instead. Only when none survives
|
|
# is the block left unadjudicated.
|
|
no_census = pr is None or (rc not in (0, None)
|
|
and not any(printed.values()))
|
|
if pr is None:
|
|
failures.append({"file": name, "block": first, "rev": rev,
|
|
"unit": "-", "claim": None,
|
|
"verdict": "RUN_FAILED", "detail": str(out)[:90]})
|
|
elif no_census:
|
|
failures.append({"file": name, "block": first, "rev": rev,
|
|
"unit": "-", "claim": None,
|
|
"verdict": "RUN_FAILED",
|
|
"detail": "rc=%s, nothing printed: %s"
|
|
% (rc, str(out)[:80])})
|
|
if no_census:
|
|
legs = [l for l in legs if l[2] == "selftest"]
|
|
if not legs:
|
|
continue
|
|
for ln, val, unit, tok, _pos in cl:
|
|
v, detail = adjudicate(legs, unit, val, revs)
|
|
verdicts.append({"file": name, "block": first, "line": ln,
|
|
"rev": rev, "unit": unit, "claim": val,
|
|
"token": tok, "verdict": v, "detail": detail})
|
|
return {"tool": name, "splits": splits, "advisory": advisory,
|
|
"unchecked": dict(unchecked),
|
|
"failures": failures, "verdicts": verdicts,
|
|
"self": is_self and run, "selftest_leg": (not run) or bool(selftest_leg and selftest_leg[0]),
|
|
"blocks_with_claims": len(blk_claims)}
|
|
|
|
|
|
def counts(rep):
|
|
"""(splits, claim_defects, leg_failures, claims, notes) -- the banner's rows.
|
|
|
|
A LEG FAILURE IS NOT A CLAIM DEFECT. RUN_FAILED says no census came back and
|
|
such a leg adjudicates nothing (see check_tool); STALE says a claim was read
|
|
against a census and disagreed. The first live run of the two-leg tool mixed
|
|
them in one `N STALE/failed` column, so four failed runs read as four bad
|
|
claims -- measured 2026-10-16 at HEAD, check_lint.py 2 and
|
|
queue_claim_check.py 2, none of them a claim disagreement. Two columns, so
|
|
the distinction survives the summary a reader actually reads.
|
|
"""
|
|
stale = [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
|
|
notes = []
|
|
if rep.get("self"):
|
|
notes.append("self: no leg run (SELF_NOT_RUN)")
|
|
elif not rep.get("selftest_leg"):
|
|
notes.append("no --selftest: leg 2 absent")
|
|
return (len(rep["splits"]), len(stale), len(rep["failures"]),
|
|
len(rep["verdicts"]), notes)
|
|
|
|
|
|
def gate_faults(reports):
|
|
"""(splits, stale) -- the two things `--fail` refuses to let pass.
|
|
|
|
The class this tool exists to catch is a docstring claim that disagrees
|
|
with the census printed beside it (STALE), and two claims of one unit that
|
|
disagree with each other (INTERNAL SPLIT). A LEG FAILURE IS NOT IN THIS
|
|
COUNT, for the reason counts() gives: a leg that produced no census
|
|
adjudicated no claim, so counting it would fail a run on a claim never
|
|
checked. rc 2, decided in main(), covers "could not grade" separately.
|
|
"""
|
|
splits = stale = 0
|
|
for rep in reports:
|
|
splits += len(rep.get("splits") or [])
|
|
stale += sum(1 for v in (rep.get("verdicts") or [])
|
|
if v.get("verdict") == "STALE")
|
|
return splits, stale
|
|
|
|
|
|
def gate_rc(reports, fail):
|
|
"""0 when the gate is off or nothing was graded faulty; 1 when it is on and
|
|
a defect claim was graded. Never 2: "could not grade" is main()'s call,
|
|
made before any report exists, and this function cannot see it."""
|
|
if not fail:
|
|
return 0
|
|
splits, stale = gate_faults(reports)
|
|
return 1 if (splits or stale) else 0
|
|
|
|
|
|
def resolve_tools(requested, tree):
|
|
"""(tools, missing). A requested name that is not tracked at the rev is
|
|
REPORTED, never silently dropped: `--tools checkin_check.py` (a bare
|
|
basename instead of a repo-relative path) scanned zero tools and exited 0,
|
|
which reads exactly like a clean pass. Measured 2026-10-18 at 8932d06."""
|
|
if not requested:
|
|
return sorted(p for p in tree
|
|
if p.startswith("tools/") and p.endswith(".py")), []
|
|
return list(requested), [t for t in requested if t not in tree]
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser(description=__doc__,
|
|
formatter_class=argparse.RawDescriptionHelpFormatter)
|
|
ap.add_argument("--repo", default=DEFAULT_REPO)
|
|
ap.add_argument("--rev", default="HEAD")
|
|
ap.add_argument("--tools", nargs="*")
|
|
ap.add_argument("--json")
|
|
ap.add_argument("--no-run", action="store_true",
|
|
help="label-only: no subprocess, no oracle, so every split "
|
|
"it prints is ADVISORY and --fail cannot fail on one -- "
|
|
"rc 1 would mean 'could not check this'")
|
|
ap.add_argument("--fail", action="store_true",
|
|
help="rc 1 if any tool reports an INTERNAL SPLIT a leg "
|
|
"VERIFIED (CHECKABLE) or a STALE claim; unverified "
|
|
"splits and leg failures are reported, never spent as "
|
|
"defects (rc 2 stays 'could not grade')")
|
|
ap.add_argument("--selftest", action="store_true")
|
|
a = ap.parse_args()
|
|
|
|
if a.selftest:
|
|
return _fixture_checks()
|
|
|
|
store = git(a.repo, "ls-tree", "-r", "--name-only", a.rev)
|
|
tree = {p for p in store.decode("utf-8", "replace").split("\n") if p}
|
|
tools, missing = resolve_tools(a.tools, tree)
|
|
if not tools:
|
|
print(f"no tools found at {a.rev}", file=sys.stderr)
|
|
return 2
|
|
if missing:
|
|
print(f"cannot grade: --tools names nothing tracked at {a.rev}: "
|
|
f"{', '.join(missing)}", file=sys.stderr)
|
|
print(" names are repo-relative, e.g. --tools tools/checkin_check.py; "
|
|
"a filter that matches nothing is not a clean scan", file=sys.stderr)
|
|
return 2
|
|
|
|
cache, reports = {}, []
|
|
with tempfile.TemporaryDirectory(prefix="doccensus-") as tmpdir:
|
|
for t in tools:
|
|
src = subprocess.run(["git", "-C", a.repo, "show", f"{a.rev}:{t}"],
|
|
capture_output=True).stdout
|
|
local = os.path.join(tmpdir, os.path.basename(t))
|
|
with open(local, "wb") as fh:
|
|
fh.write(src)
|
|
try:
|
|
rep = check_tool(a.repo, local, t, cache, run=not a.no_run,
|
|
tmpdir=tmpdir)
|
|
except subprocess.TimeoutExpired:
|
|
rep = {"tool": t, "splits": [], "blocks_with_claims": 0,
|
|
"verdicts": [{"file": t, "verdict": "TIMEOUT", "rev": "-",
|
|
"unit": "-", "claim": None, "block": 0}]}
|
|
if rep:
|
|
reports.append(rep)
|
|
|
|
print(f"tools scanned at {a.rev}: {len(reports)} "
|
|
f"({'internal-split only, label vocabulary' if a.no_run else 'docstring vs printed census'})")
|
|
tot = collections.Counter()
|
|
for rep in reports:
|
|
bad = rep["failures"] + [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
|
|
nsplits, nstale, nfail, nclaims, notes = counts(rep)
|
|
print(f"\n{rep['tool']}: {nsplits} INTERNAL SPLIT, "
|
|
f"{nstale} STALE claim(s), {nfail} leg failure(s), "
|
|
f"{nclaims} claims checked"
|
|
+ (" [" + "; ".join(notes) + "]" if notes else ""))
|
|
for s in rep["splits"]:
|
|
vals = ", ".join(f"{r['value']}@{r['line']}" for r in s["rows"])
|
|
scope = ",".join(s["revs"]) or "no rev named"
|
|
print(f" SPLIT {s['unit']} [{scope}] ({s['how']}): {vals}")
|
|
for k, c in sorted(rep["unchecked"].items()):
|
|
print(f" (not flagged: {c} split(s) in units {k})")
|
|
# A --no-run split is the one advisory class a reader of the fast mode
|
|
# actually needs to SEE: it is what the old build failed the gate on.
|
|
# Printed with its values, labelled advisory, never counted above.
|
|
for s in rep.get("advisory") or []:
|
|
if s["how"] != "LABEL_ONLY":
|
|
continue
|
|
vals = ", ".join(f"{r['value']}@{r['line']}" for r in s["rows"])
|
|
print(f" ADVISORY {s['unit']} (LABEL_ONLY, no oracle): {vals}"
|
|
f" -- reported, not graded")
|
|
for v in bad:
|
|
print(f" {v['verdict']:10s} {v['file']}:{v.get('line', '-')} "
|
|
f"{v['unit']} claimed {v['claim']} at {v['rev']} -- {v['detail']}")
|
|
for k in sorted({v["verdict"] for v in rep["verdicts"]}):
|
|
if k in ("REV_UNCHECKED", "SELF_NOT_RUN", "CLONE_LOCAL"):
|
|
n = sum(1 for v in rep["verdicts"] if v["verdict"] == k)
|
|
print(f" ({n} {k}: reported, never counted as a defect)")
|
|
for k, c in collections.Counter(v["verdict"] for v in rep["verdicts"]).items():
|
|
tot[k] += c
|
|
print("\n--- verdict totals (unit: claims) ---")
|
|
for k in sorted(tot):
|
|
print(f" {k:12s} {tot[k]}")
|
|
if a.json:
|
|
with open(a.json, "w") as fh:
|
|
json.dump({"rev": a.rev, "repo": a.repo, "reports": reports}, fh, indent=1)
|
|
print(f"wrote {a.json}")
|
|
if a.fail:
|
|
nsplit, nstale = gate_faults(reports)
|
|
if nsplit or nstale:
|
|
print(f"\nFAIL: {nsplit} INTERNAL SPLIT + {nstale} STALE claim(s) "
|
|
f"at {a.rev}"
|
|
+ (" (internal-split only: --no-run)" if a.no_run else ""))
|
|
return 1
|
|
print(f"\nOK: 0 INTERNAL SPLIT, 0 STALE claim(s) at {a.rev}"
|
|
+ (" (no oracle in this mode: nothing was verified, so nothing is "
|
|
"graded -- ADVISORY rows above are labels, not faults)"
|
|
if a.no_run else ""))
|
|
return gate_rc(reports, a.fail)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|