stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tools (volt): the two-leg doc_census_check -- leg 2 is the tool's own --selftest, and the seed defect was invisible without it

Browse Source 
The scanner reads a tool's module docstring, compares every `<value> <unit>`
claim against the census that tool PRINTS, and until this commit it had one
oracle: the tool's own run. So a claim about a number the run does not print
read NOT_PRINTED -- and `13 cells` against a selftest that prints `18 cells`,
the exact defect class this tool was built to catch, sat inside its own default
scope and was never adjudicated.

WHAT IS LANDED (907 -> 908 lines; the only edit in this commit is the splice
below, byte-measured in the check-in):

1. Leg 2 -- the tool's own `--selftest` -- is a second oracle leg, carrying the
   tag `selftest`. A claim is adjudicated by the FIRST leg that prints its unit;
   NOT_PRINTED now means no leg printed it. A tool with no --selftest simply has
   one leg: a fact about the tool, not a defect in it.
2. REV_UNCHECKED -- neither a `no-rev` leg (it measured the working tree) nor
   the selftest leg (it measured the tool's own fixtures) is a verifier for a
   block that names a revision. Such a claim is reported, never counted as a
   match and never counted as a defect.
3. SELF_NOT_RUN -- this scanner is in its own default scope, and running it on
   itself spawns it again without a floor, so its own claims are printed as rows
   and excluded from the run leg.
4. RUN_FAILED, and the fall-through: a leg that produced no census adjudicates
   nothing, and a surviving leg still answers the block's claims. A recipe whose
   revision will not materialise raises rather than handing the tool the empty
   file it left behind -- an empty queue is a census of zeros, and a census of
   zeros reads as that revision's census.
5. A recipe for append_only_owners.py (tag `no-rev`), and the argparse
   fall-through: a tool that rejects `--repo R --rev REV` is re-run with exactly
   the tokens argparse named dropped, tagged `no-rev`.

MEASURED DELTA, same 6 tools at HEAD, HEAD tool vs this one (compare.py on
head6.json / cand6.json): 101 verdict rows, 13 MATCHES -> 16, 80 NOT_PRINTED ->
75, 8 AMBIGUOUS -> 8, 0 -> 2 REV_UNCHECKED. Five rows change and three of them
are substantive: append_only_audit.py `18 cells` NOT_PRINTED -> MATCHES (leg 2);
append_only_owners.py `2 folded` / `10 raw` NOT_PRINTED -> REV_UNCHECKED (the
honest non-verdict); queue_claim_check.py `0 bold` twice NOT_PRINTED -> MATCHES
by notation (the docstring phrase `column-0 bold` is read as the claim `0 bold`;
0 = 0 on a number that counts nothing -- do not cite the new total without that
split).

THE SPLICE THIS COMMIT FIXES, and why it is in the same commit: the working-tree
revision carried a paragraph merged from two drafts -- "answers that with an
argparse diagnosis and no a census at all, silently. It is now re-run ..." --
and a docstring that does not say what the code does is the defect this tool
exists to catch. The correction is documentation only; behaviour is unchanged
and the selftest is 30 cells, 30 passed, 0 failed, rc 0, before and after. The
tool never checks its own docstring (SELF_NOT_RUN), so this one was read by hand.
This commit is contained in:
volt authored and Timmy committed 2026-10-16 22:38:20 +00:00
1 parent 74f1e066e6
commit 0031a7831f
1 file changed
+413 -53
Download Patch File Download Diff File Expand all files Collapse all files
tools/doc_census_check.py
+413 -53
View File
@@ -42,14 +42,56 @@ WHAT IT MEASURES (read-only: this tool only writes --json)
A claim is `<digits> <unitword>` with the unit word ADJACENT, which is the
archive's own notation. A bare number is not a claim and is not checked.
TWO ORACLE LEGS, AND WHY THE SECOND ONE EXISTS
Leg 1 is the tool's own run at the block's revision, exactly as above. Leg 2
is the SAME TOOL'S `--selftest`, added 2026-10-16 after the control plane
returned a docstring claim of `13 cells` against a selftest that prints `18
cells` -- the defect class this tool exists to catch, sitting inside its own
default scope and still read as NOT_PRINTED, because no leg ever passed
`--selftest` and a cell count is printed by nothing else. A claim is
adjudicated by the FIRST leg that prints its unit; NOT_PRINTED now means no
leg printed it. A tool that takes no `--selftest` simply has one leg -- a fact
about the tool, not a defect in it.
A leg that exits non-zero having printed nothing is reported RUN_FAILED, never
read as an empty census, and it adjudicates nothing either: the block's claims
are still read off any surviving leg -- leg 2 -- and are left unadjudicated
only when no leg survives, because a failed run is not a reason to stop
looking. A revision that will not materialise fails the same way rather than
producing a census at all: an empty queue is a census of zeros, and reading it
as that revision's census is the phantom this rule refuses (measured
2026-10-16 against a sha256 prefix read as a revision).
The run leg is `<script> --repo R --rev REV`. A tool that takes no `--rev`
answers it with an argparse diagnosis and prints no census at all, so the leg
is re-run with the rejected flags DROPPED -- the exact tokens argparse named,
and only those -- and is tagged `no-rev`; a recipe's leg may carry the same tag
for the same reason. Such a tool measures the tree it is POINTED AT, so a claim
in a block that names a revision is reported REV_UNCHECKED -- never counted as
verified against that revision, and never counted as a defect -- while a block
that names no revision is checked normally, since the default is what the flag
means there.
THIS TOOL DOES NOT RUN ITSELF. The default scope is every `tools/*.py` in the
tree, and this tool is one of them. Running it on itself spawns it, which
spawns it again, without a floor -- measured, not reasoned, in the check-in of
2026-10-16. The self target is therefore excluded from the run leg and every
one of its claims is reported SELF_NOT_RUN. It is deliberately still listed in
the scanned set, so the exclusion is a printed row and not a silence.
SCOPE, STATED RATHER THAN IMPLIED
Only module docstrings of .py tools are scanned: a docstring is where a census
table lives. Only revisions NAMED IN THE BLOCK are used as oracles -- a block
naming no revision is checked against the tool's default (HEAD) and tagged, so
a stale rev is never silently attributed. A tool that cannot be run at a
revision (queue_claim_check.py has no --rev; the queue is materialised from
git and handed to it by --queue) is driven through a per-tool recipe declared
below, with a temp file that is removed.
a stale rev is never silently attributed. A tool whose subject is not `--repo`
+ `--rev` is driven through a per-tool recipe declared below, with a temp file
that is removed: `queue_claim_check.py` takes a materialised QUEUE file and so
its leg IS the revision, while `append_only_owners.py` takes a tree scan that
no flag can pin to a revision, so its leg is tagged `no-rev`. A tool no recipe
covers, and that answers `--repo ... --rev ...` with an argparse diagnosis, is
re-run with the rejected flags dropped and tagged the same way -- measured at
HEAD on 2026-10-16 against `check_lint.py`, which takes neither flag and whose
default census is a real one.
WHAT THIS DOES NOT CHECK, and the measured size of the gap. A claim is only
adjudicated when the unit it names is a unit the tool PRINTS. Over the three
@@ -113,19 +155,47 @@ STOPWORDS = {
"can", "same", "each", "both", "per", "via", "b", "kb", "mb", "gb",
}
# How to run each tool at a revision. `queue_claim_check.py` takes no --rev: its
# subject is a FILE, so the file is materialised at the revision and handed to
# it. Any tool not listed here is run as `<script> --repo R --rev REV`.
# How to run each tool at a revision. Any tool not listed here is run as
# `<script> --repo R --rev REV`. A recipe returns `(argv, tag)` and the tag says
# whether the recipe could materialise the revision the block names.
#
# queue_claim_check.py takes no --rev: its subject is a FILE, and the file is
# materialised at the revision and handed to it -- so the leg IS that rev.
# A revision that will NOT materialise is a raised failure, never an empty
# file: an empty queue is a census of zeros, and a census of zeros reads as
# that revision's census. Measured 2026-10-16 -- `git show
# c3b4a290dc64bd24:WORK-QUEUE.md` (a sha256 prefix REV_LONG read as a rev)
# exits 128, and the leg it fed printed `0 opening records` at rc 0.
# append_only_owners.py takes a positional AUDIT.json and nothing else: its
# input is a tree scan, which its sibling produces over the WORKING TREE
# (`append_only_audit.py` has no --rev either). This scanner is read-only, so
# it will not add a git worktree to the repository it is only allowed to
# read: the leg is tagged no-rev and every claim in a block that names a
# revision is reported REV_UNCHECKED, never adjudicated against a tree it
# did not stand on.
def _recipe_queue(repo, rev, tmpdir):
dst = os.path.join(tmpdir, f"WORK-QUEUE.{rev}.md")
blob = subprocess.run(["git", "-C", repo, "show", f"{rev}:WORK-QUEUE.md"],
capture_output=True).stdout
p = subprocess.run(["git", "-C", repo, "show", f"{rev}:WORK-QUEUE.md"],
capture_output=True)
if p.returncode != 0:
raise RuntimeError("git show %s:WORK-QUEUE.md -> rc=%d: %s"
% (rev, p.returncode,
p.stderr.decode("utf-8", "replace").strip()[:120]))
with open(dst, "wb") as fh:
fh.write(blob)
return ["--queue", dst, "--census"]
fh.write(p.stdout)
return ["--queue", dst, "--census"], ""
RECIPES = {"queue_claim_check.py": _recipe_queue}
def _recipe_owners(repo, rev, tmpdir):
dst = os.path.join(tmpdir, f"append-only-audit.{rev}.json")
subprocess.run([sys.executable,
os.path.join(repo, "tools", "append_only_audit.py"),
"--repo", repo, "--json", dst], capture_output=True)
return [dst], "no-rev"
RECIPES = {"queue_claim_check.py": _recipe_queue,
"append_only_owners.py": _recipe_owners}
def git(repo, *a):
@@ -265,8 +335,12 @@ def keys_for(unit, printed):
return out
def oracle(printed, clone, unit, value):
"""(verdict, detail) for one claim; `printed` is one revision's map."""
def oracle(printed, clone, unit, value, tag=""):
"""(verdict, detail) for one claim; `printed` is one revision's map.
`tag` names the leg a verdict came from when it is not leg 1 ("no-rev",
"selftest"): a reader can then tell which census the number was read off.
"""
keys = keys_for(unit, printed)
vals = set()
for k in keys:
@@ -279,8 +353,32 @@ def oracle(printed, clone, unit, value):
if printed_val == value:
return "MATCHES", ""
if any(k in clone for k in keys):
return "CLONE_LOCAL", f"printed {printed_val}"
return "STALE", f"printed {printed_val}"
return "CLONE_LOCAL", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
return "STALE", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
def adjudicate(legs, unit, value, block_revs):
"""The first leg that prints the unit adjudicates the claim.
`legs` is [(printed, clone, tag)] in priority order: leg 1 (the run) first,
then the selftest leg. NEITHER a `no-rev` leg NOR the selftest leg is a
verifier for a block that names a revision: one measured a different tree,
the other measured the tool's own fixtures. Such a claim is reported
REV_UNCHECKED -- visible, counted as neither a match nor a defect.
"""
for printed, clone, tag in legs:
if not printed or not keys_for(unit, printed):
continue
if block_revs and tag == "no-rev":
return "REV_UNCHECKED", ("block names %s; the tool takes no --rev, so this "
"leg measured the working tree"
% ",".join(block_revs))
if block_revs and tag == "selftest":
return "REV_UNCHECKED", ("block names %s; the selftest leg prints this "
"tool's own fixtures, not a census at that "
"revision" % ",".join(block_revs))
return oracle(printed, clone, unit, value, tag)
return "NOT_PRINTED", ""
# --------------------------------------------------------------- self-test
@@ -307,14 +405,24 @@ THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
A SECOND BLOCK, no revision named, quoting 471 UNRESOLVED and nothing else.
"""
# Leg 2's shape: a selftest summary line, and nothing else. The cell count is
# printed here and nowhere else, which is why a claim about it needs this leg.
FAKE_SELFTEST = """\
c1 a NEW byte PASS
selftest: 18 cells, 18 passed, 0 failed
"""
def _fixture_checks():
printed, clone = parse_printed(FAKE_STDOUT)
ok = True
cells, bad = [0], [0]
def check(name, got, want):
nonlocal ok
good = got == want
cells[0] += 1
bad[0] += 0 if good else 1
ok = ok and good
print(f"{'PASS' if good else 'FAIL'} {name}")
if not good:
@@ -356,42 +464,219 @@ def _fixture_checks():
oracle(printed, clone, "blobs", 1234), ("CLONE_LOCAL", "printed 871"))
check("oracle: unprinted unit", oracle(printed, clone, "wombats", 3),
("NOT_PRINTED", ""))
# ---- leg 2 (--selftest) and the leg rules. The first check is the defect
# the leg was added for: a docstring cell count the run leg cannot see.
s2, s2c = parse_printed(FAKE_SELFTEST)
check("selftest leg: the cell count is a printed unit", sorted(s2.get("cells", {})), [18])
legs = [(printed, clone, "")] + [(s2, s2c, "selftest")]
check("adjudicate: a cell count leg 1 cannot see is STALE, not NOT_PRINTED",
adjudicate(legs, "cells", 13, []), ("STALE", "printed 18 [selftest]"))
check("adjudicate: the corrected cell count matches",
adjudicate(legs, "cells", 18, []), ("MATCHES", ""))
check("adjudicate: leg 1 wins when both legs print the unit",
adjudicate([(s2, s2c, ""), ({**s2, "cells": {99: [1]}}, s2c, "selftest")],
"cells", 99, []), ("STALE", "printed 18"))
check("adjudicate: a no-rev leg does not verify a block that names a revision",
adjudicate([(s2, s2c, "no-rev")], "cells", 18, ["f5403b2"]),
("REV_UNCHECKED", "block names f5403b2; the tool takes no --rev, so this "
"leg measured the working tree"))
check("adjudicate: the selftest leg is not a verifier at a named revision",
adjudicate([(s2, s2c, "selftest")], "cells", 18, ["f5403b2"]),
("REV_UNCHECKED", "block names f5403b2; the selftest leg prints this "
"tool's own fixtures, not a census at that revision"))
check("adjudicate: a no-rev leg IS an oracle for a block naming no revision",
adjudicate([(s2, s2c, "no-rev")], "cells", 13, []),
("STALE", "printed 18 [no-rev]"))
check("adjudicate: no leg prints the unit -> NOT_PRINTED",
adjudicate([(printed, clone, "")], "wombats", 3, []), ("NOT_PRINTED", ""))
# ---- the recipe contract, exercised rather than asserted: run_tool takes a
# recipe's (argv, tag) and the tag must survive to the leg, because the
# tag is what decides whether a claim may be adjudicated at all. A stub
# stands in for a tool so this stays hermetic and instant; the real
# owners recipe costs a 107 s tree scan and is exercised by the run.
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
stub = os.path.join(td, "stub.py")
with open(stub, "w") as fh:
fh.write("print('cells: 7')\n")
for want_tag in ("no-rev", ""):
RECIPES["stub.py"] = (lambda t: lambda repo, rev, tmp: (["--probe"], t))(want_tag)
try:
out, rc, tag = run_tool(td, stub, "HEAD", td)
finally:
RECIPES.pop("stub.py", None)
check(f"recipe: run_tool carries a recipe's tag {want_tag!r} to the leg",
(rc, tag, "cells: 7" in out), (0, want_tag, True))
def _boom(repo_, rev_, tmp_):
raise RuntimeError("boom")
RECIPES["stub.py"] = _boom
raised = ""
try:
try:
run_tool(td, stub, "HEAD", td)
except RuntimeError as exc:
raised = str(exc)
finally:
RECIPES.pop("stub.py", None)
check("recipe: an unmaterialisable revision raises, never an empty census",
raised, "boom")
# A tool that rejects `--repo`/`--rev` but has a perfectly good default
# census. The retry must drop exactly the tokens argparse named: the
# first cut re-ran it as `<script> --repo R`, which was rejected again,
# and a working tool was reported RUN_FAILED. Both halves are pinned --
# the retry happens, and it happens only for an argparse rejection.
apstub = os.path.join(td, "stub_argparse.py")
with open(apstub, "w") as fh:
fh.write("import argparse\n"
"ap = argparse.ArgumentParser()\n"
'ap.add_argument("--selftest", action="store_true")\n'
"ap.parse_args()\n"
"print('cells: 7')\n")
out, rc, tag = run_tool(td, apstub, "HEAD", td)
check("run_tool: argparse rejection retried with the rejected flags dropped",
(rc, tag, "cells: 7" in out), (0, "no-rev", True))
rcstub = os.path.join(td, "stub_rc2.py")
with open(rcstub, "w") as fh:
fh.write("import sys\n"
"sys.stderr.write('boom\\n')\n"
"sys.exit(2)\n")
out, rc, tag = run_tool(td, rcstub, "HEAD", td)
check("run_tool: a non-argparse failure is not re-tagged no-rev",
(rc, tag, "rc=2" in out), (2, "", True))
# ---- a failed run leg does not silence its block. This path lives in
# check_tool and needs a subprocess, so it is exercised end to end with
# a stub tool rather than asserted: the stub rejects every census
# invocation with an argparse diagnosis and answers only `--selftest`.
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
stub = os.path.join(td, "stub_failing_run.py")
with open(stub, "w") as fh:
fh.write("'''stub: a run leg that fails, a selftest that answers.\n"
"\n7 cells\n'''\n"
"import sys\n"
'if "--selftest" in sys.argv:\n'
' print("selftest: 7 cells, 7 passed, 0 failed")\n'
"else:\n"
' sys.stderr.write("usage: stub_failing_run.py [-h] [--selftest]\\n")\n'
" sys.exit(2)\n")
rep = check_tool(td, stub, "stub_failing_run.py", {}, run=True, tmpdir=td)
check("check_tool: a failed run leg is reported, and leg 2 still adjudicates",
([f["verdict"] for f in rep["failures"]],
[(v["unit"], v["claim"], v["verdict"]) for v in rep["verdicts"]]),
(["RUN_FAILED"], [("cells", 7, "MATCHES")]))
# ---- the banner's arithmetic, which is what a reader actually reads: a
# RUN_FAILED leg and a STALE claim are counted in two columns, so four
# failed runs cannot read as four bad claims.
synthetic = {"tool": "t", "splits": [], "unchecked": {}, "selftest_leg": True,
"failures": [{"verdict": "RUN_FAILED"}],
"verdicts": [{"verdict": "MATCHES"}, {"verdict": "STALE"}]}
check("counts: a leg failure is not a claim defect", counts(synthetic)[:4],
(0, 1, 1, 2))
check("counts: an absent leg 2 is noted",
counts({**synthetic, "selftest_leg": False})[4],
["no --selftest: leg 2 absent"])
print(f"selftest: {cells[0]} cells, {cells[0] - bad[0]} passed, {bad[0]} failed")
return 0 if ok else 1
# ------------------------------------------------------------------- driver
def run_tool(repo, script, rev, tmpdir, timeout=600):
args = RECIPES.get(os.path.basename(script), None)
if args is None:
argv = [sys.executable, script, "--repo", repo, "--rev", rev]
"""(stdout, rc, tag) for the tool's own run at `rev`.
`tag` is "" when the leg stood on the revision it was asked for and "no-rev"
when it did not: either the tool rejected `--rev` and was re-run without it,
or a recipe in RECIPES declared that it could not materialise the revision.
Such a leg measures the tree it is pointed at, so a claim in a block that
names a revision is reported REV_UNCHECKED rather than adjudicated (see
check_tool). stdout is returned; stderr is only folded in when there is no
stdout at all, so an argparse diagnosis is visible as a failure instead of
parsed as a census.
"""
recipe = RECIPES.get(os.path.basename(script), None)
if recipe is None:
argv, tag = [sys.executable, script, "--repo", repo, "--rev", rev], ""
else:
argv = [sys.executable, script] + args(repo, rev, tmpdir)
args, tag = recipe(repo, rev, tmpdir)
argv = [sys.executable, script] + args
p = subprocess.run(argv, capture_output=True, timeout=timeout)
return p.stdout.decode("utf-8", "replace"), p.returncode
out = p.stdout.decode("utf-8", "replace")
err = p.stderr.decode("utf-8", "replace")
# A tool that takes neither `--repo` nor `--rev` answers with an argparse
# diagnosis naming the flags it rejected, and NOTHING else: measured
# 2026-10-16, `check_lint.py --repo R --rev HEAD` exits 2 with stdout empty
# and `unrecognized arguments: --repo R --rev HEAD`, and the first cut of
# this rule re-ran it as `<script> --repo R` -- still rejected, still rc 2,
# so a tool with a perfectly good default census was reported RUN_FAILED.
# The retry therefore drops the tokens argparse itself named, and only
# those; a tool whose values collide with a dropped flag is a stated
# limitation, not a silent one.
m = re.search(r"unrecognized arguments:\s*(.+)$", err, re.M)
if tag != "no-rev" and p.returncode != 0 and m:
drop = set(m.group(1).split())
argv = [x for x in argv if x not in drop]
p = subprocess.run(argv, capture_output=True, timeout=timeout)
out = p.stdout.decode("utf-8", "replace")
err = p.stderr.decode("utf-8", "replace")
tag = "no-rev"
if not out.strip():
out = "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
return out, p.returncode, tag
def unit_scope(unit, printed, clone, no_run):
def run_selftest(script, timeout=300):
"""Leg 2: the tool's own `--selftest`, or None if it takes no such flag.
A cell count is printed here and nowhere else, which is the whole reason
this leg exists. `None` means the tool has no selftest to offer -- reported
as one leg, not as a failure.
"""
try:
p = subprocess.run([sys.executable, script, "--selftest"],
capture_output=True, timeout=timeout)
except subprocess.TimeoutExpired:
return "<--selftest timed out>"
out = p.stdout.decode("utf-8", "replace")
err = p.stderr.decode("utf-8", "replace")
if "unrecognized arguments" in err and "--selftest" in err:
return None
if not out.strip():
return "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
return out
def unit_scope(unit, legs, no_run):
"""Is a split in this unit checkable, and why not if it is not?
With the oracle (printed map) in hand a unit is checkable when the tool
prints it with exactly ONE value at the block's revision: the docstring and
the output then talk about one number. A unit the tool prints with 2+ values
is SECTION-SPLIT BY THE TOOL'S OWN DESIGN (`citations` prints once under
EXPLICIT and once under BARE), so a docstring that mirrors both is correct
and is reported as such, not flagged. A unit the tool never prints is not a
census unit here. Without a run, only uppercase census labels are checked.
With the oracle legs in hand a unit is checkable when some leg prints it
with exactly ONE value: the docstring and the output then talk about one
number. A unit printed with 2+ values is SECTION-SPLIT BY THE TOOL'S OWN
DESIGN (`citations` prints once under EXPLICIT and once under BARE), so a
docstring that mirrors both is correct and is reported as such, not flagged.
A unit no leg prints is not a census unit here. Without a run, only
uppercase census labels are checked.
"""
if no_run or printed is None:
if no_run or not legs:
return ("LABEL_ONLY" if is_label(unit) else "UNCHECKED_NO_ORACLE")
keys = keys_for(unit, printed)
vals = set()
for k in keys:
vals |= set(printed[k])
if not vals:
return "NOT_A_CENSUS_UNIT"
if len(vals) > 1:
saw_split = False
for printed, clone, _tag in legs:
vals = set()
for k in keys_for(unit, printed or {}):
vals |= set((printed or {})[k])
if not vals:
continue
if len(vals) > 1:
saw_split = True
continue
return "CHECKABLE"
if saw_split:
return "SECTION_SPLIT_BY_TOOL"
return "CHECKABLE"
return "NOT_A_CENSUS_UNIT"
def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240):
@@ -409,19 +694,43 @@ def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240):
if cl:
blk_claims.append((first, revs, cl, btext))
# ---- the oracle first: every revision the docstring names is run once.
# ---- the legs first. Leg 1: the tool's own run, once per revision the
# docstring names. Leg 2: its --selftest, once. The tool under test is
# never this tool: running this scanner on itself spawns it, which
# spawns it again, without a floor.
is_self = os.path.basename(script) == os.path.basename(os.path.abspath(__file__))
printed_by_rev, verdicts, failures = {}, [], []
if run:
selftest_leg = None
if run and not is_self:
for first, revs, cl, _b in blk_claims:
for rev in (revs or ["HEAD"]):
key = (name, rev)
if key not in cache:
try:
out, rc = run_tool(repo, script, rev, tmpdir)
cache[key] = (parse_printed(out), rc, out)
out, rc, tag = run_tool(repo, script, rev, tmpdir)
cache[key] = (parse_printed(out), rc, out, tag)
except Exception as exc: # noqa: BLE001
cache[key] = (None, None, f"<run failed: {exc}>")
cache[key] = (None, None, f"<run failed: {exc}>", "")
printed_by_rev.setdefault((first, rev), cache[key])
skey = (name, "<selftest>")
if skey not in cache:
try:
sout = run_selftest(script)
except Exception as exc: # noqa: BLE001
sout = f"<selftest failed: {exc}>"
cache[skey] = (parse_printed(sout) if sout else None,
0 if sout else None, sout or "", "selftest")
selftest_leg = cache[skey]
def legs_for(first, rev):
"""Priority-ordered [(printed, clone, tag)] for one block scope."""
out = []
entry = printed_by_rev.get((first, rev))
if entry and entry[0]:
out.append((entry[0][0], entry[0][1], entry[3]))
if selftest_leg and selftest_leg[0]:
out.append((selftest_leg[0][0], selftest_leg[0][1], "selftest"))
return out
# 1. INTERNAL SPLIT: one unit, one revision scope, two values, far enough
# apart to be a restatement rather than a neighbouring sub-count.
@@ -443,9 +752,8 @@ def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240):
if not far:
continue
rev_for_oracle = (items[0][0], scope[0] if len(scope) == 1 else "HEAD")
entry = printed_by_rev.get(rev_for_oracle)
pr, pr_clone = (entry[0] if entry else None) or (None, None)
why = unit_scope(unit, pr, pr_clone, not run)
why = ("SELF_NOT_RUN" if is_self and run
else unit_scope(unit, legs_for(*rev_for_oracle), not run))
if why != "CHECKABLE" and why != "LABEL_ONLY":
unchecked[why] += 1
continue
@@ -454,27 +762,72 @@ def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240):
"rows": [{"line": ln, "value": v, "block": b}
for b, ln, v, _ in items]})
# 2. ORACLE: every claim compared against what the tool prints.
# 2. ORACLE: every claim compared against what the tool prints, leg 1 first.
if run:
for first, revs, cl, _b in blk_claims:
for rev in (revs or ["HEAD"]):
pr, rc, out = printed_by_rev[(first, rev)]
if is_self:
for ln, val, unit, tok, _pos in cl:
verdicts.append({"file": name, "block": first, "line": ln,
"rev": rev, "unit": unit, "claim": val,
"token": tok, "verdict": "SELF_NOT_RUN",
"detail": "the tool under test is this tool"})
continue
pr, rc, out, _tag = printed_by_rev[(first, rev)]
printed, clone = pr if pr else (None, None)
legs = legs_for(first, rev)
# A leg with no census -- raised, or exited non-zero having
# printed nothing -- adjudicates nothing either: drop it and let
# whichever leg survives answer instead. Only when none survives
# is the block left unadjudicated.
no_census = pr is None or (rc not in (0, None)
and not any(printed.values()))
if pr is None:
failures.append({"file": name, "block": first, "rev": rev,
"unit": "-", "claim": None,
"verdict": "RUN_FAILED", "detail": str(out)[:90]})
continue
printed, clone = pr
elif no_census:
failures.append({"file": name, "block": first, "rev": rev,
"unit": "-", "claim": None,
"verdict": "RUN_FAILED",
"detail": "rc=%s, nothing printed: %s"
% (rc, str(out)[:80])})
if no_census:
legs = [l for l in legs if l[2] == "selftest"]
if not legs:
continue
for ln, val, unit, tok, _pos in cl:
v, detail = oracle(printed, clone, unit, val)
v, detail = adjudicate(legs, unit, val, revs)
verdicts.append({"file": name, "block": first, "line": ln,
"rev": rev, "unit": unit, "claim": val,
"token": tok, "verdict": v, "detail": detail})
return {"tool": name, "splits": splits, "unchecked": dict(unchecked),
"failures": failures, "verdicts": verdicts,
"self": is_self and run, "selftest_leg": (not run) or bool(selftest_leg and selftest_leg[0]),
"blocks_with_claims": len(blk_claims)}
def counts(rep):
"""(splits, claim_defects, leg_failures, claims, notes) -- the banner's rows.
A LEG FAILURE IS NOT A CLAIM DEFECT. RUN_FAILED says no census came back and
such a leg adjudicates nothing (see check_tool); STALE says a claim was read
against a census and disagreed. The first live run of the two-leg tool mixed
them in one `N STALE/failed` column, so four failed runs read as four bad
claims -- measured 2026-10-16 at HEAD, check_lint.py 2 and
queue_claim_check.py 2, none of them a claim disagreement. Two columns, so
the distinction survives the summary a reader actually reads.
"""
stale = [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
notes = []
if rep.get("self"):
notes.append("self: no leg run (SELF_NOT_RUN)")
elif not rep.get("selftest_leg"):
notes.append("no --selftest: leg 2 absent")
return (len(rep["splits"]), len(stale), len(rep["failures"]),
len(rep["verdicts"]), notes)
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
@@ -521,8 +874,11 @@ def main():
tot = collections.Counter()
for rep in reports:
bad = rep["failures"] + [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
print(f"\n{rep['tool']}: {len(rep['splits'])} INTERNAL SPLIT, "
f"{len(bad)} STALE/failed, {len(rep['verdicts'])} claims checked")
nsplits, nstale, nfail, nclaims, notes = counts(rep)
print(f"\n{rep['tool']}: {nsplits} INTERNAL SPLIT, "
f"{nstale} STALE claim(s), {nfail} leg failure(s), "
f"{nclaims} claims checked"
+ (" [" + "; ".join(notes) + "]" if notes else ""))
for s in rep["splits"]:
vals = ", ".join(f"{r['value']}@{r['line']}" for r in s["rows"])
scope = ",".join(s["revs"]) or "no rev named"
@@ -532,6 +888,10 @@ def main():
for v in bad:
print(f" {v['verdict']:10s} {v['file']}:{v.get('line', '-')} "
f"{v['unit']} claimed {v['claim']} at {v['rev']} -- {v['detail']}")
for k in sorted({v["verdict"] for v in rep["verdicts"]}):
if k in ("REV_UNCHECKED", "SELF_NOT_RUN", "CLONE_LOCAL"):
n = sum(1 for v in rep["verdicts"] if v["verdict"] == k)
print(f" ({n} {k}: reported, never counted as a defect)")
for k, c in collections.Counter(v["verdict"] for v in rep["verdicts"]).items():
tot[k] += c
print("\n--- verdict totals (unit: claims) ---")