stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7e02b24143a10fb11e262db5087a0197042297ee
troupe-standup/tools/doc_census_check.py
T
volt c3d70567a6 volt: verify gains a seventh leg -- doc-commands, graded at WORKTREE 
The gate had six legs and none of them read a docstring's own command line.
doc_command_check.py existed since 838750c but only as a tool: nothing on the
gate ran it, so a documented flag its parser never learned could land and
nobody read it. This is that leg.

  doc-commands   tools/doc_command_check.py --rev WORKTREE --fail
                 WORKTREE, not HEAD: the commit window is where a docstring is
                 edited. Measured at the worktree this lands in: 67 commands
                 over 19 oracle tools, 67 OK, rc 0.

doc_command_check.py learns --rev WORKTREE: tracked paths from `git ls-files`
(an untracked tools/x.py is not part of the commit this mode grades), bytes
from disk. Outside a work tree it is rc 2, not an empty clean run. Selftest
15 -> 18 cells: one document edited on disk but not committed is rc 0 at HEAD
and rc 1 at WORKTREE, and the flag it names is the edited one.

Three tools stopped defaulting to a fixed path on this host. verify.py,
selftest_sweep.py, write_probe.py, citation_resolve_check.py,
path_claim_check.py and doc_census_check.py now resolve DEFAULT_REPO from the
file's own location, so a bare run in a clone grades the clone.

Three selftests refuse unreadable input at rc 2 instead of reporting it as
failed content: tool_map (not a work tree), write_probe (no table), and
citation_resolve_check (no corpus). Each cell below them graded `repo`, so an
absent tree read as "your table is wrong".

Evidence, all measured on this tree before the commit:
  verify.py --selftest              55 cells, 55 passed   (docstring says 55)
  doc_command_check.py --selftest   18 cells, 18 passed   (baseline pinned 18)
  tool_map.py --repo . --rev WORKTREE --check docs/TOOLS.md   rc 0
  verify.py --repo .                7 legs, 6 pass, 1 fault -- selftest-sweep,
                                    tool_map's drift cell, which renders HEAD
                                    and so cannot pass inside a commit window
tools/selftest_baseline.tsv re-pins doc_command_check.py 15 -> 18 cells.
2026-10-22 22:05:55 +00:00

1862 lines
102 KiB
Python
Raw Blame History

#!/usr/bin/env python3
"""doc_census_check.py -- does a tool's own docstring census match the census
that tool PRINTS?
WHY THIS EXISTS
This archive's three scanners each carry a census table in their module
docstring: counts that a reader cites, that no command checks. The docstring
is prose; prose is not executed. On 2026-10-16 the control plane returned
exactly this defect against a sibling scanner, in its own words:
"The scanner's own docstring carries a number that does not match the
census it prints -- the exact class the tool exists to catch."
tools/citation_resolve_check.py said `471 UNRESOLVED` on one line and
`441 UNRESOLVED` nine lines later, over a census the tool prints as 471.
Nothing was checking the docstring against the output beside it. That is this
tool's whole job.
WHAT IT MEASURES (read-only: this tool only writes --json)
Two independent verdicts per tool, and they do not share evidence.
1. INTERNAL SPLIT -- one unit, one scope, two values. Within one docstring,
every numeric claim of the form `<value> <unit>` is collected; two claims
of ONE unit that a single revision scope covers, carrying two different
values, is a split. This catches the 441/471 instance. A split is GRADED
only when a leg verified the unit (`CHECKABLE`: the run printed it with
exactly one value); every other split -- no oracle (`LABEL_ONLY`,
`--no-run`), a unit no leg prints (`NOT_A_CENSUS_UNIT`), a unit the tool
prints per section (`SECTION_SPLIT_BY_TOOL`) -- is printed with its values
as ADVISORY and never reaches the gate. 2026-10-18: the old build failed
`--fail --no-run` on the 471/441 split at fca3f2e (rc 1) while the full run
could not see either value, so rc 1 meant "could not check this".
2. ORACLE -- the tool is RUN, at the revision its own docstring names, and
each claim is compared against the value that run prints for that unit.
MATCHES claim == printed value
STALE claim != printed value, and the printed unit is not
declared clone-local -- a defect claim
CLONE_LOCAL claim != printed value but the tool's own output marks
that unit clone-local ("blobs in THIS clone's local object
store"), so drift is the documented expectation, not a
defect. Reported, never counted as one.
AMBIGUOUS the tool prints that unit with 2+ values at that rev
(e.g. `citations` prints once per section), so there is no
single oracle value; both are printed and no claim is made
NOT_PRINTED the unit does not appear in this tool's output at all
A claim is `<digits> <unitword>` with the unit word ADJACENT, which is the
archive's own notation. A bare number is not a claim and is not checked.
A WRAPPED PAIR IS A CLAIM ONLY WHEN A LEG CAN BIND IT. A number that ENDS one
line with its unit opening the next is the same claim the archive writes on
one line, and until 2026-10-20 this reader was line-local: a docstring reflow
retired the claim silently while every command exited 0 (measured -- the
pinned pair in `queue_claim_check.py` fell out of the census and the claim
count fell with it, with no verdict moving). The reader now joins the pair,
under the same refusals the adjacency reader already applies: a number welded
to a preceding `-`, `/`, `.`, `:`, `_` or letter is not standalone, so a
date's tail and `UTF-8` and `0/1` are not counts, and a unit with a hyphen
behind it is a compound adjective. Whether the joined pair is SPENT is a
second question, and it is answered by the census rather than by a lexicon of
English: the pair is graded only when some leg prints its unit UNDER THAT
SPELLING, and every pair no leg binds is printed as WRAP_UNBOUND and counted
in the banner, never dropped. The spelling is exact and that is deliberate: a
plural fold (`notes` -> `note`) is affordable for a one-line claim, because
both sides are census voice, but a wrap is this reader's hypothesis about two
lines of prose. Measured at HEAD 2026-10-20, the fold spent `the queue's own
reply 62 / notes it` and graded a correct sentence STALE 62-vs-7; with the
fold refused, that pair is WRAP_UNBOUND and the one wrapped claim a leg really
binds -- `1,062 CONTENT_SHA256` at `citation_hash_class.py:53` -- is the only
one graded. The other five candidates are named, not graded: `69 filenames`
(`checkin_check.py:14`), `18 cells`, `24 spendable` and `0 without` (all
three in this file -- its `--selftest`, fast-mode and `--fail` paragraphs),
and `62 notes` (`queue_claim_check.py:119`). With `--no-run` there is no leg, so
no wrap is spent and each one is reported unbound -- a mode that cannot see a
census cannot bind a claim.
TWO ORACLE LEGS, AND WHY THE SECOND ONE EXISTS
Leg 1 is the tool's own run at the block's revision, exactly as above. Leg 2
is the SAME TOOL'S `--selftest`, added 2026-10-16 after the control plane
returned a docstring claim of `13 cells` against a selftest that prints `18
cells` -- the defect class this tool exists to catch, sitting inside its own
default scope and still read as NOT_PRINTED, because no leg ever passed
`--selftest` and a cell count is printed by nothing else. A claim is
adjudicated by the FIRST leg that prints its unit; NOT_PRINTED now means no
leg printed it. A tool that takes no `--selftest` simply has one leg -- a fact
about the tool, not a defect in it.
A leg that exits non-zero having printed nothing is reported RUN_FAILED, never
read as an empty census, and it adjudicates nothing either: the block's claims
are still read off any surviving leg -- leg 2 -- and are left unadjudicated
only when no leg survives, because a failed run is not a reason to stop
looking. A revision that will not materialise fails the same way rather than
producing a census at all: an empty queue is a census of zeros, and reading it
as that revision's census is the phantom this rule refuses.
A HEX RUN INTRODUCED AS A DIGEST IS NOT A REVISION. A block that quotes a
truncated sha256 in its prose is a size, not a place: read as a rev it files
the block under a revision that cannot exist, fails that leg, and then
adjudicates every claim in the block again beside the real ones. Two signals
are read, both local to the token -- a digest word in front of it (`sha256
<hex>`, `hash <hex>`), or an ellipsis behind it (`<hex>…`), which is how this
archive writes a truncated digest. See `block_revs`.
The run leg is `<script> --repo R --rev REV`. A tool that takes no `--rev`
answers it with an argparse diagnosis and prints no census at all, so the leg
is re-run with the rejected flags DROPPED -- the exact tokens argparse named,
and only those -- and is tagged `no-rev`; a recipe's leg may carry the same tag
for the same reason. Such a tool measures the tree it is POINTED AT, so a claim
in a block that names a revision is reported REV_UNCHECKED -- never counted as
verified against that revision, and never counted as a defect -- while a block
that names no revision is checked normally, since the default is what the flag
means there.
THIS TOOL DOES NOT RUN ITSELF. The default scope is every `tools/*.py` in the
tree, and this tool is one of them. Running it on itself spawns it, which
spawns it again, without a floor -- measured, not reasoned, in the check-in of
2026-10-16. The self target is therefore excluded from the run leg and every
one of its claims is reported SELF_NOT_RUN. It is deliberately still listed in
the scanned set, so the exclusion is a printed row and not a silence.
SCOPE, STATED RATHER THAN IMPLIED
Only module docstrings of .py tools are scanned: a docstring is where a census
table lives. Only revisions NAMED IN THE BLOCK are used as oracles -- a block
naming no revision is checked against the tool's default (HEAD) and tagged, so
a stale rev is never silently attributed. A tool whose subject is not `--repo`
+ `--rev` is driven through a per-tool recipe declared below, with a temp file
that is removed: `queue_claim_check.py` takes a materialised QUEUE file and so
its leg IS the revision, while `append_only_owners.py` takes a tree scan that
no flag can pin to a revision, so its leg is tagged `no-rev`. A tool no recipe
covers, and that answers `--repo ... --rev ...` with an argparse diagnosis, is
re-run with the rejected flags dropped and tagged the same way -- measured at
HEAD on 2026-10-16 against `check_lint.py`, which takes neither flag and whose
default census is a real one.
WHAT THIS DOES NOT CHECK, and the measured size of the gap. A claim is only
adjudicated when the unit it names is a unit the tool PRINTS. Over the three
scanners at d355395 the run read ninety-two claims (with the pre-2026-10-18
separator rule, which read punctuation and list markers as claims): twelve
were verified against the run they point at, eight were in units the tool
prints with two values (SECTION_SPLIT_BY_TOOL -- no oracle), and seventy-one
were in units the tool never prints at all. So the oracle's reach is narrow
and its precision is the reason: it reported exactly one bad claim on that
corpus, and that one was the returned defect. A wider rule was tried first and
falsified by its own first run -- num_agreement_check.py flagged a hundred and
seventeen restatements across the archive, too coarse to be read -- so the
narrow rule is the one that ships. Two claim forms are read (`<value> <unit>` and
`<unit>: <value>`); a claim written in neither form is not seen, and that is a
stated gap, not a silent one.
NO CENSUS OF THIS TOOL'S OWN VERDICTS APPEARS IN THIS DOCSTRING, deliberately:
this scanner scans itself, so a count of its own output written here would
have to agree with the output of the run that the same commit changes. That is
the limit citation_hash_class.py records in the same words -- a commit cannot
measure the file it introduces -- and here it is avoided rather than
discovered.
FAILING ON THE CLASS IT CATCHES (`--fail`)
Without it the process exits 0 while printing STALE rows, so no gate could
act on the one class this tool exists to catch without parsing its text.
`--fail` makes the exit code carry the verdict. The cost of the old silence
is measured twice, and the two are NOT the same measurement:
- STRUCTURAL -- reproducible right here, no archive needed. A stub tree
whose docstring says `99 widgets` beside a printed `widgets: 7` exits 0
without `--fail` and 1 with it (`gate e2e` cells in `--selftest`).
- HISTORICAL -- true only of the pre-tightening separator (`ac0e211^`).
That build at rev 4ce1aed, `--tools tools/checkin_check.py`, exits 0
with two STALE verdicts on `checkin_check.py:59`: `Never claimed 8` and
`never claimed 1`, both `printed 2 [selftest]` -- the digits in "Never 1,
never a traceback". Reproduced 2026-10-18 from this host. The tightened
separator stops reading those digits as a census: the same rev now
adjudicates 7 claims, 0 STALE. So the historical run is cited with its
build and never as current behaviour -- a claim that says "rc 0 with two
STALE" without naming the build goes stale the moment the reader is
tightened, which is the failure this tool grades other files for.
The code means:
0 ran, and graded no fault (every claim adjudicated agrees)
1 ran, and graded a fault (an INTERNAL SPLIT whose unit a leg --
or a cached vocabulary -- verified,
or a STALE claim)
2 could not grade (usage, a --tools filter that matches
nothing tracked, an unreadable
--vocab, --save-vocab without a run)
THE TWO MODES GRADE THE SAME CLASS. `--no-run` runs nothing, so it has no
oracle and cannot prove any split is a census unit: it prints every split as
ADVISORY with its values, and `--fail --no-run` exits 0. That is the ruling
of reply 22 (2026-10-18), and it fixes a real asymmetry: `--fail --no-run`
returned 1 on `citation_resolve_check.py` at fca3f2e (471@82 / 441@89) while
the same units were advisory under a full run. Measured after the repair at
the same rev, one mode apart: `--fail --no-run` rc 0 with the pair printed as
ADVISORY; `--fail` (full) rc 1 on the same split, now CHECKABLE because the
run prints `UNRESOLVED 471` -- plus the STALE claim at :89. The teeth are
kept where they can be verified, which is the only place they mean anything.
A CACHED VOCABULARY PUTS THE TEETH BACK IN THE FAST MODE, without asking it
to invent an oracle. `--save-vocab PATH` (full run) writes the census
vocabulary the run OBSERVED -- every unit key some leg printed AND that a
claim in that tool's own docstring can spend, per tool -- and
`--no-run --vocab PATH` grades a split in a unit that vocabulary names as
CACHE_CHECKABLE. The two halves are both required: the leg proves the key is
something the tool actually prints, the docstring proves some claim could
ever ask about it. A key no claim can spend (`keys_for`'s expansion of every
claimed unit) is dropped, because a permission list is only auditable when
every entry in it can be exercised -- 422 observed keys at 30c8120, 24
spendable, the other 398 unspendable by construction. That scoping is
measured to be outcome-neutral (see THE FAST MODE REPORTS WHAT IT CANNOT
GRADE); it does NOT close the cache's one over-reach, which is a word the
docstring really does claim twice (below).
What the cache proves is narrow and enough: the unit is a
census unit for that tool (a leg printed it at `<vocab rev>`), and the
disagreement being graded is INTERNAL to the docstring, so the leg never has
to supply the value. What it cannot prove is a value, so a STALE claim is
still impossible without a run, and a unit the vocabulary does not name stays
ADVISORY -- the 39/12/1 `ACS-only` class is still never spent. Measured
2026-10-18 at 814acc4 on a stub whose docstring splits 471/441 over a census
printing 471: `--no-run --fail` rc 0, `--no-run --vocab V --fail` rc 1,
`--fail` rc 1. The vocabulary's rev is printed with every CACHE_CHECKABLE
row, because a reader is entitled to know how old the census behind the
label is; it is NOT required to equal the revision being checked (unit-hood
is a property of the unit word, and the graded disagreement is internal).
THE FAST MODE REPORTS WHAT IT CANNOT GRADE. Before 2026-10-18 a runless split
in a lowercase word vanished: `unit_scope` answered UNCHECKED_NO_ORACLE, and
the printer showed only the LABEL_ONLY class, so a reader of `--no-run` saw
nothing at all for it (`vocab-precision` case E: rc 0, no line). It is now
printed as ADVISORY with its values and never graded -- silent was the only
part of that behaviour that was wrong, and silence is not a verdict. Scoping
the saved vocabulary to spendable keys is measured OUTCOME-NEUTRAL on the
same fixture (A..E still [1,1,0,0,0]) and on the tree (fast gate rc 0 before
and after; the fca3f2e tooth still rc 1 from the cache), which is the honest
description: it makes the permission list auditable, it does not change a
verdict, and it does not close the over-reach this paragraph opened with.
rc 2 keeps its meaning from the tools/ contract (6ee8fce) and is decided
before the gate runs, so "could not grade" never reads as "graded clean" or
as "graded a fault". A LEG FAILURE (RUN_FAILED) is reported in its own column
and does not fail the gate, for the reason counts() states: a leg that
produced no census adjudicated no claim. A gate that counted it would fail
runs it never checked.
USAGE
python3 doc_census_check.py [--repo DIR] [--rev HEAD] [--tools NAME...]
[--json OUT] [--fail] [--no-run] [--vocab PATH] [--save-vocab PATH]
[--selftest] # selftest: 104 cells, one per claim class
"""
import argparse
import ast
import collections
import json
import os
import re
import subprocess
import sys
import tempfile
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# A claim, in either of the two forms this archive writes:
# `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations"
# `<unit> [: =] <value>` -- "ACS = 72", "overlap = 20" (markdown bold allowed)
# A bare number is not a claim. Deliberately not "any word": `471 cannot be
# resolved` is prose, and a checker that flags prose is a checker nobody reads.
#
# THE SEPARATOR IS SPACE, OR A CLOSING DELIMITER AND A SPACE -- tightened
# 2026-10-18 after the census was read off every tool at HEAD (140 claims) and
# the separator vocabulary was printed: `''` (11 claims, a hex run read as
# digits+word: `83acffe0` -> 83 `acffe0`), `'. '` (5, an ordinal or a sentence
# end: `1. INTERNAL`, `615. Cite`), `'-'` (5, a compound adjective: `120-char`,
# `1-based`), `' '` (4, an aligned list marker: `0 ran`), and the punctuation
# spills `'; '` `' "'` `'+ '` `') '`. Only 95 of the 140 were a count and its
# unit. The loose class fired two STALE verdicts in the full run at 4ce1aed
# against `checkin_check.py:59` -- "Never 1, never a traceback", where the digit
# is a code, not a census -- which is a false defect claim of exactly the kind
# this archive grades lanes for. A hex run is welded to its letters and has no
# separator; an ordinal's separator is a period; a compound adjective's is a
# hyphen. None of the three is `<value> <unit>`.
#
# A HYPHEN AFTER THE UNIT IS THE SAME REFUSAL, made 2026-10-18 after the
# --fail gate went red at HEAD on a false claim: `queue_claim_check.py`'s axis
# list says `39 ACS-only lines carry no Q<digits> token`, `12 ACS-only lines
# are indented`, `1 ACS-only line carries a BQ exclusion phrase` -- three
# sub-counts of ONE population of 52, and the token read as a unit was the
# adjective `ACS` in `ACS-only`. With no oracle to test the label against, a
# `--no-run` split fired on 39/12/1. A hyphen immediately behind the unit word
# marks a compound adjective, not a unit noun, exactly as a hyphen behind the
# number does (`120-char`). The cost is stated: a genuinely hyphenated unit
# (`citations-per-block`) is no longer read, and the invariant that licenses
# this tightening is that the 26 MATCHES do not move -- measured before and
# after, same 26 rows.
#
# A COLON BEFORE THE NUMBER IS A LINE CITE, NOT A COUNT, made 2026-10-19 after
# the --fail gate went red at HEAD on the same tool it went red on for the
# hyphen: `queue_claim_check.py`'s docstring records its own past behaviour as
# "graded `:7303` CLOSED on Mercury's `:9869`" and "printed `open set:
# Q37:5278`", and the reader took `7303` for a count of CLOSED rows (printed
# 13) and of open rows (printed 2). Both are queue LINE NUMBERS in backticks,
# the same shape `CLAIM_EQ` already refuses one rule over ("`WORK-QUEUE.md:8586`
# is a citation"); the value/unit form had no such guard because no docstring
# had yet written `:NNNN <state word>`. Measured on every tracked docstring at
# HEAD: eleven of the hundred and twenty-three claims read carry a ':' immediately
# before the number, and every one is a line cite across four tools --
# `citation_hash_class.py` (:8467 defect),
# `citation_resolve_check.py` (four: :8003 cites, :6348 column, :9 reads, :8530
# Mercury), `queue_claim_check.py` (five, the two false STALEs among them) and
# `queue_index.py` (:7246 prose). Not one names a unit the tool prints. The
# invariant that licenses this tightening is that no MATCHES row moves, the
# same invariant the hyphen rule shipped under; the cost is that a count
# written `:13 claims` is no longer read, and no scanner writes one.
CLAIM = re.compile(
r"(?<![0-9A-Za-z_.:])(?P<num>[0-9]+(?:,[0-9]{3})*)(?P<sep>\s{1,2}|[)\]}\"'`*]\s)"
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Z_]{2,})\b(?!-)")
CLAIM_EQ = re.compile(
r"(?P<unit>[A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*\**\s*(?P<num>[0-9]+(?:,[0-9]{3})*)")
# A WRAPPED PAIR: the number ENDS one line and its unit OPENS the next. Until
# 2026-10-20 this reader was line-local, so a claim that a docstring's 79-column
# wrap cut in two was not a claim at all. That is not cosmetic: `queue_claim_check
# .py`'s pinned pair `1,229,541 B / 9,953 lines` was broken onto two lines at
# `70fa8b2` and the census silently fell from seventeen claims to sixteen with no
# verdict moving -- the archive's own instrument, blind to the archive's own
# notation, while every command exited 0. Measured at HEAD 2026-10-20 by running
# this reader over all sixteen tracked `tools/*.py` MODULE docstrings -- the only
# scope docstring_span reads: SIX
# number-ends-a-line candidates -- FOUR real counts with their unit
# (`69 filenames` at checkin_check.py:14, `1,062 CONTENT_SHA256` at
# citation_hash_class.py:53, `18 cells` and `24 spendable` in this file's
# `--selftest` and fast-mode paragraphs) and TWO words that merely open a
# sentence (`0 without` in the `--fail` paragraph here, `62 notes` at
# queue_claim_check.py:119). The same reader at `70fa8b2`, where the break was
# introduced, reads seven -- the same six plus the pinned `9,953 lines` pair --
# and spends it, so that docstring reads sixteen claims against the old reader
# and seventeen against this one. `4e6b2ef`, the commit that put the pair back on
# one line, is the control on the other side: it reads six candidates and
# seventeen claims on BOTH readers, so the six is a measurement of HEAD's prose
# and not a fallback the join quietly invents. A twelve-row artifact was produced
# that day by a looser net that counted every number ending a line with any word
# after it, non-counts included; the reader itself reads six.
# The same measurement over FUNCTION and CLASS docstrings -- which this tool does
# not read at all, module docstrings only -- finds two more real counts, `29
# literal` in append_only_audit.py and `74 files` in checkin_check.py. They are
# named here rather than quietly excluded: a scope the reader cannot see is
# worth knowing, and the join would spend either one if the scope ever widened.
#
# WHY THERE IS NO WORD LIST HERE. The obvious reader is "join the wrap unless the
# unit is a verb", and it needs an English lexicon that would refuse a real claim
# the day one is written in an unlucky voice. The discrimination is made where
# this tool already makes it: A WRAPPED PAIR IS SPENT ONLY WHEN A LEG PRINTS ITS
# UNIT (see check_tool). At HEAD that spends exactly one of the six --
# `1,062 CONTENT_SHA256`, whose census prints the label -- and the other five
# (`69 filenames`, `18 cells`, `24 spendable`, `0 without`, `62 notes`) are
# printed unbound and never graded. The structural half below is the part that
# must not be guessed at, and it is the same half the adjacency reader already
# enforces: a number welded to a preceding `-`, `/`, `.`, `:`, `_` or
# alphanumeric is not standalone (`2026-10-18`, `UTF-8` and `0/1` are three of
# the strings this refuses, each one a refusal cell in the selftest below), and
# a unit with a hyphen behind it is a compound adjective, not a unit noun.
WRAP_LEAD = re.compile(r"(?:^|[\s(\[{*`'\"])(?P<num>[0-9]+(?:,[0-9]{3})*)[ \t]*$")
WRAP_UNIT = re.compile(r"^[ \t]*(?P<unit>[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Z_]{2,})\b(?!-)")
REV_LONG = re.compile(r"\b[0-9a-f]{7,40}\b")
# A hex run introduced as a DIGEST is not a revision, and reading it as one costs
# more than a wrong label: the block is filed under a revision that cannot exist,
# its leg fails, and then every claim in the block is adjudicated once per
# spurious rev beside the real ones. Measured 2026-10-16 at HEAD on
# `tools/queue_claim_check.py`, whose UNITS block quotes the queue's size as
# "/ sha256 `fbe51445a89f81ab…`": three claims emitted FOUR times, once per
# REV_LONG token, and two `git show <sha256-prefix>:WORK-QUEUE.md` leg failures.
# Two signals, both local to the token: a digest word in front of it (sha256,
# sha1, md5, digest, hash), or an ellipsis behind it -- the archive's own way of
# writing a truncated digest. A revision cited in the ordinary way carries
# neither, and a FULL sha256 is already invisible to REV_LONG: the run has no
# word boundary to bite on.
DIGEST_BEFORE = re.compile(
r"(?:sha[-\s]?256|sha[-\s]?1|md5|digest|hash)\s*[:=]?\s*[`'\"]?\s*$", re.I)
DIGEST_AFTER = re.compile(r"^[`'\"]?\s*(?:…|\.\.\.)")
def block_revs(text):
"""The revisions a block names -- hex runs read as digests excluded."""
out = set()
for m in REV_LONG.finditer(text):
if DIGEST_BEFORE.search(text[:m.start()]) or \
DIGEST_AFTER.match(text[m.end():]):
continue
out.add(m.group(0))
return sorted(out)
# Function words and units this tool does not adjudicate. `at` in "2026-10-15 at
# rev 9188dd3" is not a census quantity; neither is `B` (bytes differ by
# revision by design, and "1,201,550 B" is a size, not a count). Stated so the
# exclusion is a rule rather than a habit.
STOPWORDS = {
"at", "the", "of", "in", "is", "are", "was", "and", "or", "to", "for", "on",
"by", "as", "it", "its", "that", "this", "these", "those", "with", "from",
"no", "not", "one", "two", "three", "all", "any", "be", "been", "has",
"have", "had", "his", "her", "our", "your", "their", "but", "if", "then",
"than", "so", "up", "out", "off", "over", "into", "run", "runs", "may",
"can", "same", "each", "both", "per", "via", "b", "kb", "mb", "gb",
}
# How to run each tool at a revision. Any tool not listed here is run as
# `<script> --repo R --rev REV`. A recipe returns `(argv, tag)` and the tag says
# whether the recipe could materialise the revision the block names.
#
# queue_claim_check.py takes no --rev: its subject is a FILE, and the file is
# materialised at the revision and handed to it -- so the leg IS that rev.
# A revision that will NOT materialise is a raised failure, never an empty
# file: an empty queue is a census of zeros, and a census of zeros reads as
# that revision's census. Measured 2026-10-16 -- `git show
# c3b4a290dc64bd24:WORK-QUEUE.md` (a sha256 prefix REV_LONG read as a rev)
# exits 128, and the leg it fed printed `0 opening records` at rc 0.
# append_only_owners.py takes a positional AUDIT.json and nothing else: its
# input is a tree scan, which its sibling produces over the WORKING TREE
# (`append_only_audit.py` has no --rev either). This scanner is read-only, so
# it will not add a git worktree to the repository it is only allowed to
# read: the leg is tagged no-rev and every claim in a block that names a
# revision is reported REV_UNCHECKED, never adjudicated against a tree it
# did not stand on.
def _recipe_queue(repo, rev, tmpdir):
dst = os.path.join(tmpdir, f"WORK-QUEUE.{rev}.md")
p = subprocess.run(["git", "-C", repo, "show", f"{rev}:WORK-QUEUE.md"],
capture_output=True)
if p.returncode != 0:
raise RuntimeError("git show %s:WORK-QUEUE.md -> rc=%d: %s"
% (rev, p.returncode,
p.stderr.decode("utf-8", "replace").strip()[:120]))
with open(dst, "wb") as fh:
fh.write(p.stdout)
return ["--queue", dst, "--census"], ""
def _recipe_owners(repo, rev, tmpdir):
dst = os.path.join(tmpdir, f"append-only-audit.{rev}.json")
subprocess.run([sys.executable,
os.path.join(repo, "tools", "append_only_audit.py"),
"--repo", repo, "--json", dst], capture_output=True)
return [dst], "no-rev"
RECIPES = {"queue_claim_check.py": _recipe_queue,
"append_only_owners.py": _recipe_owners}
def git(repo, *a):
return subprocess.run(["git", "-C", repo, *a], capture_output=True).stdout
def docstring_span(path, text):
try:
tree = ast.parse(text)
except SyntaxError:
return None
if not tree.body or not isinstance(tree.body[0], ast.Expr):
return None
node = tree.body[0]
if not isinstance(node.value, ast.Constant) or not isinstance(node.value.value, str):
return None
return node.lineno, node.end_lineno
def blocks(lines, lo, hi):
"""Blank-line-delimited paragraphs -- the same scope the sibling scanners
use (citation_resolve_check.py for inheritance, num_agreement_check.py for
quantity agreement), so one paragraph is one scope in all three."""
start, buf = lo, []
for ln in range(lo, hi + 1):
raw = lines[ln - 1]
if raw.strip() == "":
if buf:
yield start, buf
buf, start = [], ln + 1
else:
buf.append((ln, raw))
if buf:
yield start, buf
# ------------------------------------------------- printed census -> oracle
def parse_printed(stdout):
"""stdout -> {unit: {value: [line, ...]}} plus the set of clone-local units.
Four line shapes are read, all of them the scanners' own:
` RESOLVES 113` indented label then count (family rows)
` citations: 410` label then count
`corpus: 307 files at ...` count then unit
`overlap A&B = 20 ; ...` inline assignments, split on the separators
A unit whose printed line (or the line above it, for a wrapped phrase) says
"clone" is clone-local: its count is expected to move with the clone.
"""
units = collections.defaultdict(lambda: collections.defaultdict(list))
clone = set()
lines = stdout.split("\n")
def add(unit, value, ln, cloneline):
u = unit.strip().lower()
if len(u) < 3 or not re.fullmatch(r"[a-z_][a-z0-9_]*", u) or u in STOPWORDS:
return
units[u][value].append(ln)
if "clone" in cloneline.lower():
clone.add(u)
for i, raw in enumerate(lines, 1):
ctx = raw + " " + (lines[i - 2] if i >= 2 else "")
m = re.match(r"^\s{2,}([A-Za-z_][A-Za-z0-9_]*)\s+([0-9][0-9,]*)\s*$", raw)
if m: # family row: label count
add(m.group(1), int(m.group(2).replace(",", "")), i, ctx)
continue
m = re.match(r"^\s*([A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*([0-9][0-9,]*)\b", raw)
if m: # label: count
add(m.group(1), int(m.group(2).replace(",", "")), i, ctx)
for m in re.finditer(r"([0-9][0-9,]*)\s+(?:[a-z]+ )?([A-Za-z_][A-Za-z0-9_]*)\b", raw):
add(m.group(2), int(m.group(1).replace(",", "")), i, ctx)
for seg in re.split(r"[;]", raw): # inline `X = N`
m = re.match(r"^\s*(.*?)\s*[:=]\s*([0-9][0-9,]*)\s*$", seg)
if not m or not m.group(1):
continue
words = re.findall(r"[A-Za-z_][A-Za-z0-9_]*", m.group(1))
if not words:
continue
val = int(m.group(2).replace(",", ""))
add("_".join(w.lower() for w in words), val, i, ctx) # whole phrase
add(words[-1], val, i, ctx) # the nearest label: `overlap A&B`
# the scanners' own unit declaration: `... 6961 (unit: occurrences)`.
# The unit is named AFTER its number, which no other rule reads.
for m in re.finditer(r"\(\s*unit\s*:\s*([A-Za-z_][A-Za-z0-9_]*)\s*\)", raw):
left = raw[:m.start()]
nums = re.findall(r"([0-9][0-9,]*)\s*$", left.rstrip())
if nums:
add(m.group(1), int(nums[-1].replace(",", "")), i, ctx)
return {k: dict(v) for k, v in units.items()}, clone
def ordinal_at(raw, pos):
"""True when `pos` is the number that OPENS this line and 2+ spaces follow.
That is a list marker -- the exit-code block's `0 ran; nothing to report`,
`1 ran, and graded ...`, `2 could not grade` -- and the word after it is
prose, not a unit. The same shape mid-line (`WORK-QUEUE.md:8003 cites`) is
a citation's line number, not a count, and is excluded by the same rule.
"""
m = re.match(r"[ \t]*([0-9][0-9,]*)([ \t]{2,})", raw)
return bool(m) and m.start(1) == pos
def claims_in(blk):
"""[(line, value, unit, token, offset, wrapped)] for one block.
Both claim forms are read and a claim found by both is reported once. The
offset is measured on the block text so a RESTATEMENT (far apart) can be
told from an ENUMERATION of neighbouring sub-counts, the same
discrimination num_agreement_check.py applies.
`wrapped` is True for a claim whose number ends one line and whose unit opens
the next -- see WRAP_LEAD. It is a reading decision only: whether such a pair
is SPENT is decided in check_tool, by whether a leg prints its unit.
"""
out, off, starts = [], 0, []
for ln, raw in blk:
starts.append(off)
found = {}
for m in CLAIM.finditer(raw):
u = m.group("unit")
if len(u) < 3 or u.lower() in STOPWORDS or re.fullmatch(r"[qQ]\d+", u):
continue
if ordinal_at(raw, m.start()):
continue
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
for m in CLAIM_EQ.finditer(raw):
u = m.group("unit")
if (len(u) < 3 or u.lower() in STOPWORDS
or re.fullmatch(r"[qQ]\d+", u) or m.start() in found):
continue
if m.start() and re.match(r"[0-9A-Za-z_.]", raw[m.start() - 1]):
continue # `WORK-QUEUE.md:8586` is a citation
found[m.start()] = (int(m.group("num").replace(",", "")), u, m.group(0))
for pos in sorted(found):
val, u, tok = found[pos]
out.append((ln, val, u, tok, off + pos, False))
off += len(raw) + 1
for i in range(len(blk) - 1):
ln, raw = blk[i]
m = WRAP_LEAD.search(raw)
if not m:
continue
um = WRAP_UNIT.match(blk[i + 1][1])
if not um:
continue
u = um.group("unit")
if len(u) < 3 or u.lower() in STOPWORDS or re.fullmatch(r"[qQ]\d+", u):
continue
if ordinal_at(raw, m.start("num")):
continue
num = m.group("num")
out.append((ln, int(num.replace(",", "")), u, f"{num} {u}",
starts[i] + m.start("num"), True))
out.sort(key=lambda r: r[4])
return out
def is_label(unit):
"""An uppercase census label -- the vocabulary a split is checked in when
the tool has not been run (no oracle available)."""
return len(unit) >= 3 and re.fullmatch(r"[A-Z][A-Z_]*", unit) is not None
def keys_for(unit, printed):
"""Candidate oracle keys for a claimed unit word."""
cands = [unit, unit.lower(), unit.lower().rstrip("s"), unit.upper()]
seen, out = set(), []
for c in cands:
if c.lower() in printed and c.lower() not in seen:
seen.add(c.lower())
out.append(c.lower())
return out
def oracle(printed, clone, unit, value, tag=""):
"""(verdict, detail) for one claim; `printed` is one revision's map.
`tag` names the leg a verdict came from when it is not leg 1 ("no-rev",
"selftest"): a reader can then tell which census the number was read off.
"""
keys = keys_for(unit, printed)
vals = set()
for k in keys:
vals |= set(printed[k])
if not vals:
return "NOT_PRINTED", ""
if len(vals) > 1:
return "AMBIGUOUS", "printed: " + ", ".join(str(v) for v in sorted(vals))
printed_val = vals.pop()
if printed_val == value:
return "MATCHES", ""
if any(k in clone for k in keys):
return "CLONE_LOCAL", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
return "STALE", f"printed {printed_val}" + (f" [{tag}]" if tag else "")
def adjudicate(legs, unit, value, block_revs):
"""The first leg that prints the unit adjudicates the claim.
`legs` is [(printed, clone, tag)] in priority order: leg 1 (the run) first,
then the selftest leg. NEITHER a `no-rev` leg NOR the selftest leg is a
verifier for a block that names a revision: one measured a different tree,
the other measured the tool's own fixtures. Such a claim is reported
REV_UNCHECKED -- visible, counted as neither a match nor a defect.
"""
for printed, clone, tag in legs:
if not printed or not keys_for(unit, printed):
continue
if block_revs and tag == "no-rev":
return "REV_UNCHECKED", ("block names %s; the tool takes no --rev, so this "
"leg measured the working tree"
% ",".join(block_revs))
if block_revs and tag == "selftest":
return "REV_UNCHECKED", ("block names %s; the selftest leg prints this "
"tool's own fixtures, not a census at that "
"revision" % ",".join(block_revs))
return oracle(printed, clone, unit, value, tag)
return "NOT_PRINTED", ""
# --------------------------------------------------------------- self-test
# Synthetic stdout, so the parsers and the verdicts are executed, not asserted.
FAKE_STDOUT = """\
corpus: 297 markdown files at babdadc in /root/troupe-standup
--- EXPLICIT `<path>:<line>` -- a CLAIM (unit: citations) ---
citations: 410
RESOLVES 113
STALE_LINE 0
--- BARE `:<line>` -- a CANDIDATE, never a claim (unit: citations) ---
citations: 1612
UNRESOLVED 471
file mentions with no line: 6961 (unit: occurrences) -- these seed the scope
index: 871 blobs (unit: blobs in THIS clone's local object store), 680 commits
overlap A&B = 20 ; ACS-only = 52
"""
FAKE_DOC = """\
THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md)
BARE 1612 citations: 255 EXTERNAL_NAMED / 471 UNRESOLVED.
Read every figure as a citation count, never as a defect count: 441 UNRESOLVED
is not 441 errors.
A SECOND BLOCK, no revision named, quoting 471 UNRESOLVED and nothing else.
"""
# Leg 2's shape: a selftest summary line, and nothing else. The cell count is
# printed here and nowhere else, which is why a claim about it needs this leg.
FAKE_SELFTEST = """\
c1 a NEW byte PASS
selftest: 18 cells, 18 passed, 0 failed
"""
def _fixture_checks():
printed, clone = parse_printed(FAKE_STDOUT)
ok = True
cells, bad = [0], [0]
def check(name, got, want):
nonlocal ok
good = got == want
cells[0] += 1
bad[0] += 0 if good else 1
ok = ok and good
print(f"{'PASS' if good else 'FAIL'} {name}")
if not good:
print(f" got {got}")
print(f" want {want}")
check("parse: UNRESOLVED single", sorted(printed.get("unresolved", {})), [471])
check("parse: citations ambiguous", sorted(printed.get("citations", {})), [410, 1612])
check("parse: count-then-unit", sorted(printed.get("occurrences", {})), [6961])
check("parse: one word skipped", sorted(printed.get("files", {})), [297])
check("parse: clone-local marked", "blobs" in clone, True)
# the phrase rule: `ACS-only = 52` must NOT register as the unit `acs`
check("parse: inline phrase key", sorted(printed.get("acs_only", {})), [52])
lines = FAKE_DOC.split("\n")
got, first_block, revs_seen = [], [], set()
for first, blk in blocks(lines, 1, len(lines)):
revs = block_revs("\n".join(r for _, r in blk))
revs_seen |= set(revs)
for ln, val, unit, _tok, _pos, _w in claims_in(blk):
got.append((unit, val))
if first == 1:
first_block.append((unit, val))
# the fixture that matters: one unit, one rev scope, two values -> a split
check("claims: both UNRESOLVED values in one block",
[uv for uv in first_block if uv[0] == "UNRESOLVED"],
[("UNRESOLVED", 471), ("UNRESOLVED", 441)])
check("claims: `471 cannot be resolved` is not a claim",
any(u in ("cannot", "resolved") for u, _ in got), False)
# the tightened separator, one cell per form it now refuses, plus the
# positive control -- a rule that refuses everything proves nothing
def claims_of(text):
return [(u, v) for _ln, v, u, _t, _p, _w in claims_in([(1, text)])]
check("claims: a welded digit-and-letter run is not a claim",
claims_of("sha 83acffe0 / rev 9188dd3 at HEAD"), [])
check("claims: an ordinal is not a claim",
claims_of("1. INTERNAL SPLIT -- no oracle, no run"), [])
check("claims: a list marker is not a claim",
claims_of("0 ran; nothing to report"), [])
check("claims: a sentence end is not a claim",
claims_of("read as UTF-8. Never 1, never a traceback"), [])
check("claims: a compound adjective is not a claim",
claims_of("a 120-char window and a 1-based index"), [])
# the same refusal on the unit side, which is where the 2026-10-18 false
# split came from: `ACS-only` is a compound adjective, not a unit
check("claims: a hyphenated unit word is a compound adjective, not a claim",
claims_of("39 ACS-only lines carry no Q<digits> token"), [])
check("claims: the same unit unhyphenated is still a claim",
claims_of("39 ACS lines carry no Q<digits> token"), [("ACS", 39)])
check("claims: the tightening keeps a count and its unit",
claims_of("471 UNRESOLVED and 1,612 citations"),
[("UNRESOLVED", 471), ("citations", 1612)])
# the colon rule, 2026-10-19: a queue line number in backticks followed by a
# state word is a CITE; the positive control shows the guard is only the
# character immediately before the number, not a blanket refusal
check("claims: a line cite is not a claim",
claims_of("graded `:7303` CLOSED on Mercury's `:9869`"), [])
check("claims: a count beside a line cite is still a claim",
claims_of("open set: Q37:5278 while 13 rows close"), [("rows", 13)])
check("claims: the block's rev is read", "babdadc" in revs_seen, True)
# THE WRAPPED PAIR, 2026-10-20. A count that ends one line with its unit
# opening the next is the same claim the archive writes on one line, and
# every ADMIT text below is a real wrap copied out of the docstring it was
# measured in -- not an invented example. The four ADMIT cells are the four
# counts at HEAD; the fifth candidate, `62 notes`, is a verb and is settled
# by binding (wrap_bound, the FOLD cells below) rather than by these cells.
# The four refusal cells are the structural half, synthesized rather than
# copied because at HEAD no welded or hyphenated string survives to the
# candidate stage for a text to be lifted from.
def claims_lines(text):
blk = [(i + 1, raw) for i, raw in enumerate(text.split("\n"))]
return [(u, v, w) for _ln, v, u, _t, _p, w in claims_in(blk)]
check("wrap: a number ending a line and its unit opening the next is a claim",
claims_lines("Twenty-five of the 69\nfilenames are not the form"),
[("filenames", 69, True)])
check("wrap: a thousands separator survives the join",
claims_lines("594 BLOB_OID / 1,062\nCONTENT_SHA256 and the rest"),
[("BLOB_OID", 594, False), ("CONTENT_SHA256", 1062, True)])
check("wrap: a backtick before the number and a closing one after the unit",
claims_lines("a selftest that prints `18\ncells` -- the defect class"),
[("cells", 18, True)])
check("wrap: an adjective is a unit when it is the count's own word",
claims_lines("observed keys at 30c8120, 24\nspendable, the other 398"),
[("spendable", 24, True)])
check("wrap: a claim that does not wrap is not tagged",
claims_lines("a selftest that prints 18 cells"), [("cells", 18, False)])
check("wrap: the same text wrapped and unwrapped is two claims, not three",
claims_lines("prints 18 cells\nand 18\ncells"),
[("cells", 18, False), ("cells", 18, True)])
# the four refusals, one cell each -- the structural half no candidate at
# HEAD exercises, because the reader refuses these strings before the join
check("wrap: a date's tail is welded to its hyphens, not a count",
claims_lines("--today 2026-10-18\npython3 tools/x.py"), [])
check("wrap: `UTF-8` ends a line and is still not a count",
claims_lines("a malformed name, non-UTF-8\nbytes are refused"), [])
check("wrap: a slash-separated pair (`0/1`) is not a count",
claims_lines("exit 0/1\npython3 tools/x.py"), [])
check("wrap: a hyphen behind the joined unit is a compound adjective",
claims_lines("the queue read 65 same-party / 9\nother-party. Neither"),
[])
# THE FOLD, as a unit rule: keys_for folds a plural (it must -- a one-line
# claim and a printed label are both census voice), wrap_bound does not.
check("wrap: a plural verb is bound by no fold -- `notes` is not `note`",
wrap_bound("notes", [({"note": {7}}, set(), "")]), False)
check("wrap: and a unit printed under its own spelling IS bound",
wrap_bound("notes", [({"notes": {62}}, set(), "")]), True)
check("wrap: an uppercase label binds to the lowercase printed key",
wrap_bound("CONTENT_SHA256",
[({"content_sha256": {1062}}, set(), "")]), True)
# the digest rule, one signal per cell, and the negative control between
# them -- the text of each is the text of the defect, measured at HEAD
check("revs: `sha256 <hex>` is a digest, not the block's revision",
block_revs("queue 1,170,483 B / sha256 `c3b4a290dc64bd24`"), [])
check("revs: an ellipsis-truncated hex run is a digest, not a revision",
block_revs("at `28ca8e5` (1,193,681 B / `c3b4a290dc64bd24…`)"),
["28ca8e5"])
check("revs: an ordinarily cited revision is still the block's revision",
block_revs("Run 37 read (`29a4872`, queue 1,170,483 B) and `28ca8e5`"),
["28ca8e5", "29a4872"])
check("oracle: 471 matches", oracle(printed, clone, "UNRESOLVED", 471),
("MATCHES", ""))
check("oracle: 441 is stale", oracle(printed, clone, "UNRESOLVED", 441),
("STALE", "printed 471"))
check("oracle: ambiguous unit makes no claim",
oracle(printed, clone, "citations", 1612),
("AMBIGUOUS", "printed: 410, 1612"))
check("oracle: clone-local is not a defect",
oracle(printed, clone, "blobs", 1234), ("CLONE_LOCAL", "printed 871"))
check("oracle: unprinted unit", oracle(printed, clone, "wombats", 3),
("NOT_PRINTED", ""))
# ---- leg 2 (--selftest) and the leg rules. The first check is the defect
# the leg was added for: a docstring cell count the run leg cannot see.
s2, s2c = parse_printed(FAKE_SELFTEST)
check("selftest leg: the cell count is a printed unit", sorted(s2.get("cells", {})), [18])
legs = [(printed, clone, "")] + [(s2, s2c, "selftest")]
check("adjudicate: a cell count leg 1 cannot see is STALE, not NOT_PRINTED",
adjudicate(legs, "cells", 13, []), ("STALE", "printed 18 [selftest]"))
check("adjudicate: the corrected cell count matches",
adjudicate(legs, "cells", 18, []), ("MATCHES", ""))
check("adjudicate: leg 1 wins when both legs print the unit",
adjudicate([(s2, s2c, ""), ({**s2, "cells": {99: [1]}}, s2c, "selftest")],
"cells", 99, []), ("STALE", "printed 18"))
check("adjudicate: a no-rev leg does not verify a block that names a revision",
adjudicate([(s2, s2c, "no-rev")], "cells", 18, ["f5403b2"]),
("REV_UNCHECKED", "block names f5403b2; the tool takes no --rev, so this "
"leg measured the working tree"))
check("adjudicate: the selftest leg is not a verifier at a named revision",
adjudicate([(s2, s2c, "selftest")], "cells", 18, ["f5403b2"]),
("REV_UNCHECKED", "block names f5403b2; the selftest leg prints this "
"tool's own fixtures, not a census at that revision"))
check("adjudicate: a no-rev leg IS an oracle for a block naming no revision",
adjudicate([(s2, s2c, "no-rev")], "cells", 13, []),
("STALE", "printed 18 [no-rev]"))
check("adjudicate: no leg prints the unit -> NOT_PRINTED",
adjudicate([(printed, clone, "")], "wombats", 3, []), ("NOT_PRINTED", ""))
# ---- the recipe contract, exercised rather than asserted: run_tool takes a
# recipe's (argv, tag) and the tag must survive to the leg, because the
# tag is what decides whether a claim may be adjudicated at all. A stub
# stands in for a tool so this stays hermetic and instant; the real
# owners recipe costs a 107 s tree scan and is exercised by the run.
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
stub = os.path.join(td, "stub.py")
with open(stub, "w") as fh:
fh.write("print('cells: 7')\n")
for want_tag in ("no-rev", ""):
RECIPES["stub.py"] = (lambda t: lambda repo, rev, tmp: (["--probe"], t))(want_tag)
try:
out, rc, tag = run_tool(td, stub, "HEAD", td)
finally:
RECIPES.pop("stub.py", None)
check(f"recipe: run_tool carries a recipe's tag {want_tag!r} to the leg",
(rc, tag, "cells: 7" in out), (0, want_tag, True))
def _boom(repo_, rev_, tmp_):
raise RuntimeError("boom")
RECIPES["stub.py"] = _boom
raised = ""
try:
try:
run_tool(td, stub, "HEAD", td)
except RuntimeError as exc:
raised = str(exc)
finally:
RECIPES.pop("stub.py", None)
check("recipe: an unmaterialisable revision raises, never an empty census",
raised, "boom")
# A tool that rejects `--repo`/`--rev` but has a perfectly good default
# census. The retry must drop exactly the tokens argparse named: the
# first cut re-ran it as `<script> --repo R`, which was rejected again,
# and a working tool was reported RUN_FAILED. Both halves are pinned --
# the retry happens, and it happens only for an argparse rejection.
apstub = os.path.join(td, "stub_argparse.py")
with open(apstub, "w") as fh:
fh.write("import argparse\n"
"ap = argparse.ArgumentParser()\n"
'ap.add_argument("--selftest", action="store_true")\n'
"ap.parse_args()\n"
"print('cells: 7')\n")
out, rc, tag = run_tool(td, apstub, "HEAD", td)
check("run_tool: argparse rejection retried with the rejected flags dropped",
(rc, tag, "cells: 7" in out), (0, "no-rev", True))
rcstub = os.path.join(td, "stub_rc2.py")
with open(rcstub, "w") as fh:
fh.write("import sys\n"
"sys.stderr.write('boom\\n')\n"
"sys.exit(2)\n")
out, rc, tag = run_tool(td, rcstub, "HEAD", td)
check("run_tool: a non-argparse failure is not re-tagged no-rev",
(rc, tag, "rc=2" in out), (2, "", True))
# ---- a failed run leg does not silence its block. This path lives in
# check_tool and needs a subprocess, so it is exercised end to end with
# a stub tool rather than asserted: the stub rejects every census
# invocation with an argparse diagnosis and answers only `--selftest`.
with tempfile.TemporaryDirectory(prefix="doccensus-selftest-") as td:
stub = os.path.join(td, "stub_failing_run.py")
with open(stub, "w") as fh:
fh.write("'''stub: a run leg that fails, a selftest that answers.\n"
"\n7 cells\n'''\n"
"import sys\n"
'if "--selftest" in sys.argv:\n'
' print("selftest: 7 cells, 7 passed, 0 failed")\n'
"else:\n"
' sys.stderr.write("usage: stub_failing_run.py [-h] [--selftest]\\n")\n'
" sys.exit(2)\n")
rep = check_tool(td, stub, "stub_failing_run.py", {}, run=True, tmpdir=td)
check("check_tool: a failed run leg is reported, and leg 2 still adjudicates",
([f["verdict"] for f in rep["failures"]],
[(v["unit"], v["claim"], v["verdict"]) for v in rep["verdicts"]]),
(["RUN_FAILED"], [("cells", 7, "MATCHES")]))
# ---- the banner's arithmetic, which is what a reader actually reads: a
# RUN_FAILED leg and a STALE claim are counted in two columns, so four
# failed runs cannot read as four bad claims.
synthetic = {"tool": "t", "splits": [], "unchecked": {}, "selftest_leg": True,
"failures": [{"verdict": "RUN_FAILED"}],
"verdicts": [{"verdict": "MATCHES"}, {"verdict": "STALE"}]}
check("counts: a leg failure is not a claim defect", counts(synthetic)[:4],
(0, 1, 1, 2))
check("counts: an absent leg 2 is noted",
counts({**synthetic, "selftest_leg": False})[4],
["no --selftest: leg 2 absent"])
# --tools: a filter that matches nothing must not read as a clean scan.
check("tools: a bare basename is reported missing, not scanned as zero",
resolve_tools(["checkin_check.py"], {"tools/checkin_check.py"}),
(["checkin_check.py"], ["checkin_check.py"]))
check("tools: no filter means every tracked tool and nothing missing",
resolve_tools(None, {"tools/b.py", "tools/a.py", "README.md"}),
(["tools/a.py", "tools/b.py"], []))
# ---- the gate. Before `--fail` this tool exited 0 while printing STALE
# rows -- the two historical STALEs at 4ce1aed require the
# pre-tightening reader, `ac0e211^`, so they are cited with that build
# and not as current behaviour (see FAILING ON THE CLASS IT CATCHES) --
# so the one class it exists to catch could not fail anything. The
# first five cells pin the decision; the last four pin the PROCESS exit
# code, because a helper returning 1 proves nothing about a gate.
stale_rep = {"tool": "t", "splits": [], "unchecked": {}, "failures": [],
"verdicts": [{"verdict": "STALE"}], "selftest_leg": True}
split_rep = {**stale_rep, "splits": [{"unit": "UNRESOLVED", "rows": []}],
"verdicts": [{"verdict": "MATCHES"}]}
# a leg failure alone: reported, never spent as a defect
clean_rep = {**stale_rep, "failures": [{"verdict": "RUN_FAILED"}],
"verdicts": [{"verdict": "MATCHES"},
{"verdict": "NOT_PRINTED"},
{"verdict": "AMBIGUOUS"},
{"verdict": "REV_UNCHECKED"}]}
check("gate: off is rc 0 even with a STALE claim", gate_rc([stale_rep], False), 0)
check("gate: on and a STALE claim -> rc 1", gate_rc([stale_rep], True), 1)
check("gate: on and an INTERNAL SPLIT alone -> rc 1", gate_rc([split_rep], True), 1)
check("gate: on and a leg failure alone -> rc 0 (reported, not spent)",
gate_rc([clean_rep], True), 0)
check("gate: the two columns sum across tools", gate_faults([stale_rep, split_rep]),
(1, 1))
# ---- the mode symmetry (reply 22, 2026-10-18). A split is graded only when
# a leg verified its unit; without a run there is no oracle, so the
# same docstring that fails a full run must NOT fail `--no-run`. Both
# halves are pinned: the advisory side, and the teeth that survive.
check("mode: --no-run can only label a split, never verify it",
unit_scope("UNRESOLVED", [], True), "LABEL_ONLY")
check("mode: a unit no leg prints is not a census unit",
unit_scope("WOMBATS", [({"widgets": {7: [1]}}, {}, "")], False),
"NOT_A_CENSUS_UNIT")
with tempfile.TemporaryDirectory(prefix="doccensus-mode-") as td:
split_stub = os.path.join(td, "stub_split.py")
with open(split_stub, "w") as fh:
fh.write('"""A stub whose docstring splits one unit.\n'
"\n471 UNRESOLVED at the head.\n" + "pad " * 80 + "\n"
'441 UNRESOLVED at the tail.\n"""\n'
"print('UNRESOLVED: 471')\n")
rep_norun = check_tool(td, split_stub, "stub_split.py", {}, run=False,
tmpdir=td)
rep_run = check_tool(td, split_stub, "stub_split.py", {}, run=True,
tmpdir=td)
check("mode: --no-run prints the split, graded as none",
([s["unit"] for s in rep_norun["splits"]],
[(s["unit"], s["how"]) for s in rep_norun["advisory"]]),
([], [("UNRESOLVED", "LABEL_ONLY")]))
check("mode: the same split is CHECKABLE once a leg prints the unit",
[(s["unit"], s["how"]) for s in rep_run["splits"]],
[("UNRESOLVED", "CHECKABLE")])
check("mode: --fail cannot fail on a split it could not verify",
gate_rc([rep_norun], True), 0)
check("mode: and still fails on the verified one", gate_rc([rep_run], True), 1)
# a cached vocabulary grades unit-hood without a run -- and only when it
# names the unit, which is the whole difference between a proof and a
# guess (reply 22, option 2)
check("vocab: a cached census unit is recognised",
vocab_has("UNRESOLVED", {"unresolved", "citations"}), True)
check("vocab: a unit the vocabulary does not name is not",
vocab_has("WOMBATS", {"unresolved"}), False)
rep_cached = check_tool(td, split_stub, "stub_split.py", {}, run=False,
tmpdir=td, vocab={"unresolved"}, vocab_rev="abc1234")
check("vocab: the same runless split grades as CACHE_CHECKABLE",
[(s["unit"], s["how"], s["oracle_rev"]) for s in rep_cached["splits"]],
[("UNRESOLVED", "CACHE_CHECKABLE", "abc1234")])
check("vocab: and the gate spends it", gate_rc([rep_cached], True), 1)
rep_other = check_tool(td, split_stub, "stub_split.py", {}, run=False,
tmpdir=td, vocab={"wombats"}, vocab_rev="abc1234")
check("vocab: a vocabulary naming another unit leaves it advisory",
([s["unit"] for s in rep_other["splits"]], gate_rc([rep_other], True)),
([], 0))
with tempfile.TemporaryDirectory(prefix="doccensus-gate-") as td:
os.makedirs(os.path.join(td, "tools"))
for name, claim in (("stub_stale.py", 99), ("stub_ok.py", 7)):
with open(os.path.join(td, "tools", name), "w") as fh:
fh.write(f'"""A stub whose docstring says {claim} widgets."""\n'
"print('widgets: 7')\n")
with open(os.path.join(td, "tools", "stub_split.py"), "w") as fh:
fh.write('"""A stub whose docstring splits one unit.\n'
"\n471 UNRESOLVED at the head.\n" + "pad " * 80 + "\n"
'441 UNRESOLVED at the tail.\n"""\n'
"print('UNRESOLVED: 471')\n")
# THE WRAPPED PAIR, three stubs that differ only in whether a leg can
# bind the unit: agreeing, disagreeing, and a wrap that is prose.
with open(os.path.join(td, "tools", "stub_wrap_ok.py"), "w") as fh:
fh.write('"""A stub whose claim wraps onto the next line.\n'
"\nWe measured 7\n"
'widgets at HEAD.\n"""\n'
"print('widgets: 7')\n")
with open(os.path.join(td, "tools", "stub_wrap_stale.py"), "w") as fh:
fh.write('"""A stub whose wrapped claim disagrees with its census.\n'
"\nWe measured 9\n"
'widgets at HEAD.\n"""\n'
"print('widgets: 7')\n")
with open(os.path.join(td, "tools", "stub_wrap_unbound.py"), "w") as fh:
fh.write('"""A stub whose wrap is prose, not a census.\n'
"\nIt exited 0\n"
'without a census.\n"""\n'
"print('widgets: 7')\n")
# THE FOLD, end to end -- the false fault measured on the real archive at
# HEAD (queue_claim_check.py:119, `reply 62 / notes it`). `notes` folds to
# the printed key `note`, which the loose printed reader picked out of a
# PASS line's prose, so a plural fold would spend this sentence and grade
# it STALE. The pair must land as unbound instead.
with open(os.path.join(td, "tools", "stub_wrap_fold.py"), "w") as fh:
fh.write('"""A stub whose wrap is a verb that folds onto a key.\n'
"\nThe queue's own reply 62\n"
'notes it).\n"""\n'
"print('note: 7')\n")
# ...and the boundary: the SAME text on ONE line is still a claim, still
# folded, still graded -- the narrowing is the wrap's price, not a new
# rule for the adjacency reader.
with open(os.path.join(td, "tools", "stub_note_adjacent.py"), "w") as fh:
fh.write('"""A stub whose one-line claim folds onto a key.\n'
"\nThe queue's own reply 62 notes it).\n\"\"\"\n"
"print('note: 7')\n")
genv = dict(os.environ, GIT_AUTHOR_NAME="t", GIT_AUTHOR_EMAIL="t@t",
GIT_COMMITTER_NAME="t", GIT_COMMITTER_EMAIL="t@t")
for sub in (["init", "-q"], ["add", "-A"], ["commit", "-qm", "fixture"]):
subprocess.run(["git", "-C", td] + sub, capture_output=True, env=genv)
def run_gate(*extra):
return subprocess.run([sys.executable, os.path.abspath(__file__),
"--repo", td, "--rev", "HEAD"] + list(extra),
capture_output=True, env=genv).returncode
check("gate e2e: a STALE claim fails the process (rc 1)", run_gate("--fail"), 1)
check("gate e2e: the same tree without --fail is rc 0 (the old silence)",
run_gate(), 0)
check("gate e2e: a matching claim passes the gate (rc 0)",
run_gate("--tools", "tools/stub_ok.py", "--fail"), 0)
check("gate e2e: a --tools filter matching nothing is rc 2, not 1",
run_gate("--tools", "tools/nope.py", "--fail"), 2)
# the mode pair, as PROCESS exit codes: same tree, same split, the two
# modes must not disagree about what they verified
check("gate e2e: --fail --no-run on a split is rc 0 (no oracle, advisory)",
run_gate("--tools", "tools/stub_split.py", "--no-run", "--fail"), 0)
check("gate e2e: --fail with the run grades that same split (rc 1)",
run_gate("--tools", "tools/stub_split.py", "--fail"), 1)
# THE WRAPPED PAIR, END TO END. Three stubs whose only difference is
# whether a leg prints the unit the wrap names, so these cells grade the
# SPENDING rule and not just the reading rule above.
check("gate e2e: a wrapped claim a leg binds, agreeing, is rc 0",
run_gate("--tools", "tools/stub_wrap_ok.py", "--fail"), 0)
check("gate e2e: and its wrapped VALUE is graded -- disagreeing is rc 1",
run_gate("--tools", "tools/stub_wrap_stale.py", "--fail"), 1)
check("gate e2e: a wrapped pair no leg binds is not a claim (rc 0)",
run_gate("--tools", "tools/stub_wrap_unbound.py", "--fail"), 0)
check("gate e2e: --no-run spends no wrap and fails on none (rc 0)",
run_gate("--tools", "tools/stub_wrap_stale.py", "--no-run",
"--fail"), 0)
# THE FOLD: a plural verb must not be bound to a singular printed key by
# a wrap, and the same text on one line must still be graded.
check("gate e2e: a wrapped verb is NOT bound by a plural fold (rc 0)",
run_gate("--tools", "tools/stub_wrap_fold.py", "--fail"), 0)
check("gate e2e: the same words on one line ARE folded and graded (rc 1)",
run_gate("--tools", "tools/stub_note_adjacent.py", "--fail"), 1)
# the process exit code through the vocabulary, both directions, plus
# the two ways a vocabulary can fail to be usable at all
vpath = os.path.join(td, "vocab-names-unit.json")
with open(vpath, "w") as fh:
json.dump({"rev": "abc1234",
"tools": {"tools/stub_split.py": ["unresolved"]}}, fh)
check("gate e2e: --no-run --vocab grades a split the vocabulary names (rc 1)",
run_gate("--tools", "tools/stub_split.py", "--no-run",
"--vocab", vpath, "--fail"), 1)
vother = os.path.join(td, "vocab-other-unit.json")
with open(vother, "w") as fh:
json.dump({"rev": "abc1234",
"tools": {"tools/stub_split.py": ["wombats"]}}, fh)
check("gate e2e: a vocabulary that does not name the unit leaves rc 0",
run_gate("--tools", "tools/stub_split.py", "--no-run",
"--vocab", vother, "--fail"), 0)
check("gate e2e: an unreadable --vocab is rc 2, not 0 and not 1",
run_gate("--tools", "tools/stub_split.py", "--no-run",
"--vocab", os.path.join(td, "nope.json"), "--fail"), 2)
check("gate e2e: --save-vocab without a run is rc 2",
run_gate("--tools", "tools/stub_split.py", "--no-run",
"--save-vocab", os.path.join(td, "v0.json"), "--fail"), 2)
# the round trip a gate runner would actually run: a full run writes the
# vocabulary it observed, the fast mode reads it back and still fails
vrt = os.path.join(td, "vocab-roundtrip.json")
check("gate e2e: a full run writes the vocabulary it observed (rc 1)",
run_gate("--tools", "tools/stub_split.py", "--save-vocab", vrt,
"--fail"), 1)
with open(vrt) as fh:
check("gate e2e: the written vocabulary names the unit the leg printed",
sorted(json.load(fh)["tools"].get("tools/stub_split.py", [])),
["unresolved"])
check("gate e2e: the round trip fails the fast mode on the same split",
run_gate("--tools", "tools/stub_split.py", "--no-run",
"--vocab", vrt, "--fail"), 1)
# ---- the saved vocabulary is SCOPED to what a claim can spend, and the
# fast mode reports the splits it cannot grade (2026-10-18). The
# first three cells pin the matcher identity that makes the scoping
# safe; the last six pin the process, because a helper returning a
# set proves nothing about the file a runner reads.
check("vocab scope: a claimed unit expands the way keys_for reads it",
sorted(spendable_keys(["citations"])), ["citation", "citations"])
check("vocab scope: every key it admits is one vocab_has can spend",
all(vocab_has("citations", {k})
for k in spendable_keys(["citations"])), True)
check("vocab scope: a digest no docstring claims is not spendable",
"abcdef12" in spendable_keys(["UNRESOLVED", "citations"]), False)
with open(os.path.join(td, "tools", "stub_vocab.py"), "w") as fh:
fh.write('"""A stub that prints its census and a digest.\n\n'
'It found 7 widgets.\n"""\n'
"print('widgets: 7')\nprint('abcdef12: 3')\n")
with open(os.path.join(td, "tools", "stub_lower.py"), "w") as fh:
fh.write('"""A stub whose docstring splits a lowercase word.\n\n'
"We did 5 above the fold.\n" + "pad " * 80 + "\n"
'Then 7 above the fold.\n"""\nprint(\'above: 3\')\n')
for sub in (["add", "-A"], ["commit", "-qm", "vocab scope"]):
subprocess.run(["git", "-C", td] + sub, capture_output=True, env=genv)
vscope = os.path.join(td, "vocab-scope.json")
check("gate e2e: --save-vocab on a stub printing a digest (rc 0)",
run_gate("--tools", "tools/stub_vocab.py", "--save-vocab", vscope,
"--fail"), 0)
with open(vscope) as fh:
check("gate e2e: the saved vocabulary holds the claimed unit only",
json.load(fh)["tools"].get("tools/stub_vocab.py"), ["widgets"])
vlow = os.path.join(td, "vocab-lower.json")
with open(vlow, "w") as fh:
json.dump({"rev": "abc1234",
"tools": {"tools/stub_lower.py": ["above"]}}, fh)
check("gate e2e: a claimed lowercase word IS spendable -- the over-reach",
run_gate("--tools", "tools/stub_lower.py", "--no-run",
"--vocab", vlow, "--fail"), 1)
def run_gate_out(*extra):
p = subprocess.run([sys.executable, os.path.abspath(__file__),
"--repo", td, "--rev", "HEAD"] + list(extra),
capture_output=True, env=genv)
return p.returncode, p.stdout.decode("utf-8", "replace")
rc_low, out_low = run_gate_out("--tools", "tools/stub_lower.py",
"--no-run", "--fail")
check("gate e2e: a runless lowercase split is rc 0 (nothing verified)",
rc_low, 0)
check("gate e2e: and it is PRINTED as advisory, not swallowed",
[l.strip() for l in out_low.splitlines()
if l.strip().startswith("ADVISORY above (no census label")],
["ADVISORY above (no census label and no cached vocabulary): "
"5@3, 7@5 -- reported, not graded"])
# A WRAP IS NEVER SILENT: the banner says how many pairs were read
# across a break and how many were spent, on both outcomes.
_rc, out_wb = run_gate_out("--tools", "tools/stub_wrap_ok.py")
check("gate e2e: the banner says the wrap was read and spent",
[l.strip() for l in out_wb.splitlines()
if l.strip().startswith("wrapped:")],
["wrapped: 1 pair(s) read across a line break, 1 spent, "
"0 not a claim (no leg prints the unit)"])
_rc, out_wu = run_gate_out("--tools", "tools/stub_wrap_unbound.py")
check("gate e2e: an unbound wrap is counted and named, not dropped",
[l.strip() for l in out_wu.splitlines()
if l.strip().startswith(("wrapped:", "WRAP_UNBOUND"))],
["wrapped: 1 pair(s) read across a line break, 0 spent, "
"1 not a claim (no leg prints the unit)",
"WRAP_UNBOUND tools/stub_wrap_unbound.py:3 0 without -- the "
"unit is printed by no leg"])
_rc, out_wf = run_gate_out("--tools", "tools/stub_wrap_fold.py")
check("gate e2e: a folded verb is named UNBOUND, not graded STALE",
[l.strip() for l in out_wf.splitlines()
if l.strip().startswith(("wrapped:", "WRAP_UNBOUND", "STALE"))],
["wrapped: 1 pair(s) read across a line break, 0 spent, "
"1 not a claim (no leg prints the unit)",
"WRAP_UNBOUND tools/stub_wrap_fold.py:3 62 notes -- the unit "
"is printed by no leg"])
print(f"selftest: {cells[0]} cells, {cells[0] - bad[0]} passed, {bad[0]} failed")
return 0 if ok else 1
# ------------------------------------------------------------------- driver
def run_tool(repo, script, rev, tmpdir, timeout=600):
"""(stdout, rc, tag) for the tool's own run at `rev`.
`tag` is "" when the leg stood on the revision it was asked for and "no-rev"
when it did not: either the tool rejected `--rev` and was re-run without it,
or a recipe in RECIPES declared that it could not materialise the revision.
Such a leg measures the tree it is pointed at, so a claim in a block that
names a revision is reported REV_UNCHECKED rather than adjudicated (see
check_tool). stdout is returned; stderr is only folded in when there is no
stdout at all, so an argparse diagnosis is visible as a failure instead of
parsed as a census.
"""
recipe = RECIPES.get(os.path.basename(script), None)
if recipe is None:
argv, tag = [sys.executable, script, "--repo", repo, "--rev", rev], ""
else:
args, tag = recipe(repo, rev, tmpdir)
argv = [sys.executable, script] + args
p = subprocess.run(argv, capture_output=True, timeout=timeout)
out = p.stdout.decode("utf-8", "replace")
err = p.stderr.decode("utf-8", "replace")
# A tool that takes neither `--repo` nor `--rev` answers with an argparse
# diagnosis naming the flags it rejected, and NOTHING else: measured
# 2026-10-16, `check_lint.py --repo R --rev HEAD` exits 2 with stdout empty
# and `unrecognized arguments: --repo R --rev HEAD`, and the first cut of
# this rule re-ran it as `<script> --repo R` -- still rejected, still rc 2,
# so a tool with a perfectly good default census was reported RUN_FAILED.
# The retry therefore drops the tokens argparse itself named, and only
# those; a tool whose values collide with a dropped flag is a stated
# limitation, not a silent one.
m = re.search(r"unrecognized arguments:\s*(.+)$", err, re.M)
if tag != "no-rev" and p.returncode != 0 and m:
drop = set(m.group(1).split())
argv = [x for x in argv if x not in drop]
p = subprocess.run(argv, capture_output=True, timeout=timeout)
out = p.stdout.decode("utf-8", "replace")
err = p.stderr.decode("utf-8", "replace")
tag = "no-rev"
if not out.strip():
out = "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
return out, p.returncode, tag
def run_selftest(script, timeout=300):
"""Leg 2: the tool's own `--selftest`, or None if it takes no such flag.
A cell count is printed here and nowhere else, which is the whole reason
this leg exists. `None` means the tool has no selftest to offer -- reported
as one leg, not as a failure.
"""
try:
p = subprocess.run([sys.executable, script, "--selftest"],
capture_output=True, timeout=timeout)
except subprocess.TimeoutExpired:
return "<--selftest timed out>"
out = p.stdout.decode("utf-8", "replace")
err = p.stderr.decode("utf-8", "replace")
if "unrecognized arguments" in err and "--selftest" in err:
return None
if not out.strip():
return "<no stdout; rc=%d> %s" % (p.returncode, err[:200].replace("\n", " "))
return out
def unit_scope(unit, legs, no_run):
"""Is a split in this unit checkable, and why not if it is not?
With the oracle legs in hand a unit is checkable when some leg prints it
with exactly ONE value: the docstring and the output then talk about one
number. A unit printed with 2+ values is SECTION-SPLIT BY THE TOOL'S OWN
DESIGN (`citations` prints once under EXPLICIT and once under BARE), so a
docstring that mirrors both is correct and is reported as such, not flagged.
A unit no leg prints is not a census unit here. Without a run, only
uppercase census labels are checked -- and a cached vocabulary (see
vocab_has) can widen that, which is why the cache decision is taken in
check_tool, not here: this function answers about the legs it was given.
"""
if no_run or not legs:
return ("LABEL_ONLY" if is_label(unit) else "UNCHECKED_NO_ORACLE")
saw_split = False
for printed, clone, _tag in legs:
vals = set()
for k in keys_for(unit, printed or {}):
vals |= set((printed or {})[k])
if not vals:
continue
if len(vals) > 1:
saw_split = True
continue
return "CHECKABLE"
if saw_split:
return "SECTION_SPLIT_BY_TOOL"
return "NOT_A_CENSUS_UNIT"
def wrap_bound(unit, legs):
"""True when some leg prints this unit under its OWN spelling.
This is keys_for WITHOUT the plural fold, and the narrowing is the whole
point. `keys_for` tries `unit.lower().rstrip("s")` because a one-line claim
and a printed label are both hand-written census voice (`18 claims` vs
`claim: 18`), so folding the plural costs nothing there. A WRAP IS NOT
HAND-WRITTEN CENSUS VOICE -- it is two lines this reader hypothesised into
one claim, and the word that opens the second line is a word in a sentence
far more often than it is a label: `the queue's own reply 62 / notes it`
folds `notes` onto a printed key `note`, and the tool printed that key only
because `parse_printed`'s loose reader took it out of the prose of its own
PASS line (`PASS v2 loose key: a Q7 continuation note is not an opening
record`). Measured 2026-10-20: that fold spent the pair and graded the
docstring STALE 62-vs-7 -- a false fault about a correct sentence.
So a wrap binds on the unit's own spelling or not at all; a pair that only a
fold would bind is printed as WRAP_UNBOUND and never graded, which is the
same landing place as a pair no leg touches.
"""
u = unit.lower()
return any(u in (printed or {}) for printed, _clone, _tag in legs)
def vocab_has(unit, vocab):
"""True when a cached census vocabulary names this unit word.
The vocabulary is a set of the unit KEYS a full run observed (`unresolved`,
`citations`, `acs_only` ...), and `keys_for` is the same matcher the oracle
uses -- so a cache hit means exactly "a leg printed this unit under this
key", no letter of the rule re-invented for the cache.
"""
return bool(keys_for(unit, {u: [0] for u in (vocab or ())}))
def spendable_keys(units):
"""The vocabulary keys a claim on one of `units` can actually spend.
`vocab_has` answers `keys_for(unit, ...)`, so the set of keys any claim in a
docstring can reach is fixed: exactly the four spellings `keys_for` tries,
lowercased, for the units that docstring claims. A saved vocabulary entry
outside that set can never be consulted -- it is a permission nobody can
use -- so `--save-vocab` writes the intersection of what the legs observed
with this set. Same matcher on both sides, no second rule invented.
"""
out = set()
for u in units or ():
for c in (u, u.lower(), u.lower().rstrip("s"), u.upper()):
out.add(c.lower())
return out
def check_tool(repo, script, name, cache, run=True, tmpdir=".", isolation=240,
vocab=None, vocab_rev=None):
text = open(script, encoding="utf-8", errors="replace").read()
span = docstring_span(script, text)
if span is None:
return None
lines = text.split("\n")
lo, hi = span
blk_claims = []
for first, blk in blocks(lines, lo, hi):
btext = "\n".join(raw for _, raw in blk)
revs = block_revs(btext)
cl = claims_in(blk)
if cl:
blk_claims.append((first, revs, cl, btext))
# ---- the legs first. Leg 1: the tool's own run, once per revision the
# docstring names. Leg 2: its --selftest, once. The tool under test is
# never this tool: running this scanner on itself spawns it, which
# spawns it again, without a floor.
is_self = os.path.basename(script) == os.path.basename(os.path.abspath(__file__))
printed_by_rev, verdicts, failures = {}, [], []
selftest_leg = None
if run and not is_self:
for first, revs, cl, _b in blk_claims:
for rev in (revs or ["HEAD"]):
key = (name, rev)
if key not in cache:
try:
out, rc, tag = run_tool(repo, script, rev, tmpdir)
cache[key] = (parse_printed(out), rc, out, tag)
except Exception as exc: # noqa: BLE001
cache[key] = (None, None, f"<run failed: {exc}>", "")
printed_by_rev.setdefault((first, rev), cache[key])
skey = (name, "<selftest>")
if skey not in cache:
try:
sout = run_selftest(script)
except Exception as exc: # noqa: BLE001
sout = f"<selftest failed: {exc}>"
cache[skey] = (parse_printed(sout) if sout else None,
0 if sout else None, sout or "", "selftest")
selftest_leg = cache[skey]
def legs_for(first, rev):
"""Priority-ordered [(printed, clone, tag)] for one block scope."""
out = []
entry = printed_by_rev.get((first, rev))
if entry and entry[0]:
out.append((entry[0][0], entry[0][1], entry[3]))
if selftest_leg and selftest_leg[0]:
out.append((selftest_leg[0][0], selftest_leg[0][1], "selftest"))
return out
# A WRAPPED PAIR IS SPENT ONLY WHEN A LEG CAN BIND ITS UNIT. Joining a line
# break is a reading decision and it is the only place in this tool where a
# claim is manufactured out of two lines instead of found inside one, so it
# gets the tool's own test for census-hood rather than a lexicon of English:
# a unit some leg prints IS a unit; a word that merely opens a sentence is
# not, and the difference is measured on the tool's output rather than
# asserted in a word list. Measured at HEAD, spent: `1,062 CONTENT_SHA256`
# (citation_hash_class.py:53). Measured at HEAD, NOT spent and named in the
# banner: `69 filenames` (checkin_check.py:14, no leg prints `filenames`),
# `18 cells`, `24 spendable` and `0 without` (doc_census_check.py self), and
# `62 notes` (queue_claim_check.py:119, `reply 62 / notes it` -- prose, and
# only a plural fold would have bound it; see wrap_bound). Nothing is dropped
# silently -- every unbound pair is counted and printed (the `wrapped:` line
# in the banner), because a candidate that vanishes is the defect this rule
# exists to close. With no run there is no leg, so no wrap is spent and every
# one is reported unbound: a mode that cannot see a census cannot bind a
# claim, which is the same contract unit_scope already applies to a split.
wrap_candidates, wrap_unbound = 0, []
if any(w for _f, _r, cl, _b in blk_claims for _ln, _v, _u, _t, _p, w in cl):
kept_blocks = []
for first, revs, cl, btext in blk_claims:
rows = []
for ln, val, unit, tok, pos, wrapped in cl:
if not wrapped:
rows.append((ln, val, unit, tok, pos, wrapped))
continue
wrap_candidates += 1
bound = any(wrap_bound(unit, legs_for(first, r))
for r in (revs or ["HEAD"]))
if bound:
rows.append((ln, val, unit, tok, pos, wrapped))
else:
wrap_unbound.append({"file": name, "line": ln, "unit": unit,
"claim": val, "token": tok})
if rows:
kept_blocks.append((first, revs, rows, btext))
blk_claims = kept_blocks
# 1. INTERNAL SPLIT: one unit, one revision scope, two values, far enough
# apart to be a restatement rather than a neighbouring sub-count.
splits, advisory, unchecked = [], [], collections.Counter()
by_unit = collections.defaultdict(list)
for first, revs, cl, _b in blk_claims:
scope = tuple(revs)
for ln, val, unit, _t, pos, _w in cl:
by_unit[(unit.upper() if unit.isupper() else unit.lower(), scope)].append(
(first, ln, val, pos))
for (unit, scope), items in sorted(by_unit.items()):
vals = {v for _, _, v, _ in items}
if len(vals) < 2:
continue
# a restatement is a pair far apart; an enumeration is neighbours
far = any(abs(b[3] - a[3]) >= isolation
for a, b in [(x, y) for x in items for y in items]
if a[1] != b[1] and a[2] != b[2])
if not far:
continue
rev_for_oracle = (items[0][0], scope[0] if len(scope) == 1 else "HEAD")
why = ("SELF_NOT_RUN" if is_self and run
else unit_scope(unit, legs_for(*rev_for_oracle), not run))
oracle_rev = None
# A CACHED VOCABULARY WIDENS WHAT A RUNLESS MODE MAY GRADE, and only in
# the one direction it can prove: a unit some leg printed at the
# vocabulary's rev is a census unit, and the disagreement here is
# internal to this docstring, so the leg's VALUE is never needed. It is
# consulted only when nothing ran -- with a run, a unit the live leg
# does not print is evidence AGAINST census-hood and the cache must not
# overrule it. A STALE claim never enters through here: `why` only
# decides splits.
if why in ("LABEL_ONLY", "UNCHECKED_NO_ORACLE") and not run \
and vocab_has(unit, vocab):
why, oracle_rev = "CACHE_CHECKABLE", vocab_rev
row = {"unit": unit, "revs": list(scope), "values": sorted(vals),
"how": why, "oracle_rev": oracle_rev,
"rows": [{"line": ln, "value": v, "block": b}
for b, ln, v, _ in items]}
# THE ONLY DOOR INTO `splits`. A split is graded only when a leg ran and
# printed its unit with exactly one value (CHECKABLE), or when a cached
# vocabulary proves the unit is a census unit and no run happened
# (CACHE_CHECKABLE): then the docstring and the census are talking about
# one number and a disagreement is a defect the gate can verify. Every
# other reason -- no oracle at all (LABEL_ONLY, --no-run), a unit no leg
# prints (NOT_A_CENSUS_UNIT), a unit the tool itself prints per section
# (SECTION_SPLIT_BY_TOOL), this tool reading itself (SELF_NOT_RUN) -- is
# reported with its values and never reaches the gate. Decided 2026-10-18
# (reply 22) after `--fail --no-run` returned 1 on the 471/441 split at
# fca3f2e while the full run returned 0 for the same units it could not
# see: rc 1 must never mean "I could not check this".
if why not in ("CHECKABLE", "CACHE_CHECKABLE"):
unchecked[why] += 1
advisory.append(row)
continue
splits.append(row)
# 2. ORACLE: every claim compared against what the tool prints, leg 1 first.
if run:
for first, revs, cl, _b in blk_claims:
for rev in (revs or ["HEAD"]):
if is_self:
for ln, val, unit, tok, _pos, _w in cl:
verdicts.append({"file": name, "block": first, "line": ln,
"rev": rev, "unit": unit, "claim": val,
"token": tok, "verdict": "SELF_NOT_RUN",
"detail": "the tool under test is this tool"})
continue
pr, rc, out, _tag = printed_by_rev[(first, rev)]
printed, clone = pr if pr else (None, None)
legs = legs_for(first, rev)
# A leg with no census -- raised, or exited non-zero having
# printed nothing -- adjudicates nothing either: drop it and let
# whichever leg survives answer instead. Only when none survives
# is the block left unadjudicated.
no_census = pr is None or (rc not in (0, None)
and not any(printed.values()))
if pr is None:
failures.append({"file": name, "block": first, "rev": rev,
"unit": "-", "claim": None,
"verdict": "RUN_FAILED", "detail": str(out)[:90]})
elif no_census:
failures.append({"file": name, "block": first, "rev": rev,
"unit": "-", "claim": None,
"verdict": "RUN_FAILED",
"detail": "rc=%s, nothing printed: %s"
% (rc, str(out)[:80])})
if no_census:
legs = [l for l in legs if l[2] == "selftest"]
if not legs:
continue
for ln, val, unit, tok, _pos, _w in cl:
v, detail = adjudicate(legs, unit, val, revs)
verdicts.append({"file": name, "block": first, "line": ln,
"rev": rev, "unit": unit, "claim": val,
"token": tok, "verdict": v, "detail": detail})
return {"tool": name, "splits": splits, "advisory": advisory,
"unchecked": dict(unchecked),
"claim_units": sorted({(u.upper() if u.isupper() else u.lower())
for _f, _r, cl, _b in blk_claims
for _ln, _v, u, _t, _p, _w in cl}),
"failures": failures, "verdicts": verdicts,
"wrap_candidates": wrap_candidates, "wrap_unbound": wrap_unbound,
"self": is_self and run, "selftest_leg": (not run) or bool(selftest_leg and selftest_leg[0]),
"blocks_with_claims": len(blk_claims)}
def counts(rep):
"""(splits, claim_defects, leg_failures, claims, notes) -- the banner's rows.
A LEG FAILURE IS NOT A CLAIM DEFECT. RUN_FAILED says no census came back and
such a leg adjudicates nothing (see check_tool); STALE says a claim was read
against a census and disagreed. The first live run of the two-leg tool mixed
them in one `N STALE/failed` column, so four failed runs read as four bad
claims -- measured 2026-10-16 at HEAD, check_lint.py 2 and
queue_claim_check.py 2, none of them a claim disagreement. Two columns, so
the distinction survives the summary a reader actually reads.
"""
stale = [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
notes = []
if rep.get("self"):
notes.append("self: no leg run (SELF_NOT_RUN)")
elif not rep.get("selftest_leg"):
notes.append("no --selftest: leg 2 absent")
return (len(rep["splits"]), len(stale), len(rep["failures"]),
len(rep["verdicts"]), notes)
def gate_faults(reports):
"""(splits, stale) -- the two things `--fail` refuses to let pass.
The class this tool exists to catch is a docstring claim that disagrees
with the census printed beside it (STALE), and two claims of one unit that
disagree with each other (INTERNAL SPLIT). A LEG FAILURE IS NOT IN THIS
COUNT, for the reason counts() gives: a leg that produced no census
adjudicated no claim, so counting it would fail a run on a claim never
checked. rc 2, decided in main(), covers "could not grade" separately.
"""
splits = stale = 0
for rep in reports:
splits += len(rep.get("splits") or [])
stale += sum(1 for v in (rep.get("verdicts") or [])
if v.get("verdict") == "STALE")
return splits, stale
def gate_rc(reports, fail):
"""0 when the gate is off or nothing was graded faulty; 1 when it is on and
a defect claim was graded. Never 2: "could not grade" is main()'s call,
made before any report exists, and this function cannot see it."""
if not fail:
return 0
splits, stale = gate_faults(reports)
return 1 if (splits or stale) else 0
def resolve_tools(requested, tree):
"""(tools, missing). A requested name that is not tracked at the rev is
REPORTED, never silently dropped: `--tools checkin_check.py` (a bare
basename instead of a repo-relative path) scanned zero tools and exited 0,
which reads exactly like a clean pass. Measured 2026-10-18 at 8932d06."""
if not requested:
return sorted(p for p in tree
if p.startswith("tools/") and p.endswith(".py")), []
return list(requested), [t for t in requested if t not in tree]
def main():
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--repo", default=DEFAULT_REPO)
ap.add_argument("--rev", default="HEAD")
ap.add_argument("--tools", nargs="*")
ap.add_argument("--json")
ap.add_argument("--no-run", action="store_true",
help="label-only: no subprocess, no oracle, so every split "
"it prints is ADVISORY and --fail cannot fail on one -- "
"rc 1 would mean 'could not check this'")
ap.add_argument("--fail", action="store_true",
help="rc 1 if any tool reports an INTERNAL SPLIT a leg "
"VERIFIED (CHECKABLE) or a STALE claim; unverified "
"splits and leg failures are reported, never spent as "
"defects (rc 2 stays 'could not grade')")
ap.add_argument("--vocab", metavar="PATH",
help="a census vocabulary written by --save-vocab: under "
"--no-run, a split in a unit the vocabulary names is "
"GRADED (CACHE_CHECKABLE) instead of labelled. The "
"vocabulary proves unit-hood, never a value, so no "
"STALE claim can enter this way")
ap.add_argument("--save-vocab", metavar="PATH",
help="write the census vocabulary THIS RUN observed (unit "
"keys per tool) for a later --no-run --vocab; needs "
"the run, so it is refused with --no-run (rc 2)")
ap.add_argument("--selftest", action="store_true")
a = ap.parse_args()
if a.selftest:
return _fixture_checks()
if a.save_vocab and a.no_run:
print("cannot save a vocabulary: --save-vocab needs the census a run "
"prints, and --no-run runs nothing", file=sys.stderr)
return 2
vocab, vocab_rev = {}, None
if a.vocab:
try:
with open(a.vocab, encoding="utf-8") as fh:
vdoc = json.load(fh)
vocab = {t: set(u) for t, u in (vdoc.get("tools") or {}).items()}
vocab_rev = vdoc.get("rev")
except (OSError, ValueError, AttributeError, TypeError) as exc:
print(f"cannot grade: --vocab {a.vocab}: {exc}", file=sys.stderr)
return 2
if not a.no_run:
print("note: --vocab is read only by the runless mode; a live leg is "
"the oracle when the tool runs", file=sys.stderr)
store = git(a.repo, "ls-tree", "-r", "--name-only", a.rev)
tree = {p for p in store.decode("utf-8", "replace").split("\n") if p}
tools, missing = resolve_tools(a.tools, tree)
if not tools:
print(f"no tools found at {a.rev}", file=sys.stderr)
return 2
if missing:
print(f"cannot grade: --tools names nothing tracked at {a.rev}: "
f"{', '.join(missing)}", file=sys.stderr)
print(" names are repo-relative, e.g. --tools tools/checkin_check.py; "
"a filter that matches nothing is not a clean scan", file=sys.stderr)
return 2
cache, reports = {}, []
with tempfile.TemporaryDirectory(prefix="doccensus-") as tmpdir:
for t in tools:
src = subprocess.run(["git", "-C", a.repo, "show", f"{a.rev}:{t}"],
capture_output=True).stdout
local = os.path.join(tmpdir, os.path.basename(t))
with open(local, "wb") as fh:
fh.write(src)
try:
rep = check_tool(a.repo, local, t, cache, run=not a.no_run,
tmpdir=tmpdir, vocab=vocab.get(t),
vocab_rev=vocab_rev)
except subprocess.TimeoutExpired:
rep = {"tool": t, "splits": [], "blocks_with_claims": 0,
"verdicts": [{"file": t, "verdict": "TIMEOUT", "rev": "-",
"unit": "-", "claim": None, "block": 0}]}
if rep:
reports.append(rep)
print(f"tools scanned at {a.rev}: {len(reports)} "
f"({'internal-split only, label vocabulary' if a.no_run else 'docstring vs printed census'})")
if a.vocab:
print(f"vocabulary: {a.vocab} -- census at {vocab_rev}, "
f"{len(vocab)} tool(s), {sum(len(v) for v in vocab.values())} unit(s)"
+ ("" if a.no_run else " (not read: a live leg is the oracle)"))
tot = collections.Counter()
for rep in reports:
bad = rep["failures"] + [v for v in rep["verdicts"] if v["verdict"] == "STALE"]
nsplits, nstale, nfail, nclaims, notes = counts(rep)
print(f"\n{rep['tool']}: {nsplits} INTERNAL SPLIT, "
f"{nstale} STALE claim(s), {nfail} leg failure(s), "
f"{nclaims} claims checked"
+ (" [" + "; ".join(notes) + "]" if notes else ""))
if rep.get("wrap_candidates"):
print(f" wrapped: {rep['wrap_candidates']} pair(s) read across a "
f"line break, {rep['wrap_candidates'] - len(rep.get('wrap_unbound') or [])} "
f"spent, {len(rep.get('wrap_unbound') or [])} not a claim "
f"(no leg prints the unit)")
for u in rep.get("wrap_unbound") or []:
print(f" WRAP_UNBOUND {u['file']}:{u['line']} "
f"{u['token']} -- the unit is printed by no leg")
for s in rep["splits"]:
vals = ", ".join(f"{r['value']}@{r['line']}" for r in s["rows"])
scope = ",".join(s["revs"]) or "no rev named"
prov = (f", vocabulary at {s['oracle_rev']}"
if s.get("oracle_rev") else "")
print(f" SPLIT {s['unit']} [{scope}] ({s['how']}{prov}): {vals}")
for k, c in sorted(rep["unchecked"].items()):
# in the fast mode the UNCHECKED_NO_ORACLE rows are printed one by
# one above, so the count would be a second, worse copy of them
if a.no_run and k == "UNCHECKED_NO_ORACLE":
continue
print(f" (not flagged: {c} split(s) in units {k})")
# A --no-run split is the one advisory class a reader of the fast mode
# actually needs to SEE: it is what the old build failed the gate on.
# Printed with its values, labelled advisory, never counted above.
#
# BOTH runless classes print (2026-10-18). The LABEL_ONLY class (an
# uppercase census label with no oracle) was shown here; a lowercase
# word answered UNCHECKED_NO_ORACLE and was shown NOWHERE, so a runless
# split in a lowercase word read as no split at all. Not graded either
# way -- the fix is to the reporting, not the verdict.
for s in rep.get("advisory") or []:
if s["how"] == "LABEL_ONLY":
why = "LABEL_ONLY, no oracle"
elif s["how"] == "UNCHECKED_NO_ORACLE" and a.no_run:
why = "no census label and no cached vocabulary"
else:
continue
vals = ", ".join(f"{r['value']}@{r['line']}" for r in s["rows"])
print(f" ADVISORY {s['unit']} ({why}): {vals}"
f" -- reported, not graded")
for v in bad:
print(f" {v['verdict']:10s} {v['file']}:{v.get('line', '-')} "
f"{v['unit']} claimed {v['claim']} at {v['rev']} -- {v['detail']}")
verdict_counts = collections.Counter(v["verdict"] for v in rep["verdicts"])
for k in sorted(verdict_counts):
if k in ("REV_UNCHECKED", "SELF_NOT_RUN", "CLONE_LOCAL"):
print(f" ({verdict_counts[k]} {k}: reported, never counted as a defect)")
for k, c in verdict_counts.items():
tot[k] += c
print("\n--- verdict totals (unit: claims) ---")
for k in sorted(tot):
print(f" {k:12s} {tot[k]}")
if a.json:
try:
with open(a.json, "w") as fh:
json.dump({"rev": a.rev, "repo": a.repo, "reports": reports}, fh, indent=1)
except OSError as exc:
print("cannot write --json " + a.json + ": " + str(exc),
file=sys.stderr)
return 2
print(f"wrote {a.json}")
if a.save_vocab:
# The vocabulary is READ OFF THE RUN, not off the docstrings: only the
# keys `parse_printed` saw in a leg's stdout count as census units, so a
# tool whose leg failed contributes nothing and cannot hand the fast
# mode a vocabulary it never printed.
#
# AND IT IS SCOPED TO WHAT A CLAIM CAN SPEND (2026-10-18). The leg says
# the key exists; the tool's own docstring says whether any claim will
# ever ask about it. Digests (`abc1234`) and diffstat words (`added`,
# `gone`) come off a leg's stdout but no docstring claims them, so they
# were 398 of 422 permissions nobody could use. Dropping them changes no
# verdict -- see spendable_keys and the offline check in the artifact --
# and it makes the file a list a reader can audit line by line.
per_tool = {r["tool"]: r for r in reports if r}
units = collections.defaultdict(set)
seen = collections.defaultdict(set)
for (name, rev), entry in cache.items():
if rev == "<selftest>" or not entry[0]:
continue
seen[name].update(entry[0][0])
keep = spendable_keys((per_tool.get(name) or {}).get("claim_units"))
units[name].update(k for k in entry[0][0] if k in keep)
observed = sum(len(v) for v in seen.values())
kept = sum(len(v) for v in units.values())
try:
with open(a.save_vocab, "w", encoding="utf-8") as fh:
json.dump({"rev": a.rev, "repo": a.repo,
"tools": {k: sorted(v) for k, v in units.items()}},
fh, indent=1, sort_keys=True)
except OSError as exc:
print("cannot write --save-vocab " + a.save_vocab + ": " + str(exc),
file=sys.stderr)
return 2
print(f"wrote vocabulary {a.save_vocab}: {len(units)} tool(s), "
f"{kept} unit(s) at {a.rev} ({observed} key(s) observed, "
f"{observed - kept} unspendable by any claim)")
if a.fail:
nsplit, nstale = gate_faults(reports)
if nsplit or nstale:
print(f"\nFAIL: {nsplit} INTERNAL SPLIT + {nstale} STALE claim(s) "
f"at {a.rev}"
+ (" (graded on a cached census vocabulary: --no-run --vocab)"
if a.no_run else ""))
return 1
print(f"\nOK: 0 INTERNAL SPLIT, 0 STALE claim(s) at {a.rev}"
+ (" (no oracle in this mode: nothing was verified, so nothing is "
"graded -- ADVISORY rows above are labels, not faults"
+ ("; the vocabulary graded any split in a unit it names)"
if a.vocab else ")")
if a.no_run else ""))
return gate_rc(reports, a.fail)
if __name__ == "__main__":
sys.exit(main())
Reference in new issue View Git Blame Copy Permalink