From f5403b2510af7896bd5db01056701c39eba767e1 Mon Sep 17 00:00:00 2001 From: Timmy Date: Fri, 16 Oct 2026 06:34:08 +0000 Subject: [PATCH] tools (volt): the docstring-vs-printed-census checker, and the defect it was built to catch fixed tools/doc_census_check.py (548 L / 25,058 B / sha256 dbbbdc01bb19dbcbbe87141c90a80e0bff19c98557c80f91c81347c53c98f1c5) checks a scanner's module docstring against the census that scanner PRINTS, two independent ways: (1) INTERNAL SPLIT -- one unit, one revision scope, two values in one paragraph, caught with no subprocess at all; (2) ORACLE -- the tool is re-run at the revision its docstring names and every ' ' claim is compared with the value that run prints (MATCHES / STALE / CLONE_LOCAL / AMBIGUOUS / NOT_PRINTED). A unit the tool prints with two values is SECTION_SPLIT_BY_TOOL and no claim is made about it; a unit the tool never prints is NOT_A_CENSUS_UNIT and no claim is made about it either. Selftest 14 fixtures, 14 passed, rc 0; the only write path is --json. THE DEFECT IT CATCHES, AND THE FIX: the control plane returned it against this lane's own resolver (reply 04, d355395) -- tools/citation_resolve_check.py:82 stated 471 UNRESOLVED while :89 stated 441 UNRESOLVED, over a census the tool prints as 471. That is the first finding of the new tool on its first run (INTERNAL SPLIT UNRESOLVED [babdadc] 471@82 / 441@89; STALE UNRESOLVED claimed 441 at babdadc, printed 471). HERE IT IS FIXED, my file and my edit: 2 insertions / 2 deletions, both on the one line; byte count unchanged at 20,460 B because 441 and 471 are the same width; sha256 7d063b39e2e4f9e5691b720cb57895b29450ece77051a7db6a8584dfd7173237 -> 40314b34f9860c4c42623f669c1339ed1d7b0ffc5fe604d56d6e9112b893e3a6. The fix is docstring-only and changes no measurement: the tool's own selftest is 11 fixtures / 11 passed / rc 0 both ways, and the census at babdadc is byte-for-byte the same after the edit as before it (297 files; EXPLICIT 410 = 113 RESOLVES / 0 STALE_LINE / 297 EXTERNAL_PATH; BARE 1612 = 119 AS_NAMED / 716 AS_SELF / 51 AMBIGUOUS / 255 EXTERNAL_NAMED / 471 UNRESOLVED; 6,961 mentions; 101 distinct targets). MEASURED COVERAGE, stated narrow rather than rounded up: at d355395 the run read 92 claims across the three scanners -- 12 verified against the run they cite, 8 in units the tool prints with two values (no oracle), 71 in units the tool never prints. The oracle reported exactly one bad claim on that corpus, and it was the returned defect. A WIDER RULE WAS TRIED FIRST AND FALSIFIED BY ITS OWN FIRST RUN: tools/num_agreement_check.py, block-scope same-unit-two-values over the whole archive, flagged 447 restatements at first and 117 after an isolation rule (240 chars) -- too coarse to be read, and it is NOT landed; the file is kept out of the tree as evidence. Read-only in fact: only write path is --json, git status --porcelain clean after every run. No other file touched, no secret printed. Volt . --- tools/citation_resolve_check.py | 4 +- tools/doc_census_check.py | 548 ++++++++++++++++++++++++++++++++ 2 files changed, 550 insertions(+), 2 deletions(-) create mode 100644 tools/doc_census_check.py diff --git a/tools/citation_resolve_check.py b/tools/citation_resolve_check.py index 3f1c74b..554a5cf 100644 --- a/tools/citation_resolve_check.py +++ b/tools/citation_resolve_check.py @@ -86,8 +86,8 @@ THE CENSUS AT HEAD, WITH ITS UNITS (measured 2026-10-16, rev babdadc, 297 .md) number this tool exists to make checkable, and it is the honest form of the ledger's own "a token quoted without its class". mentions 6961 file mentions carrying no line -- these seed the scope. - Read every figure as a citation count, never as a defect count: 441 UNRESOLVED - is not 441 errors. The tool's own catch is the worked example below. + Read every figure as a citation count, never as a defect count: 471 UNRESOLVED + is not 471 errors. The tool's own catch is the worked example below. WORKED CATCH -- this tool's first finding is one of MY OWN artifacts checkins/2026-10-15-volt.md:9 reads "the three open items belong to other lanes diff --git a/tools/doc_census_check.py b/tools/doc_census_check.py new file mode 100644 index 0000000..cd61f90 --- /dev/null +++ b/tools/doc_census_check.py @@ -0,0 +1,548 @@ +#!/usr/bin/env python3 +"""doc_census_check.py -- does a tool's own docstring census match the census +that tool PRINTS? + +WHY THIS EXISTS + This archive's three scanners each carry a census table in their module + docstring: counts that a reader cites, that no command checks. The docstring + is prose; prose is not executed. On 2026-10-16 the control plane returned + exactly this defect against a sibling scanner, in its own words: + + "The scanner's own docstring carries a number that does not match the + census it prints -- the exact class the tool exists to catch." + + tools/citation_resolve_check.py said `471 UNRESOLVED` on one line and + `441 UNRESOLVED` nine lines later, over a census the tool prints as 471. + Nothing was checking the docstring against the output beside it. That is this + tool's whole job. + +WHAT IT MEASURES (read-only: this tool only writes --json) + Two independent verdicts per tool, and they do not share evidence. + + 1. INTERNAL SPLIT -- no oracle, no run. Within one docstring, every numeric + claim of the form ` ` is collected; two claims of ONE unit + that a single revision scope covers, carrying two different values, is a + split. This is the verdict that catches the 441/471 instance, and it is + why the tool is useful with no subprocess at all. + + 2. ORACLE -- the tool is RUN, at the revision its own docstring names, and + each claim is compared against the value that run prints for that unit. + MATCHES claim == printed value + STALE claim != printed value, and the printed unit is not + declared clone-local -- a defect claim + CLONE_LOCAL claim != printed value but the tool's own output marks + that unit clone-local ("blobs in THIS clone's local object + store"), so drift is the documented expectation, not a + defect. Reported, never counted as one. + AMBIGUOUS the tool prints that unit with 2+ values at that rev + (e.g. `citations` prints once per section), so there is no + single oracle value; both are printed and no claim is made + NOT_PRINTED the unit does not appear in this tool's output at all + + A claim is ` ` with the unit word ADJACENT, which is the + archive's own notation. A bare number is not a claim and is not checked. + +SCOPE, STATED RATHER THAN IMPLIED + Only module docstrings of .py tools are scanned: a docstring is where a census + table lives. Only revisions NAMED IN THE BLOCK are used as oracles -- a block + naming no revision is checked against the tool's default (HEAD) and tagged, so + a stale rev is never silently attributed. A tool that cannot be run at a + revision (queue_claim_check.py has no --rev; the queue is materialised from + git and handed to it by --queue) is driven through a per-tool recipe declared + below, with a temp file that is removed. + + WHAT THIS DOES NOT CHECK, and the measured size of the gap. A claim is only + adjudicated when the unit it names is a unit the tool PRINTS. Over the three + scanners at d355395 the run read ninety-two claims: twelve were verified + against the run they point at, eight were in units the tool prints with two + values (SECTION_SPLIT_BY_TOOL -- no oracle), and seventy-one were in units the + tool never prints at all. So the oracle's reach is narrow and its precision is + the reason: it reported exactly one bad claim on that corpus, and that one was + the returned defect. A wider rule was tried first and falsified by its own + first run -- num_agreement_check.py flagged a hundred and seventeen + restatements across the archive, too coarse to be read -- so the narrow rule + is the one that ships. Two claim forms are read (` ` and + `: `); a claim written in neither form is not seen, and that is a + stated gap, not a silent one. + + NO CENSUS OF THIS TOOL'S OWN VERDICTS APPEARS IN THIS DOCSTRING, deliberately: + this scanner scans itself, so a count of its own output written here would + have to agree with the output of the run that the same commit changes. That is + the limit citation_hash_class.py records in the same words -- a commit cannot + measure the file it introduces -- and here it is avoided rather than + discovered. + +USAGE + python3 doc_census_check.py [--repo DIR] [--rev HEAD] [--tools NAME...] + [--json OUT] [--no-run] [--selftest] +""" +import argparse +import ast +import collections +import json +import os +import re +import subprocess +import sys +import tempfile + +DEFAULT_REPO = "/root/troupe-standup" + +# A claim, in either of the two forms this archive writes: +# ` ` -- "471 UNRESOLVED", "1,612 citations" +# ` [: =] ` -- "ACS = 72", "overlap = 20" (markdown bold allowed) +# A bare number is not a claim. Deliberately not "any word": `471 cannot be +# resolved` is prose, and a checker that flags prose is a checker nobody reads. +CLAIM = re.compile( + r"(?[0-9][0-9,]*)(?P[^A-Za-z0-9\n]{0,2})" + r"(?P[A-Za-z_][A-Za-z0-9_]*|[A-Z][A-Z_]{2,})\b") +CLAIM_EQ = re.compile( + r"(?P[A-Za-z_][A-Za-z0-9_]*)\s*[:=]\s*\**\s*(?P[0-9][0-9,]*)") +REV_LONG = re.compile(r"\b[0-9a-f]{7,40}\b") + +# Function words and units this tool does not adjudicate. `at` in "2026-10-15 at +# rev 9188dd3" is not a census quantity; neither is `B` (bytes differ by +# revision by design, and "1,201,550 B" is a size, not a count). Stated so the +# exclusion is a rule rather than a habit. +STOPWORDS = { + "at", "the", "of", "in", "is", "are", "was", "and", "or", "to", "for", "on", + "by", "as", "it", "its", "that", "this", "these", "those", "with", "from", + "no", "not", "one", "two", "three", "all", "any", "be", "been", "has", + "have", "had", "his", "her", "our", "your", "their", "but", "if", "then", + "than", "so", "up", "out", "off", "over", "into", "run", "runs", "may", + "can", "same", "each", "both", "per", "via", "b", "kb", "mb", "gb", +} + +# How to run each tool at a revision. `queue_claim_check.py` takes no --rev: its +# subject is a FILE, so the file is materialised at the revision and handed to +# it. Any tool not listed here is run as `