volt: verify gains a seventh leg -- doc-commands, graded at WORKTREE

The gate had six legs and none of them read a docstring's own command line.
doc_command_check.py existed since 838750c but only as a tool: nothing on the
gate ran it, so a documented flag its parser never learned could land and
nobody read it. This is that leg.

  doc-commands   tools/doc_command_check.py --rev WORKTREE --fail
                 WORKTREE, not HEAD: the commit window is where a docstring is
                 edited. Measured at the worktree this lands in: 67 commands
                 over 19 oracle tools, 67 OK, rc 0.

doc_command_check.py learns --rev WORKTREE: tracked paths from `git ls-files`
(an untracked tools/x.py is not part of the commit this mode grades), bytes
from disk. Outside a work tree it is rc 2, not an empty clean run. Selftest
15 -> 18 cells: one document edited on disk but not committed is rc 0 at HEAD
and rc 1 at WORKTREE, and the flag it names is the edited one.

Three tools stopped defaulting to a fixed path on this host. verify.py,
selftest_sweep.py, write_probe.py, citation_resolve_check.py,
path_claim_check.py and doc_census_check.py now resolve DEFAULT_REPO from the
file's own location, so a bare run in a clone grades the clone.

Three selftests refuse unreadable input at rc 2 instead of reporting it as
failed content: tool_map (not a work tree), write_probe (no table), and
citation_resolve_check (no corpus). Each cell below them graded `repo`, so an
absent tree read as "your table is wrong".

Evidence, all measured on this tree before the commit:
  verify.py --selftest              55 cells, 55 passed   (docstring says 55)
  doc_command_check.py --selftest   18 cells, 18 passed   (baseline pinned 18)
  tool_map.py --repo . --rev WORKTREE --check docs/TOOLS.md   rc 0
  verify.py --repo .                7 legs, 6 pass, 1 fault -- selftest-sweep,
                                    tool_map's drift cell, which renders HEAD
                                    and so cannot pass inside a commit window
tools/selftest_baseline.tsv re-pins doc_command_check.py 15 -> 18 cells.
This commit is contained in:
volt committed 2026-10-22 22:05:55 +00:00
1 parent af32c34762
commit c3d70567a6
10 files changed
+192 -23

No files matched your search

+3 -2
View File
@@ -19,7 +19,7 @@ module docstring, flags from the tool's own `--help`, probe rows from
| `citation_hash_class.py` | resolve every hex token in a corpus to its hash family, and flag the tokens that carry no stated class. | `--repo` `--rev` `--corpus` `--json` `--window` `--min-len` `--only` `--context` `--siblings` `--selftest` | 1 / 0 / 1 | - | | `citation_hash_class.py` | resolve every hex token in a corpus to its hash family, and flag the tokens that carry no stated class. | `--repo` `--rev` `--corpus` `--json` `--window` `--min-len` `--only` `--context` `--siblings` `--selftest` | 1 / 0 / 1 | - |
| `citation_resolve_check.py` | does every `<file>:<line>` citation in this archive's prose actually resolve, and to which file? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--selftest` | 1 / 0 / 1 | - | | `citation_resolve_check.py` | does every `<file>:<line>` citation in this archive's prose actually resolve, and to which file? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--selftest` | 1 / 0 / 1 | - |
| `doc_census_check.py` | does a tool's own docstring census match the census that tool PRINTS? | `--repo` `--rev` `--tools` `--json` `--no-run` `--fail` `--vocab` `--save-vocab` `--selftest` | 1 / 0 / 1 | - | | `doc_census_check.py` | does a tool's own docstring census match the census that tool PRINTS? | `--repo` `--rev` `--tools` `--json` `--no-run` `--fail` `--vocab` `--save-vocab` `--selftest` | 1 / 0 / 1 | - |
| `doc_command_check.py` | a command a docstring tells you to run must be a command the tool still accepts. | `--repo` `--rev` `--tools` `--json` `--fail` `--selftest` | 1 / 0 / 1 | - | | `doc_command_check.py` | a command a docstring tells you to run must be a command the tool still accepts. | `--repo` `--rev` `--tools` `--json` `--fail` `--selftest` | 1 / 0 / 1 | `doc-commands` |
| `gate_blame.py` | which landing turned a gate leg red? | `--repo` `--leg` `--range` `--ref` `--queue` `--index` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - | | `gate_blame.py` | which landing turned a gate leg red? | `--repo` `--leg` `--range` `--ref` `--queue` `--index` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
| `landing_check.py` | is every file at this revision actually served by the forge? | `--repo` `--rev` `--branch` `--base-url` `--path` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - | | `landing_check.py` | is every file at this revision actually served by the forge? | `--repo` `--rev` `--branch` `--base-url` `--path` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
| `path_claim_check.py` | does every in-namespace path this archive names exist at this revision? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--fail-on-missing` `--baseline` `--selftest` | 1 / 0 / 1 | - | | `path_claim_check.py` | does every in-namespace path this archive names exist at this revision? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--fail-on-missing` `--baseline` `--selftest` | 1 / 0 / 1 | - |
@@ -30,7 +30,7 @@ module docstring, flags from the tool's own `--help`, probe rows from
| `selftest_sweep.py` | every instrument proves itself, in one command, and the number it advertises is the number it prints. | `--repo` `--tools-dir` `--timeout` `--json` `--baseline` `--no-baseline` `--write-baseline` `--quiet` `--verbose` `--selftest` | 1 / 0 / 1 | `selftest-sweep` | | `selftest_sweep.py` | every instrument proves itself, in one command, and the number it advertises is the number it prints. | `--repo` `--tools-dir` `--timeout` `--json` `--baseline` `--no-baseline` `--write-baseline` `--quiet` `--verbose` `--selftest` | 1 / 0 / 1 | `selftest-sweep` |
| `served_drift_check.py` | does the host actually serve the tree at this revision? | `--root` `--rev` `--base-url` `--prefix` `--exclude` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - | | `served_drift_check.py` | does the host actually serve the tree at this revision? | `--root` `--rev` `--base-url` `--prefix` `--exclude` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
| `tool_map.py` | the instruments get a map, and the map is generated from the tree. | `--repo` `--rev` `--write` `--check` `--json` `--selftest` | 1 / 0 / 1 | - | | `tool_map.py` | the instruments get a map, and the map is generated from the tree. | `--repo` `--rev` `--write` `--check` `--json` `--selftest` | 1 / 0 / 1 | - |
| `verify.py` | the HOME the index gate did not have: one command, six legs, one exit code, and a row per leg so a partial run cannot look like a clean one. | `--repo` `--queue` `--index` `--checkin-day` `--fast` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - | | `verify.py` | the HOME the index gate did not have: one command, seven legs, one exit code, and a row per leg so a partial run cannot look like a clean one. | `--repo` `--queue` `--index` `--checkin-day` `--fast` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
| `write_probe.py` | Measure the WRITE end of every tool in tools/: `--json PATH` at a target the host cannot write. | `--repo` `--only` `--timeout` `--json` `--selftest` `--verbose` | 1 / 0 / 1 | `write-axis` | | `write_probe.py` | Measure the WRITE end of every tool in tools/: `--json PATH` at a target the host cannot write. | `--repo` `--only` `--timeout` `--json` `--selftest` `--verbose` | 1 / 0 / 1 | `write-axis` |
## The gate ## The gate
@@ -45,6 +45,7 @@ dropped by `--fast`.
| `reply-headings` | `reply_check.py` | no | grade every reply's heading against its own filename | | `reply-headings` | `reply_check.py` | no | grade every reply's heading against its own filename |
| `index-current` | `queue_index.py` | no | render the index and byte-compare it with the document | | `index-current` | `queue_index.py` | no | render the index and byte-compare it with the document |
| `append-audit` | `queue_append_audit.py` | no | grade the newest queue-touching commit in its own tree | | `append-audit` | `queue_append_audit.py` | no | grade the newest queue-touching commit in its own tree |
| `doc-commands` | `doc_command_check.py` | no | every command a tool's docstring tells a reader to run: its flags must be options that tool's --help accepts |
| `selftest-sweep` | `selftest_sweep.py` | yes | every tool's --selftest and both read-refusal axes | | `selftest-sweep` | `selftest_sweep.py` | yes | every tool's --selftest and both read-refusal axes |
| `write-axis` | `write_probe.py` | no | a --json target this host cannot write: rc 2, one line naming it, never a traceback -- and the same rule for every caller-path write flag (--write, --save-vocab) | | `write-axis` | `write_probe.py` | no | a --json target this host cannot write: rc 2, one line naming it, never a traceback -- and the same rule for every caller-path write flag (--write, --save-vocab) |
+10 -1
View File
@@ -133,11 +133,12 @@ USAGE
import argparse import argparse
import collections import collections
import json import json
import os
import re import re
import subprocess import subprocess
import sys import sys
DEFAULT_REPO = "/root/troupe-standup" DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# The extensions this archive actually cites by line. Deliberately a closed list: # The extensions this archive actually cites by line. Deliberately a closed list:
# an open `\.[a-z]+` would swallow clock times and domain names. # an open `\.[a-z]+` would swallow clock times and domain names.
@@ -401,6 +402,14 @@ def _expect(here, text, store):
def selftest(repo): def selftest(repo):
# The fixtures grade real citations out of `repo`, so a repo that cannot be
# read turns every fixture into a mismatch -- eight failures that say "your
# content is wrong" when the truth is "I never saw your content". Refuse at
# rc 2, the code this tool's own main() uses for the same condition.
problem = precondition(repo, "HEAD")
if problem:
print(problem, file=sys.stderr)
return 2
store = Store(repo, "HEAD") store = Store(repo, "HEAD")
n = bad = 0 n = bad = 0
for here, text, want in FIXTURES: for here, text, want in FIXTURES:
+1 -1
View File
@@ -259,7 +259,7 @@ import subprocess
import sys import sys
import tempfile import tempfile
DEFAULT_REPO = "/root/troupe-standup" DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# A claim, in either of the two forms this archive writes: # A claim, in either of the two forms this archive writes:
# `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations" # `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations"
+90 -4
View File
@@ -50,6 +50,13 @@ revision: a command documented at `R` is checked against the tool at `R`. The
default is `HEAD`. A revision that does not materialise is rc 2, never an empty default is `HEAD`. A revision that does not materialise is rc 2, never an empty
clean run. clean run.
`--rev WORKTREE` is not a revision: it reads the files on disk and lists them
from the INDEX, so it grades the tree you are about to commit. `HEAD` cannot
see an uncommitted docstring, which is the window a lane writes in -- the same
blind spot `tool_map.py --rev WORKTREE` exists for. The two are measured
apart in the selftest: one document, edited on disk but not committed, is
rc 0 at `HEAD` and rc 1 at `WORKTREE`.
EXIT CODES (the tools/ contract) EXIT CODES (the tools/ contract)
0 every documented command was adjudicated and none was rejected 0 every documented command was adjudicated and none was rejected
1 at least one documented command is a fault (FLAG_NOT_ACCEPTED, TOOL_MISSING) 1 at least one documented command is a fault (FLAG_NOT_ACCEPTED, TOOL_MISSING)
@@ -59,10 +66,12 @@ EXIT CODES (the tools/ contract)
USAGE USAGE
python3 doc_command_check.py --repo . [--rev REV] [--tools a.py,b.py] python3 doc_command_check.py --repo . [--rev REV] [--tools a.py,b.py]
[--json OUT] [--fail] [--json OUT] [--fail]
python3 doc_command_check.py --selftest # 15 cells, oracle and verdicts python3 doc_command_check.py --selftest # 18 cells, oracle and verdicts
""" """
import argparse import argparse
import ast import ast
import contextlib
import io
import json import json
import os import os
import re import re
@@ -86,6 +95,11 @@ OPT_RE = re.compile(r"(?<![\w-])(--[A-Za-z][\w-]*|-[A-Za-z])(?![\w-])")
HELP_TIMEOUT = 60 HELP_TIMEOUT = 60
# Not a revision: read the files on disk, list them from the index. The same
# spelling `tool_map.py` uses, for the same reason -- a commit window is
# otherwise ungraded and the defect lands at HEAD with nobody having read it.
WORKTREE = "WORKTREE"
class CannotGrade(Exception): class CannotGrade(Exception):
"""Input this tool could not read or could not recognise: rc 2, never rc 1. """Input this tool could not read or could not recognise: rc 2, never rc 1.
@@ -97,9 +111,17 @@ class CannotGrade(Exception):
def tracked_tools(repo, rev): def tracked_tools(repo, rev):
"""(rc, stdout) of `git ls-tree` -> the tool paths in the tree at `rev`.""" """(rc, stdout) of `git ls-tree` -> the tool paths in the tree at `rev`."""
proc = subprocess.run( if rev == WORKTREE:
["git", "-C", repo, "ls-tree", "-r", "--name-only", rev, "--", "tools"], # The index, not the disk: an untracked `tools/x.py` is not part of the
capture_output=True, text=True) # commit this mode grades, and including it would grade a file that is
# about to be left behind. `git ls-files` fails outside a work tree,
# which is the rc 2 this mode owes a caller who ran it anywhere else.
proc = subprocess.run(["git", "-C", repo, "ls-files", "--", "tools"],
capture_output=True, text=True)
else:
proc = subprocess.run(
["git", "-C", repo, "ls-tree", "-r", "--name-only", rev, "--", "tools"],
capture_output=True, text=True)
if proc.returncode != 0: if proc.returncode != 0:
raise CannotGrade("cannot read repo: %s at %s: %s" raise CannotGrade("cannot read repo: %s at %s: %s"
% (repo, rev, (proc.stderr or "").strip()[:200])) % (repo, rev, (proc.stderr or "").strip()[:200]))
@@ -109,6 +131,18 @@ def tracked_tools(repo, rev):
def blob(repo, rev, path): def blob(repo, rev, path):
"""The bytes of `path` at `rev`, or None when the tree does not hold it.""" """The bytes of `path` at `rev`, or None when the tree does not hold it."""
if rev == WORKTREE:
try:
with open(os.path.join(repo, *path.split("/")),
"r", encoding="utf-8") as fh:
return fh.read()
except FileNotFoundError:
return None
except OSError as exc:
raise CannotGrade("cannot read %s at WORKTREE: %s" % (path, exc))
except UnicodeDecodeError as exc:
raise CannotGrade("cannot read %s at WORKTREE: not UTF-8: %s"
% (path, exc))
proc = subprocess.run(["git", "-C", repo, "show", "%s:%s" % (rev, path)], proc = subprocess.run(["git", "-C", repo, "show", "%s:%s" % (rev, path)],
capture_output=True) capture_output=True)
if proc.returncode != 0: if proc.returncode != 0:
@@ -294,6 +328,58 @@ def selftest():
harvested = set(OPT_RE.findall("usage: x [-h] [--fail] [--json JSON]\n")) harvested = set(OPT_RE.findall("usage: x [-h] [--fail] [--json JSON]\n"))
cell("bracketed usage options are harvested", harvested >= {"-h", "--fail", "--json"}) cell("bracketed usage options are harvested", harvested >= {"-h", "--fail", "--json"})
# WORKTREE: the tree you are about to commit, not the last commit. One
# document, edited on disk and NOT committed: HEAD reads the blob (clean),
# WORKTREE reads the disk (the documented flag its parser never learned).
with tempfile.TemporaryDirectory(prefix="doc-cmd-wt-") as td:
repo = os.path.join(td, "repo")
os.makedirs(os.path.join(repo, "tools"))
tool = os.path.join(repo, "tools", "oracle_tool.py")
def write_doc(extra=""):
with open(tool, "w", encoding="utf-8") as fh:
fh.write("#!/usr/bin/env python3\n"
'"""USAGE\n'
" python3 oracle_tool.py --present\n"
+ extra +
'"""\n'
"import argparse\n"
"ap = argparse.ArgumentParser()\n"
"ap.add_argument('--present')\n"
"ap.parse_args()\n")
def git(*args):
subprocess.run(["git", "-C", repo] + list(args),
capture_output=True, text=True)
write_doc()
git("init", "-q")
git("add", "-A")
git("-c", "user.name=selftest", "-c", "user.email=selftest@invalid",
"-c", "commit.gpgsign=false", "commit", "-q", "-m", "fixture")
write_doc(" python3 oracle_tool.py --gone-after-edit\n")
head_rep = census(repo, "HEAD")
wt_rep = census(repo, WORKTREE)
cell("WORKTREE reads the document on disk, HEAD reads the blob at HEAD",
[r["verdict"] for r in wt_rep["rows"]] == ["OK", "FLAG_NOT_ACCEPTED"]
and [r["verdict"] for r in head_rep["rows"]] == ["OK"],
([r["verdict"] for r in head_rep["rows"]],
[r["verdict"] for r in wt_rep["rows"]]))
cell("the uncommitted command's flag is the one named",
[r["missing"] for r in wt_rep["rows"] if r["missing"]] ==
[["--gone-after-edit"]],
[r["missing"] for r in wt_rep["rows"] if r["missing"]])
outsider = os.path.join(td, "not-a-repo")
os.makedirs(outsider)
buf = io.StringIO()
with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf):
rc_nogit = run(["--repo", outsider, "--rev", WORKTREE, "--fail"])
cell("WORKTREE outside a git work tree is rc 2, not a clean run",
rc_nogit == 2 and "cannot read repo" in buf.getvalue(),
(rc_nogit, buf.getvalue().strip()[:80]))
# verdicts # verdicts
rep = {"rev": "HEAD", "oracles": 1, "rows": [ rep = {"rev": "HEAD", "oracles": 1, "rows": [
{"doc": "tools/a.py", "target": "tools/a.py", "verdict": "OK", {"doc": "tools/a.py", "target": "tools/a.py", "verdict": "OK",
+1 -1
View File
@@ -126,7 +126,7 @@ import re
import subprocess import subprocess
import sys import sys
DEFAULT_REPO = "/root/troupe-standup" DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# The ledger this archive carries, REPO-relative: resolved against --repo, not # The ledger this archive carries, REPO-relative: resolved against --repo, not
# against the caller's cwd, so a bare run against a clone reads that clone's # against the caller's cwd, so a bare run against a clone reads that clone's
+1 -1
View File
@@ -17,7 +17,7 @@ checkin_check.py 36 cells
citation_hash_class.py 19 fixtures citation_hash_class.py 19 fixtures
citation_resolve_check.py 14 fixtures citation_resolve_check.py 14 fixtures
doc_census_check.py 104 cells doc_census_check.py 104 cells
doc_command_check.py 15 cells doc_command_check.py 18 cells
gate_blame.py 21 cells gate_blame.py 21 cells
landing_check.py 26 cells landing_check.py 26 cells
path_claim_check.py 27 cells path_claim_check.py 27 cells
1 # selftest_baseline.tsv -- the cell count each tool's selftest prints, pinned.
17 citation_hash_class.py
18 citation_resolve_check.py
19 doc_census_check.py
20 doc_command_check.py
21 gate_blame.py
22 landing_check.py
23 path_claim_check.py
+7 -1
View File
@@ -149,6 +149,12 @@ import subprocess
import sys import sys
import tempfile import tempfile
# The default target is the checkout this file lives in -- the same path in
# place, a different path in a copy. An absolute literal here would make a bare
# `--repo`-less run grade /root/troupe-standup from anywhere on the host, which
# is the wrong-tree defect `tool_map.py` documents at length.
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
UNITS = ("cells", "fixtures") UNITS = ("cells", "fixtures")
# `selftest: 18 cells, 18 passed, 0 failed` # `selftest: 18 cells, 18 passed, 0 failed`
@@ -1037,7 +1043,7 @@ def selftest():
def main(argv=None): def main(argv=None):
ap = argparse.ArgumentParser( ap = argparse.ArgumentParser(
description="Run every tool's --selftest and check its advertised cell count.") description="Run every tool's --selftest and check its advertised cell count.")
ap.add_argument("--repo", default="/root/troupe-standup") ap.add_argument("--repo", default=DEFAULT_REPO)
ap.add_argument("--tools-dir", default=None, ap.add_argument("--tools-dir", default=None,
help="sweep this directory instead of the repo's tracked tools/") help="sweep this directory instead of the repo's tracked tools/")
ap.add_argument("--timeout", type=int, default=120, metavar="S") ap.add_argument("--timeout", type=int, default=120, metavar="S")
+10
View File
@@ -394,6 +394,16 @@ def render(data):
def selftest(repo): def selftest(repo):
cells = [] cells = []
# The currency cell below grades `repo`; if `repo` is not a work tree that
# cell can only fail, which reads as "the document drifted" when the truth
# is "no tree was there". Refuse at rc 2, as main() already does for the
# same input.
rc, _, _ = git(repo, "rev-parse", "--show-toplevel")
if rc != 0:
print(f"tool_map: cannot read repo: {repo} is not a git work tree",
file=sys.stderr)
return 2
def cell(label, ok, detail=""): def cell(label, ok, detail=""):
cells.append({"label": label, "ok": bool(ok), "detail": detail}) cells.append({"label": label, "ok": bool(ok), "detail": detail})
+54 -11
View File
@@ -1,5 +1,5 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""verify.py -- the HOME the index gate did not have: one command, six legs, """verify.py -- the HOME the index gate did not have: one command, seven legs,
one exit code, and a row per leg so a partial run cannot look like a clean one. one exit code, and a row per leg so a partial run cannot look like a clean one.
WHY THIS EXISTS WHY THIS EXISTS
@@ -54,6 +54,13 @@ WHAT IT RUNS -- one row per leg, cheapest first, never in parallel
the derived document, byte-compared against a fresh render the derived document, byte-compared against a fresh render
append-audit tools/queue_append_audit.py --guard --to HEAD append-audit tools/queue_append_audit.py --guard --to HEAD
the newest queue-touching commit, graded in ITS OWN tree the newest queue-touching commit, graded in ITS OWN tree
doc-commands tools/doc_command_check.py --rev WORKTREE --fail
every command a tool's docstring tells a reader to run: each
flag must be one that tool's own `--help` prints. READ-ONLY --
it runs `--help` and nothing else. WORKTREE, not HEAD, so the
document being written is the document graded; this leg did
not exist before 2026-10-22, and every docstring command in
the archive was prose until it did
selftest-sweep tools/selftest_sweep.py selftest-sweep tools/selftest_sweep.py
every tool's --selftest plus its two read-refusal axes every tool's --selftest plus its two read-refusal axes
(skip with --fast; the row is still printed and counted) (skip with --fast; the row is still printed and counted)
@@ -139,7 +146,7 @@ TOOLS ARE RESOLVED FROM --repo, NOT FROM THIS FILE'S DIRECTORY
The path must still be the ROOT: a subdirectory is rc 2 naming the root, The path must still be the ROOT: a subdirectory is rc 2 naming the root,
because widening `--repo` silently would grade a tree nobody named. because widening `--repo` silently would grade a tree nobody named.
`--selftest` fixture corpus, 51 cells `--selftest` fixture corpus, 55 cells
""" """
import argparse import argparse
@@ -169,6 +176,9 @@ LEGS = (
"what": "render the index and byte-compare it with the document"}, "what": "render the index and byte-compare it with the document"},
{"key": "append-audit", "tool": "queue_append_audit.py", "slow": False, {"key": "append-audit", "tool": "queue_append_audit.py", "slow": False,
"what": "grade the newest queue-touching commit in its own tree"}, "what": "grade the newest queue-touching commit in its own tree"},
{"key": "doc-commands", "tool": "doc_command_check.py", "slow": False,
"what": "every command a tool's docstring tells a reader to run: its "
"flags must be options that tool's --help accepts"},
{"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True, {"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True,
"what": "every tool's --selftest and both read-refusal axes"}, "what": "every tool's --selftest and both read-refusal axes"},
{"key": "write-axis", "tool": "write_probe.py", "slow": False, {"key": "write-axis", "tool": "write_probe.py", "slow": False,
@@ -349,6 +359,13 @@ def leg_argv(key, repo, queue, index, checkin_day=None):
return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"), return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"),
"--repo", repo, "--queue", queue, "--index", index, "--repo", repo, "--queue", queue, "--index", index,
"--guard", "--to", "HEAD"] "--guard", "--to", "HEAD"]
if key == "doc-commands":
# WORKTREE, not HEAD: the commit window is where a docstring is edited,
# and a leg that read HEAD would grade the last commit while the lane
# was writing the next one. The mode is the tool's own, measured there.
return [sys.executable, os.path.join(repo, "tools",
"doc_command_check.py"),
"--repo", repo, "--rev", "WORKTREE", "--fail"]
if key == "selftest-sweep": if key == "selftest-sweep":
return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"), return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"),
"--repo", repo, "--quiet"] "--repo", repo, "--quiet"]
@@ -608,8 +625,8 @@ def _verify(*argv):
GREEN_ALL = {"checkin_check.py": 0, "reply_check.py": 0, "queue_index.py": 0, GREEN_ALL = {"checkin_check.py": 0, "reply_check.py": 0, "queue_index.py": 0,
"queue_append_audit.py": 0, "selftest_sweep.py": 0, "queue_append_audit.py": 0, "doc_command_check.py": 0,
"write_probe.py": 0} "selftest_sweep.py": 0, "write_probe.py": 0}
def selftest(): def selftest():
@@ -626,11 +643,11 @@ def selftest():
failures.append("%s\n got: %r\n want: %r" % (label, got, want)) failures.append("%s\n got: %r\n want: %r" % (label, got, want))
with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td: with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td:
# 1. six green stubs -> rc 0 # 1. seven green stubs -> rc 0
repo = _fixture(os.path.join(td, "c1"), GREEN_ALL) repo = _fixture(os.path.join(td, "c1"), GREEN_ALL)
rc, out = _verify("--repo", repo) rc, out = _verify("--repo", repo)
check("1 all six legs green -> rc 0", rc, 0) check("1 all seven legs green -> rc 0", rc, 0)
check("1b summary counts the six legs", "verify: 6 legs, 6 pass, 0 fault, " check("1b summary counts the seven legs", "verify: 7 legs, 7 pass, 0 fault, "
"0 ungradable, 0 skipped" in out, True) "0 ungradable, 0 skipped" in out, True)
# 2. a stale index is a FAULT, and its own line reaches the row # 2. a stale index is a FAULT, and its own line reaches the row
@@ -686,7 +703,7 @@ def selftest():
obj = json.load(open(jpath)) if os.path.exists(jpath) else {} obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])], check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])],
["checkin", "reply-headings", "index-current", "append-audit", ["checkin", "reply-headings", "index-current", "append-audit",
"selftest-sweep", "write-axis"]) "doc-commands", "selftest-sweep", "write-axis"])
check("7b --json carries the same rc the process returned", check("7b --json carries the same rc the process returned",
obj.get("rc"), rc) obj.get("rc"), rc)
@@ -716,8 +733,8 @@ def selftest():
outs={"queue_index.py": "STALE: line 1\n"}) outs={"queue_index.py": "STALE: line 1\n"})
rc, out = _verify("--repo", repo) rc, out = _verify("--repo", repo)
order = [out.find(k) for k in ("checkin", "reply-headings", "index-current", order = [out.find(k) for k in ("checkin", "reply-headings", "index-current",
"append-audit", "selftest-sweep", "append-audit", "doc-commands",
"write-axis")] "selftest-sweep", "write-axis")]
check("11 legs print in fixed order", order == sorted(order) check("11 legs print in fixed order", order == sorted(order)
and -1 not in order, True) and -1 not in order, True)
@@ -729,7 +746,7 @@ def selftest():
check("12 skipped leg is its own count in --json", check("12 skipped leg is its own count in --json",
obj.get("counts", {}).get("SKIP"), 1) obj.get("counts", {}).get("SKIP"), 1)
check("12b and the skipped leg is not counted as a pass", check("12b and the skipped leg is not counted as a pass",
obj.get("counts", {}).get(PASS), 5) obj.get("counts", {}).get(PASS), 6)
# 13. the detail line must not contradict the verdict: a row graded # 13. the detail line must not contradict the verdict: a row graded
# FAULT beside "0 fatal" is the SILENT failure this gate exists to catch. # FAULT beside "0 fatal" is the SILENT failure this gate exists to catch.
@@ -902,6 +919,32 @@ def selftest():
rc, out = _verify("--repo", bare, "--fast") rc, out = _verify("--repo", bare, "--fast")
check("26 a bare repository -> rc 2", rc, 2) check("26 a bare repository -> rc 2", rc, 2)
# 28. THE SEVENTH LEG, and the tree it must read. The tool is called
# with `--rev WORKTREE --fail`: the commit window is where a docstring is
# edited, and a leg reading HEAD would grade the last commit while the
# lane wrote the next one. Measured here as the argv, not as a promise.
av = os.path.join(td, "c28-argv")
repo = _fixture(os.path.join(td, "c28"), GREEN_ALL,
argv_markers={"doc_command_check.py": av})
rc, out = _verify("--repo", repo, "--fast")
seen = open(av).read() if os.path.exists(av) else ""
check("28 the doc-commands leg runs --rev WORKTREE --fail",
"--rev WORKTREE" in seen and "--fail" in seen, True)
check("28b and the leg runs the tool inside --repo (this stub wrote it)",
os.path.exists(av), True)
# 29. a documented command its own tool would reject is a FAULT on the
# gate, named on the row -- the class that was prose until this leg.
repo = _fixture(os.path.join(td, "c29"),
dict(GREEN_ALL, **{"doc_command_check.py": 1}),
outs={"doc_command_check.py":
"FLAG_NOT_ACCEPTED tools/x.py -> tools/x.py "
"missing=['--gone']\n"})
rc, out = _verify("--repo", repo, "--fast")
check("29 a rejected docstring command -> rc 1", rc, 1)
check("29b the fault row is named doc-commands and carries its reason",
"fault: doc-commands" in out and "FLAG_NOT_ACCEPTED" in out, True)
# 27. the count in this file's own docstring is a CLAIM, so it is graded # 27. the count in this file's own docstring is a CLAIM, so it is graded
# against the count that actually ran. On 2026-10-19 the docstring and # against the count that actually ran. On 2026-10-19 the docstring and
# the run said 50 while the argparse help string still said 42 and # the run said 50 while the argparse help string still said 42 and
+15 -1
View File
@@ -130,6 +130,11 @@ import tempfile
from collections import Counter from collections import Counter
JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']") JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']")
# Same rule as every other tool here: the default target is the checkout this
# file lives in, so a bare run in a clone grades the clone rather than a fixed
# path on this host.
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
# A caller-path write flag other than --json. The class is the two name families # A caller-path write flag other than --json. The class is the two name families
# this repo uses for "write the thing you were given a path for"; see THE SECOND # this repo uses for "write the thing you were given a path for"; see THE SECOND
# AXIS above for the boundary and why --emit-baseline is not in it. # AXIS above for the boundary and why --emit-baseline is not in it.
@@ -407,6 +412,15 @@ def selftest(repo):
""" """
bad = [] bad = []
# Cells below read `repo` (the tool table comes from its tracked tools/), so
# an unreadable repo turns three of them into FAIL -- "your table is wrong"
# when the truth is "I never saw your tree". Refuse at rc 2, the same code
# and the same sentence main() gives this input.
names, why = tracked_tools(repo)
if names is None:
print("cannot read: " + why, file=sys.stderr)
return 2
def cell(name, ok, why=""): def cell(name, ok, why=""):
print((" ok " if ok else " FAIL ") + name print((" ok " if ok else " FAIL ") + name
+ ("" if ok else " [%s]" % why)) + ("" if ok else " [%s]" % why))
@@ -585,7 +599,7 @@ def selftest(repo):
def main(argv=None): def main(argv=None):
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0]) ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
ap.add_argument("--repo", default="/root/troupe-standup") ap.add_argument("--repo", default=DEFAULT_REPO)
ap.add_argument("--only", default=None, metavar="TOOL") ap.add_argument("--only", default=None, metavar="TOOL")
ap.add_argument("--timeout", type=int, default=180, metavar="S") ap.add_argument("--timeout", type=int, default=180, metavar="S")
ap.add_argument("--json", default=None, metavar="PATH") ap.add_argument("--json", default=None, metavar="PATH")