volt: verify gains a seventh leg -- doc-commands, graded at WORKTREE
The gate had six legs and none of them read a docstring's own command line.
doc_command_check.py existed since 838750c but only as a tool: nothing on the
gate ran it, so a documented flag its parser never learned could land and
nobody read it. This is that leg.
doc-commands tools/doc_command_check.py --rev WORKTREE --fail
WORKTREE, not HEAD: the commit window is where a docstring is
edited. Measured at the worktree this lands in: 67 commands
over 19 oracle tools, 67 OK, rc 0.
doc_command_check.py learns --rev WORKTREE: tracked paths from `git ls-files`
(an untracked tools/x.py is not part of the commit this mode grades), bytes
from disk. Outside a work tree it is rc 2, not an empty clean run. Selftest
15 -> 18 cells: one document edited on disk but not committed is rc 0 at HEAD
and rc 1 at WORKTREE, and the flag it names is the edited one.
Three tools stopped defaulting to a fixed path on this host. verify.py,
selftest_sweep.py, write_probe.py, citation_resolve_check.py,
path_claim_check.py and doc_census_check.py now resolve DEFAULT_REPO from the
file's own location, so a bare run in a clone grades the clone.
Three selftests refuse unreadable input at rc 2 instead of reporting it as
failed content: tool_map (not a work tree), write_probe (no table), and
citation_resolve_check (no corpus). Each cell below them graded `repo`, so an
absent tree read as "your table is wrong".
Evidence, all measured on this tree before the commit:
verify.py --selftest 55 cells, 55 passed (docstring says 55)
doc_command_check.py --selftest 18 cells, 18 passed (baseline pinned 18)
tool_map.py --repo . --rev WORKTREE --check docs/TOOLS.md rc 0
verify.py --repo . 7 legs, 6 pass, 1 fault -- selftest-sweep,
tool_map's drift cell, which renders HEAD
and so cannot pass inside a commit window
tools/selftest_baseline.tsv re-pins doc_command_check.py 15 -> 18 cells.
This commit is contained in:
1 parent
af32c34762
commit
c3d70567a6
10 files changed
+192
-23
No files matched your search
+3
-2
@@ -19,7 +19,7 @@ module docstring, flags from the tool's own `--help`, probe rows from
|
|||||||
| `citation_hash_class.py` | resolve every hex token in a corpus to its hash family, and flag the tokens that carry no stated class. | `--repo` `--rev` `--corpus` `--json` `--window` `--min-len` `--only` `--context` `--siblings` `--selftest` | 1 / 0 / 1 | - |
|
| `citation_hash_class.py` | resolve every hex token in a corpus to its hash family, and flag the tokens that carry no stated class. | `--repo` `--rev` `--corpus` `--json` `--window` `--min-len` `--only` `--context` `--siblings` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `citation_resolve_check.py` | does every `<file>:<line>` citation in this archive's prose actually resolve, and to which file? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--selftest` | 1 / 0 / 1 | - |
|
| `citation_resolve_check.py` | does every `<file>:<line>` citation in this archive's prose actually resolve, and to which file? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `doc_census_check.py` | does a tool's own docstring census match the census that tool PRINTS? | `--repo` `--rev` `--tools` `--json` `--no-run` `--fail` `--vocab` `--save-vocab` `--selftest` | 1 / 0 / 1 | - |
|
| `doc_census_check.py` | does a tool's own docstring census match the census that tool PRINTS? | `--repo` `--rev` `--tools` `--json` `--no-run` `--fail` `--vocab` `--save-vocab` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `doc_command_check.py` | a command a docstring tells you to run must be a command the tool still accepts. | `--repo` `--rev` `--tools` `--json` `--fail` `--selftest` | 1 / 0 / 1 | - |
|
| `doc_command_check.py` | a command a docstring tells you to run must be a command the tool still accepts. | `--repo` `--rev` `--tools` `--json` `--fail` `--selftest` | 1 / 0 / 1 | `doc-commands` |
|
||||||
| `gate_blame.py` | which landing turned a gate leg red? | `--repo` `--leg` `--range` `--ref` `--queue` `--index` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
| `gate_blame.py` | which landing turned a gate leg red? | `--repo` `--leg` `--range` `--ref` `--queue` `--index` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `landing_check.py` | is every file at this revision actually served by the forge? | `--repo` `--rev` `--branch` `--base-url` `--path` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
| `landing_check.py` | is every file at this revision actually served by the forge? | `--repo` `--rev` `--branch` `--base-url` `--path` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `path_claim_check.py` | does every in-namespace path this archive names exist at this revision? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--fail-on-missing` `--baseline` `--selftest` | 1 / 0 / 1 | - |
|
| `path_claim_check.py` | does every in-namespace path this archive names exist at this revision? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--fail-on-missing` `--baseline` `--selftest` | 1 / 0 / 1 | - |
|
||||||
@@ -30,7 +30,7 @@ module docstring, flags from the tool's own `--help`, probe rows from
|
|||||||
| `selftest_sweep.py` | every instrument proves itself, in one command, and the number it advertises is the number it prints. | `--repo` `--tools-dir` `--timeout` `--json` `--baseline` `--no-baseline` `--write-baseline` `--quiet` `--verbose` `--selftest` | 1 / 0 / 1 | `selftest-sweep` |
|
| `selftest_sweep.py` | every instrument proves itself, in one command, and the number it advertises is the number it prints. | `--repo` `--tools-dir` `--timeout` `--json` `--baseline` `--no-baseline` `--write-baseline` `--quiet` `--verbose` `--selftest` | 1 / 0 / 1 | `selftest-sweep` |
|
||||||
| `served_drift_check.py` | does the host actually serve the tree at this revision? | `--root` `--rev` `--base-url` `--prefix` `--exclude` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
| `served_drift_check.py` | does the host actually serve the tree at this revision? | `--root` `--rev` `--base-url` `--prefix` `--exclude` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `tool_map.py` | the instruments get a map, and the map is generated from the tree. | `--repo` `--rev` `--write` `--check` `--json` `--selftest` | 1 / 0 / 1 | - |
|
| `tool_map.py` | the instruments get a map, and the map is generated from the tree. | `--repo` `--rev` `--write` `--check` `--json` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `verify.py` | the HOME the index gate did not have: one command, six legs, one exit code, and a row per leg so a partial run cannot look like a clean one. | `--repo` `--queue` `--index` `--checkin-day` `--fast` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
| `verify.py` | the HOME the index gate did not have: one command, seven legs, one exit code, and a row per leg so a partial run cannot look like a clean one. | `--repo` `--queue` `--index` `--checkin-day` `--fast` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
|
||||||
| `write_probe.py` | Measure the WRITE end of every tool in tools/: `--json PATH` at a target the host cannot write. | `--repo` `--only` `--timeout` `--json` `--selftest` `--verbose` | 1 / 0 / 1 | `write-axis` |
|
| `write_probe.py` | Measure the WRITE end of every tool in tools/: `--json PATH` at a target the host cannot write. | `--repo` `--only` `--timeout` `--json` `--selftest` `--verbose` | 1 / 0 / 1 | `write-axis` |
|
||||||
|
|
||||||
## The gate
|
## The gate
|
||||||
@@ -45,6 +45,7 @@ dropped by `--fast`.
|
|||||||
| `reply-headings` | `reply_check.py` | no | grade every reply's heading against its own filename |
|
| `reply-headings` | `reply_check.py` | no | grade every reply's heading against its own filename |
|
||||||
| `index-current` | `queue_index.py` | no | render the index and byte-compare it with the document |
|
| `index-current` | `queue_index.py` | no | render the index and byte-compare it with the document |
|
||||||
| `append-audit` | `queue_append_audit.py` | no | grade the newest queue-touching commit in its own tree |
|
| `append-audit` | `queue_append_audit.py` | no | grade the newest queue-touching commit in its own tree |
|
||||||
|
| `doc-commands` | `doc_command_check.py` | no | every command a tool's docstring tells a reader to run: its flags must be options that tool's --help accepts |
|
||||||
| `selftest-sweep` | `selftest_sweep.py` | yes | every tool's --selftest and both read-refusal axes |
|
| `selftest-sweep` | `selftest_sweep.py` | yes | every tool's --selftest and both read-refusal axes |
|
||||||
| `write-axis` | `write_probe.py` | no | a --json target this host cannot write: rc 2, one line naming it, never a traceback -- and the same rule for every caller-path write flag (--write, --save-vocab) |
|
| `write-axis` | `write_probe.py` | no | a --json target this host cannot write: rc 2, one line naming it, never a traceback -- and the same rule for every caller-path write flag (--write, --save-vocab) |
|
||||||
|
|
||||||
|
|||||||
@@ -133,11 +133,12 @@ USAGE
|
|||||||
import argparse
|
import argparse
|
||||||
import collections
|
import collections
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import re
|
import re
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
DEFAULT_REPO = "/root/troupe-standup"
|
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
# The extensions this archive actually cites by line. Deliberately a closed list:
|
# The extensions this archive actually cites by line. Deliberately a closed list:
|
||||||
# an open `\.[a-z]+` would swallow clock times and domain names.
|
# an open `\.[a-z]+` would swallow clock times and domain names.
|
||||||
@@ -401,6 +402,14 @@ def _expect(here, text, store):
|
|||||||
|
|
||||||
|
|
||||||
def selftest(repo):
|
def selftest(repo):
|
||||||
|
# The fixtures grade real citations out of `repo`, so a repo that cannot be
|
||||||
|
# read turns every fixture into a mismatch -- eight failures that say "your
|
||||||
|
# content is wrong" when the truth is "I never saw your content". Refuse at
|
||||||
|
# rc 2, the code this tool's own main() uses for the same condition.
|
||||||
|
problem = precondition(repo, "HEAD")
|
||||||
|
if problem:
|
||||||
|
print(problem, file=sys.stderr)
|
||||||
|
return 2
|
||||||
store = Store(repo, "HEAD")
|
store = Store(repo, "HEAD")
|
||||||
n = bad = 0
|
n = bad = 0
|
||||||
for here, text, want in FIXTURES:
|
for here, text, want in FIXTURES:
|
||||||
|
|||||||
@@ -259,7 +259,7 @@ import subprocess
|
|||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
DEFAULT_REPO = "/root/troupe-standup"
|
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
# A claim, in either of the two forms this archive writes:
|
# A claim, in either of the two forms this archive writes:
|
||||||
# `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations"
|
# `<value> <unit>` -- "471 UNRESOLVED", "1,612 citations"
|
||||||
|
|||||||
@@ -50,6 +50,13 @@ revision: a command documented at `R` is checked against the tool at `R`. The
|
|||||||
default is `HEAD`. A revision that does not materialise is rc 2, never an empty
|
default is `HEAD`. A revision that does not materialise is rc 2, never an empty
|
||||||
clean run.
|
clean run.
|
||||||
|
|
||||||
|
`--rev WORKTREE` is not a revision: it reads the files on disk and lists them
|
||||||
|
from the INDEX, so it grades the tree you are about to commit. `HEAD` cannot
|
||||||
|
see an uncommitted docstring, which is the window a lane writes in -- the same
|
||||||
|
blind spot `tool_map.py --rev WORKTREE` exists for. The two are measured
|
||||||
|
apart in the selftest: one document, edited on disk but not committed, is
|
||||||
|
rc 0 at `HEAD` and rc 1 at `WORKTREE`.
|
||||||
|
|
||||||
EXIT CODES (the tools/ contract)
|
EXIT CODES (the tools/ contract)
|
||||||
0 every documented command was adjudicated and none was rejected
|
0 every documented command was adjudicated and none was rejected
|
||||||
1 at least one documented command is a fault (FLAG_NOT_ACCEPTED, TOOL_MISSING)
|
1 at least one documented command is a fault (FLAG_NOT_ACCEPTED, TOOL_MISSING)
|
||||||
@@ -59,10 +66,12 @@ EXIT CODES (the tools/ contract)
|
|||||||
USAGE
|
USAGE
|
||||||
python3 doc_command_check.py --repo . [--rev REV] [--tools a.py,b.py]
|
python3 doc_command_check.py --repo . [--rev REV] [--tools a.py,b.py]
|
||||||
[--json OUT] [--fail]
|
[--json OUT] [--fail]
|
||||||
python3 doc_command_check.py --selftest # 15 cells, oracle and verdicts
|
python3 doc_command_check.py --selftest # 18 cells, oracle and verdicts
|
||||||
"""
|
"""
|
||||||
import argparse
|
import argparse
|
||||||
import ast
|
import ast
|
||||||
|
import contextlib
|
||||||
|
import io
|
||||||
import json
|
import json
|
||||||
import os
|
import os
|
||||||
import re
|
import re
|
||||||
@@ -86,6 +95,11 @@ OPT_RE = re.compile(r"(?<![\w-])(--[A-Za-z][\w-]*|-[A-Za-z])(?![\w-])")
|
|||||||
|
|
||||||
HELP_TIMEOUT = 60
|
HELP_TIMEOUT = 60
|
||||||
|
|
||||||
|
# Not a revision: read the files on disk, list them from the index. The same
|
||||||
|
# spelling `tool_map.py` uses, for the same reason -- a commit window is
|
||||||
|
# otherwise ungraded and the defect lands at HEAD with nobody having read it.
|
||||||
|
WORKTREE = "WORKTREE"
|
||||||
|
|
||||||
|
|
||||||
class CannotGrade(Exception):
|
class CannotGrade(Exception):
|
||||||
"""Input this tool could not read or could not recognise: rc 2, never rc 1.
|
"""Input this tool could not read or could not recognise: rc 2, never rc 1.
|
||||||
@@ -97,9 +111,17 @@ class CannotGrade(Exception):
|
|||||||
|
|
||||||
def tracked_tools(repo, rev):
|
def tracked_tools(repo, rev):
|
||||||
"""(rc, stdout) of `git ls-tree` -> the tool paths in the tree at `rev`."""
|
"""(rc, stdout) of `git ls-tree` -> the tool paths in the tree at `rev`."""
|
||||||
proc = subprocess.run(
|
if rev == WORKTREE:
|
||||||
["git", "-C", repo, "ls-tree", "-r", "--name-only", rev, "--", "tools"],
|
# The index, not the disk: an untracked `tools/x.py` is not part of the
|
||||||
capture_output=True, text=True)
|
# commit this mode grades, and including it would grade a file that is
|
||||||
|
# about to be left behind. `git ls-files` fails outside a work tree,
|
||||||
|
# which is the rc 2 this mode owes a caller who ran it anywhere else.
|
||||||
|
proc = subprocess.run(["git", "-C", repo, "ls-files", "--", "tools"],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
else:
|
||||||
|
proc = subprocess.run(
|
||||||
|
["git", "-C", repo, "ls-tree", "-r", "--name-only", rev, "--", "tools"],
|
||||||
|
capture_output=True, text=True)
|
||||||
if proc.returncode != 0:
|
if proc.returncode != 0:
|
||||||
raise CannotGrade("cannot read repo: %s at %s: %s"
|
raise CannotGrade("cannot read repo: %s at %s: %s"
|
||||||
% (repo, rev, (proc.stderr or "").strip()[:200]))
|
% (repo, rev, (proc.stderr or "").strip()[:200]))
|
||||||
@@ -109,6 +131,18 @@ def tracked_tools(repo, rev):
|
|||||||
|
|
||||||
def blob(repo, rev, path):
|
def blob(repo, rev, path):
|
||||||
"""The bytes of `path` at `rev`, or None when the tree does not hold it."""
|
"""The bytes of `path` at `rev`, or None when the tree does not hold it."""
|
||||||
|
if rev == WORKTREE:
|
||||||
|
try:
|
||||||
|
with open(os.path.join(repo, *path.split("/")),
|
||||||
|
"r", encoding="utf-8") as fh:
|
||||||
|
return fh.read()
|
||||||
|
except FileNotFoundError:
|
||||||
|
return None
|
||||||
|
except OSError as exc:
|
||||||
|
raise CannotGrade("cannot read %s at WORKTREE: %s" % (path, exc))
|
||||||
|
except UnicodeDecodeError as exc:
|
||||||
|
raise CannotGrade("cannot read %s at WORKTREE: not UTF-8: %s"
|
||||||
|
% (path, exc))
|
||||||
proc = subprocess.run(["git", "-C", repo, "show", "%s:%s" % (rev, path)],
|
proc = subprocess.run(["git", "-C", repo, "show", "%s:%s" % (rev, path)],
|
||||||
capture_output=True)
|
capture_output=True)
|
||||||
if proc.returncode != 0:
|
if proc.returncode != 0:
|
||||||
@@ -294,6 +328,58 @@ def selftest():
|
|||||||
harvested = set(OPT_RE.findall("usage: x [-h] [--fail] [--json JSON]\n"))
|
harvested = set(OPT_RE.findall("usage: x [-h] [--fail] [--json JSON]\n"))
|
||||||
cell("bracketed usage options are harvested", harvested >= {"-h", "--fail", "--json"})
|
cell("bracketed usage options are harvested", harvested >= {"-h", "--fail", "--json"})
|
||||||
|
|
||||||
|
# WORKTREE: the tree you are about to commit, not the last commit. One
|
||||||
|
# document, edited on disk and NOT committed: HEAD reads the blob (clean),
|
||||||
|
# WORKTREE reads the disk (the documented flag its parser never learned).
|
||||||
|
with tempfile.TemporaryDirectory(prefix="doc-cmd-wt-") as td:
|
||||||
|
repo = os.path.join(td, "repo")
|
||||||
|
os.makedirs(os.path.join(repo, "tools"))
|
||||||
|
tool = os.path.join(repo, "tools", "oracle_tool.py")
|
||||||
|
|
||||||
|
def write_doc(extra=""):
|
||||||
|
with open(tool, "w", encoding="utf-8") as fh:
|
||||||
|
fh.write("#!/usr/bin/env python3\n"
|
||||||
|
'"""USAGE\n'
|
||||||
|
" python3 oracle_tool.py --present\n"
|
||||||
|
+ extra +
|
||||||
|
'"""\n'
|
||||||
|
"import argparse\n"
|
||||||
|
"ap = argparse.ArgumentParser()\n"
|
||||||
|
"ap.add_argument('--present')\n"
|
||||||
|
"ap.parse_args()\n")
|
||||||
|
|
||||||
|
def git(*args):
|
||||||
|
subprocess.run(["git", "-C", repo] + list(args),
|
||||||
|
capture_output=True, text=True)
|
||||||
|
|
||||||
|
write_doc()
|
||||||
|
git("init", "-q")
|
||||||
|
git("add", "-A")
|
||||||
|
git("-c", "user.name=selftest", "-c", "user.email=selftest@invalid",
|
||||||
|
"-c", "commit.gpgsign=false", "commit", "-q", "-m", "fixture")
|
||||||
|
write_doc(" python3 oracle_tool.py --gone-after-edit\n")
|
||||||
|
|
||||||
|
head_rep = census(repo, "HEAD")
|
||||||
|
wt_rep = census(repo, WORKTREE)
|
||||||
|
cell("WORKTREE reads the document on disk, HEAD reads the blob at HEAD",
|
||||||
|
[r["verdict"] for r in wt_rep["rows"]] == ["OK", "FLAG_NOT_ACCEPTED"]
|
||||||
|
and [r["verdict"] for r in head_rep["rows"]] == ["OK"],
|
||||||
|
([r["verdict"] for r in head_rep["rows"]],
|
||||||
|
[r["verdict"] for r in wt_rep["rows"]]))
|
||||||
|
cell("the uncommitted command's flag is the one named",
|
||||||
|
[r["missing"] for r in wt_rep["rows"] if r["missing"]] ==
|
||||||
|
[["--gone-after-edit"]],
|
||||||
|
[r["missing"] for r in wt_rep["rows"] if r["missing"]])
|
||||||
|
|
||||||
|
outsider = os.path.join(td, "not-a-repo")
|
||||||
|
os.makedirs(outsider)
|
||||||
|
buf = io.StringIO()
|
||||||
|
with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf):
|
||||||
|
rc_nogit = run(["--repo", outsider, "--rev", WORKTREE, "--fail"])
|
||||||
|
cell("WORKTREE outside a git work tree is rc 2, not a clean run",
|
||||||
|
rc_nogit == 2 and "cannot read repo" in buf.getvalue(),
|
||||||
|
(rc_nogit, buf.getvalue().strip()[:80]))
|
||||||
|
|
||||||
# verdicts
|
# verdicts
|
||||||
rep = {"rev": "HEAD", "oracles": 1, "rows": [
|
rep = {"rev": "HEAD", "oracles": 1, "rows": [
|
||||||
{"doc": "tools/a.py", "target": "tools/a.py", "verdict": "OK",
|
{"doc": "tools/a.py", "target": "tools/a.py", "verdict": "OK",
|
||||||
|
|||||||
@@ -126,7 +126,7 @@ import re
|
|||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
|
||||||
DEFAULT_REPO = "/root/troupe-standup"
|
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
# The ledger this archive carries, REPO-relative: resolved against --repo, not
|
# The ledger this archive carries, REPO-relative: resolved against --repo, not
|
||||||
# against the caller's cwd, so a bare run against a clone reads that clone's
|
# against the caller's cwd, so a bare run against a clone reads that clone's
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ checkin_check.py 36 cells
|
|||||||
citation_hash_class.py 19 fixtures
|
citation_hash_class.py 19 fixtures
|
||||||
citation_resolve_check.py 14 fixtures
|
citation_resolve_check.py 14 fixtures
|
||||||
doc_census_check.py 104 cells
|
doc_census_check.py 104 cells
|
||||||
doc_command_check.py 15 cells
|
doc_command_check.py 18 cells
|
||||||
gate_blame.py 21 cells
|
gate_blame.py 21 cells
|
||||||
landing_check.py 26 cells
|
landing_check.py 26 cells
|
||||||
path_claim_check.py 27 cells
|
path_claim_check.py 27 cells
|
||||||
|
|||||||
|
@@ -149,6 +149,12 @@ import subprocess
|
|||||||
import sys
|
import sys
|
||||||
import tempfile
|
import tempfile
|
||||||
|
|
||||||
|
# The default target is the checkout this file lives in -- the same path in
|
||||||
|
# place, a different path in a copy. An absolute literal here would make a bare
|
||||||
|
# `--repo`-less run grade /root/troupe-standup from anywhere on the host, which
|
||||||
|
# is the wrong-tree defect `tool_map.py` documents at length.
|
||||||
|
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
|
|
||||||
UNITS = ("cells", "fixtures")
|
UNITS = ("cells", "fixtures")
|
||||||
|
|
||||||
# `selftest: 18 cells, 18 passed, 0 failed`
|
# `selftest: 18 cells, 18 passed, 0 failed`
|
||||||
@@ -1037,7 +1043,7 @@ def selftest():
|
|||||||
def main(argv=None):
|
def main(argv=None):
|
||||||
ap = argparse.ArgumentParser(
|
ap = argparse.ArgumentParser(
|
||||||
description="Run every tool's --selftest and check its advertised cell count.")
|
description="Run every tool's --selftest and check its advertised cell count.")
|
||||||
ap.add_argument("--repo", default="/root/troupe-standup")
|
ap.add_argument("--repo", default=DEFAULT_REPO)
|
||||||
ap.add_argument("--tools-dir", default=None,
|
ap.add_argument("--tools-dir", default=None,
|
||||||
help="sweep this directory instead of the repo's tracked tools/")
|
help="sweep this directory instead of the repo's tracked tools/")
|
||||||
ap.add_argument("--timeout", type=int, default=120, metavar="S")
|
ap.add_argument("--timeout", type=int, default=120, metavar="S")
|
||||||
|
|||||||
@@ -394,6 +394,16 @@ def render(data):
|
|||||||
def selftest(repo):
|
def selftest(repo):
|
||||||
cells = []
|
cells = []
|
||||||
|
|
||||||
|
# The currency cell below grades `repo`; if `repo` is not a work tree that
|
||||||
|
# cell can only fail, which reads as "the document drifted" when the truth
|
||||||
|
# is "no tree was there". Refuse at rc 2, as main() already does for the
|
||||||
|
# same input.
|
||||||
|
rc, _, _ = git(repo, "rev-parse", "--show-toplevel")
|
||||||
|
if rc != 0:
|
||||||
|
print(f"tool_map: cannot read repo: {repo} is not a git work tree",
|
||||||
|
file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
def cell(label, ok, detail=""):
|
def cell(label, ok, detail=""):
|
||||||
cells.append({"label": label, "ok": bool(ok), "detail": detail})
|
cells.append({"label": label, "ok": bool(ok), "detail": detail})
|
||||||
|
|
||||||
|
|||||||
+54
-11
@@ -1,5 +1,5 @@
|
|||||||
#!/usr/bin/env python3
|
#!/usr/bin/env python3
|
||||||
"""verify.py -- the HOME the index gate did not have: one command, six legs,
|
"""verify.py -- the HOME the index gate did not have: one command, seven legs,
|
||||||
one exit code, and a row per leg so a partial run cannot look like a clean one.
|
one exit code, and a row per leg so a partial run cannot look like a clean one.
|
||||||
|
|
||||||
WHY THIS EXISTS
|
WHY THIS EXISTS
|
||||||
@@ -54,6 +54,13 @@ WHAT IT RUNS -- one row per leg, cheapest first, never in parallel
|
|||||||
the derived document, byte-compared against a fresh render
|
the derived document, byte-compared against a fresh render
|
||||||
append-audit tools/queue_append_audit.py --guard --to HEAD
|
append-audit tools/queue_append_audit.py --guard --to HEAD
|
||||||
the newest queue-touching commit, graded in ITS OWN tree
|
the newest queue-touching commit, graded in ITS OWN tree
|
||||||
|
doc-commands tools/doc_command_check.py --rev WORKTREE --fail
|
||||||
|
every command a tool's docstring tells a reader to run: each
|
||||||
|
flag must be one that tool's own `--help` prints. READ-ONLY --
|
||||||
|
it runs `--help` and nothing else. WORKTREE, not HEAD, so the
|
||||||
|
document being written is the document graded; this leg did
|
||||||
|
not exist before 2026-10-22, and every docstring command in
|
||||||
|
the archive was prose until it did
|
||||||
selftest-sweep tools/selftest_sweep.py
|
selftest-sweep tools/selftest_sweep.py
|
||||||
every tool's --selftest plus its two read-refusal axes
|
every tool's --selftest plus its two read-refusal axes
|
||||||
(skip with --fast; the row is still printed and counted)
|
(skip with --fast; the row is still printed and counted)
|
||||||
@@ -139,7 +146,7 @@ TOOLS ARE RESOLVED FROM --repo, NOT FROM THIS FILE'S DIRECTORY
|
|||||||
The path must still be the ROOT: a subdirectory is rc 2 naming the root,
|
The path must still be the ROOT: a subdirectory is rc 2 naming the root,
|
||||||
because widening `--repo` silently would grade a tree nobody named.
|
because widening `--repo` silently would grade a tree nobody named.
|
||||||
|
|
||||||
`--selftest` fixture corpus, 51 cells
|
`--selftest` fixture corpus, 55 cells
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
@@ -169,6 +176,9 @@ LEGS = (
|
|||||||
"what": "render the index and byte-compare it with the document"},
|
"what": "render the index and byte-compare it with the document"},
|
||||||
{"key": "append-audit", "tool": "queue_append_audit.py", "slow": False,
|
{"key": "append-audit", "tool": "queue_append_audit.py", "slow": False,
|
||||||
"what": "grade the newest queue-touching commit in its own tree"},
|
"what": "grade the newest queue-touching commit in its own tree"},
|
||||||
|
{"key": "doc-commands", "tool": "doc_command_check.py", "slow": False,
|
||||||
|
"what": "every command a tool's docstring tells a reader to run: its "
|
||||||
|
"flags must be options that tool's --help accepts"},
|
||||||
{"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True,
|
{"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True,
|
||||||
"what": "every tool's --selftest and both read-refusal axes"},
|
"what": "every tool's --selftest and both read-refusal axes"},
|
||||||
{"key": "write-axis", "tool": "write_probe.py", "slow": False,
|
{"key": "write-axis", "tool": "write_probe.py", "slow": False,
|
||||||
@@ -349,6 +359,13 @@ def leg_argv(key, repo, queue, index, checkin_day=None):
|
|||||||
return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"),
|
return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"),
|
||||||
"--repo", repo, "--queue", queue, "--index", index,
|
"--repo", repo, "--queue", queue, "--index", index,
|
||||||
"--guard", "--to", "HEAD"]
|
"--guard", "--to", "HEAD"]
|
||||||
|
if key == "doc-commands":
|
||||||
|
# WORKTREE, not HEAD: the commit window is where a docstring is edited,
|
||||||
|
# and a leg that read HEAD would grade the last commit while the lane
|
||||||
|
# was writing the next one. The mode is the tool's own, measured there.
|
||||||
|
return [sys.executable, os.path.join(repo, "tools",
|
||||||
|
"doc_command_check.py"),
|
||||||
|
"--repo", repo, "--rev", "WORKTREE", "--fail"]
|
||||||
if key == "selftest-sweep":
|
if key == "selftest-sweep":
|
||||||
return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"),
|
return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"),
|
||||||
"--repo", repo, "--quiet"]
|
"--repo", repo, "--quiet"]
|
||||||
@@ -608,8 +625,8 @@ def _verify(*argv):
|
|||||||
|
|
||||||
|
|
||||||
GREEN_ALL = {"checkin_check.py": 0, "reply_check.py": 0, "queue_index.py": 0,
|
GREEN_ALL = {"checkin_check.py": 0, "reply_check.py": 0, "queue_index.py": 0,
|
||||||
"queue_append_audit.py": 0, "selftest_sweep.py": 0,
|
"queue_append_audit.py": 0, "doc_command_check.py": 0,
|
||||||
"write_probe.py": 0}
|
"selftest_sweep.py": 0, "write_probe.py": 0}
|
||||||
|
|
||||||
|
|
||||||
def selftest():
|
def selftest():
|
||||||
@@ -626,11 +643,11 @@ def selftest():
|
|||||||
failures.append("%s\n got: %r\n want: %r" % (label, got, want))
|
failures.append("%s\n got: %r\n want: %r" % (label, got, want))
|
||||||
|
|
||||||
with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td:
|
with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td:
|
||||||
# 1. six green stubs -> rc 0
|
# 1. seven green stubs -> rc 0
|
||||||
repo = _fixture(os.path.join(td, "c1"), GREEN_ALL)
|
repo = _fixture(os.path.join(td, "c1"), GREEN_ALL)
|
||||||
rc, out = _verify("--repo", repo)
|
rc, out = _verify("--repo", repo)
|
||||||
check("1 all six legs green -> rc 0", rc, 0)
|
check("1 all seven legs green -> rc 0", rc, 0)
|
||||||
check("1b summary counts the six legs", "verify: 6 legs, 6 pass, 0 fault, "
|
check("1b summary counts the seven legs", "verify: 7 legs, 7 pass, 0 fault, "
|
||||||
"0 ungradable, 0 skipped" in out, True)
|
"0 ungradable, 0 skipped" in out, True)
|
||||||
|
|
||||||
# 2. a stale index is a FAULT, and its own line reaches the row
|
# 2. a stale index is a FAULT, and its own line reaches the row
|
||||||
@@ -686,7 +703,7 @@ def selftest():
|
|||||||
obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
|
obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
|
||||||
check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])],
|
check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])],
|
||||||
["checkin", "reply-headings", "index-current", "append-audit",
|
["checkin", "reply-headings", "index-current", "append-audit",
|
||||||
"selftest-sweep", "write-axis"])
|
"doc-commands", "selftest-sweep", "write-axis"])
|
||||||
check("7b --json carries the same rc the process returned",
|
check("7b --json carries the same rc the process returned",
|
||||||
obj.get("rc"), rc)
|
obj.get("rc"), rc)
|
||||||
|
|
||||||
@@ -716,8 +733,8 @@ def selftest():
|
|||||||
outs={"queue_index.py": "STALE: line 1\n"})
|
outs={"queue_index.py": "STALE: line 1\n"})
|
||||||
rc, out = _verify("--repo", repo)
|
rc, out = _verify("--repo", repo)
|
||||||
order = [out.find(k) for k in ("checkin", "reply-headings", "index-current",
|
order = [out.find(k) for k in ("checkin", "reply-headings", "index-current",
|
||||||
"append-audit", "selftest-sweep",
|
"append-audit", "doc-commands",
|
||||||
"write-axis")]
|
"selftest-sweep", "write-axis")]
|
||||||
check("11 legs print in fixed order", order == sorted(order)
|
check("11 legs print in fixed order", order == sorted(order)
|
||||||
and -1 not in order, True)
|
and -1 not in order, True)
|
||||||
|
|
||||||
@@ -729,7 +746,7 @@ def selftest():
|
|||||||
check("12 skipped leg is its own count in --json",
|
check("12 skipped leg is its own count in --json",
|
||||||
obj.get("counts", {}).get("SKIP"), 1)
|
obj.get("counts", {}).get("SKIP"), 1)
|
||||||
check("12b and the skipped leg is not counted as a pass",
|
check("12b and the skipped leg is not counted as a pass",
|
||||||
obj.get("counts", {}).get(PASS), 5)
|
obj.get("counts", {}).get(PASS), 6)
|
||||||
|
|
||||||
# 13. the detail line must not contradict the verdict: a row graded
|
# 13. the detail line must not contradict the verdict: a row graded
|
||||||
# FAULT beside "0 fatal" is the SILENT failure this gate exists to catch.
|
# FAULT beside "0 fatal" is the SILENT failure this gate exists to catch.
|
||||||
@@ -902,6 +919,32 @@ def selftest():
|
|||||||
rc, out = _verify("--repo", bare, "--fast")
|
rc, out = _verify("--repo", bare, "--fast")
|
||||||
check("26 a bare repository -> rc 2", rc, 2)
|
check("26 a bare repository -> rc 2", rc, 2)
|
||||||
|
|
||||||
|
# 28. THE SEVENTH LEG, and the tree it must read. The tool is called
|
||||||
|
# with `--rev WORKTREE --fail`: the commit window is where a docstring is
|
||||||
|
# edited, and a leg reading HEAD would grade the last commit while the
|
||||||
|
# lane wrote the next one. Measured here as the argv, not as a promise.
|
||||||
|
av = os.path.join(td, "c28-argv")
|
||||||
|
repo = _fixture(os.path.join(td, "c28"), GREEN_ALL,
|
||||||
|
argv_markers={"doc_command_check.py": av})
|
||||||
|
rc, out = _verify("--repo", repo, "--fast")
|
||||||
|
seen = open(av).read() if os.path.exists(av) else ""
|
||||||
|
check("28 the doc-commands leg runs --rev WORKTREE --fail",
|
||||||
|
"--rev WORKTREE" in seen and "--fail" in seen, True)
|
||||||
|
check("28b and the leg runs the tool inside --repo (this stub wrote it)",
|
||||||
|
os.path.exists(av), True)
|
||||||
|
|
||||||
|
# 29. a documented command its own tool would reject is a FAULT on the
|
||||||
|
# gate, named on the row -- the class that was prose until this leg.
|
||||||
|
repo = _fixture(os.path.join(td, "c29"),
|
||||||
|
dict(GREEN_ALL, **{"doc_command_check.py": 1}),
|
||||||
|
outs={"doc_command_check.py":
|
||||||
|
"FLAG_NOT_ACCEPTED tools/x.py -> tools/x.py "
|
||||||
|
"missing=['--gone']\n"})
|
||||||
|
rc, out = _verify("--repo", repo, "--fast")
|
||||||
|
check("29 a rejected docstring command -> rc 1", rc, 1)
|
||||||
|
check("29b the fault row is named doc-commands and carries its reason",
|
||||||
|
"fault: doc-commands" in out and "FLAG_NOT_ACCEPTED" in out, True)
|
||||||
|
|
||||||
# 27. the count in this file's own docstring is a CLAIM, so it is graded
|
# 27. the count in this file's own docstring is a CLAIM, so it is graded
|
||||||
# against the count that actually ran. On 2026-10-19 the docstring and
|
# against the count that actually ran. On 2026-10-19 the docstring and
|
||||||
# the run said 50 while the argparse help string still said 42 and
|
# the run said 50 while the argparse help string still said 42 and
|
||||||
|
|||||||
+15
-1
@@ -130,6 +130,11 @@ import tempfile
|
|||||||
from collections import Counter
|
from collections import Counter
|
||||||
|
|
||||||
JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']")
|
JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']")
|
||||||
|
|
||||||
|
# Same rule as every other tool here: the default target is the checkout this
|
||||||
|
# file lives in, so a bare run in a clone grades the clone rather than a fixed
|
||||||
|
# path on this host.
|
||||||
|
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||||||
# A caller-path write flag other than --json. The class is the two name families
|
# A caller-path write flag other than --json. The class is the two name families
|
||||||
# this repo uses for "write the thing you were given a path for"; see THE SECOND
|
# this repo uses for "write the thing you were given a path for"; see THE SECOND
|
||||||
# AXIS above for the boundary and why --emit-baseline is not in it.
|
# AXIS above for the boundary and why --emit-baseline is not in it.
|
||||||
@@ -407,6 +412,15 @@ def selftest(repo):
|
|||||||
"""
|
"""
|
||||||
bad = []
|
bad = []
|
||||||
|
|
||||||
|
# Cells below read `repo` (the tool table comes from its tracked tools/), so
|
||||||
|
# an unreadable repo turns three of them into FAIL -- "your table is wrong"
|
||||||
|
# when the truth is "I never saw your tree". Refuse at rc 2, the same code
|
||||||
|
# and the same sentence main() gives this input.
|
||||||
|
names, why = tracked_tools(repo)
|
||||||
|
if names is None:
|
||||||
|
print("cannot read: " + why, file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
def cell(name, ok, why=""):
|
def cell(name, ok, why=""):
|
||||||
print((" ok " if ok else " FAIL ") + name
|
print((" ok " if ok else " FAIL ") + name
|
||||||
+ ("" if ok else " [%s]" % why))
|
+ ("" if ok else " [%s]" % why))
|
||||||
@@ -585,7 +599,7 @@ def selftest(repo):
|
|||||||
|
|
||||||
def main(argv=None):
|
def main(argv=None):
|
||||||
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
ap = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||||
ap.add_argument("--repo", default="/root/troupe-standup")
|
ap.add_argument("--repo", default=DEFAULT_REPO)
|
||||||
ap.add_argument("--only", default=None, metavar="TOOL")
|
ap.add_argument("--only", default=None, metavar="TOOL")
|
||||||
ap.add_argument("--timeout", type=int, default=180, metavar="S")
|
ap.add_argument("--timeout", type=int, default=180, metavar="S")
|
||||||
ap.add_argument("--json", default=None, metavar="PATH")
|
ap.add_argument("--json", default=None, metavar="PATH")
|
||||||
|
|||||||
Reference in new issue
Block a user