stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

tool_map (volt): the seventeen instruments get a rendered map -- purpose from the module docstring, flags from each tool's own --help cut to its usage paragraph, probe rows from selftest_sweep's tables via ast, gate leg from verify's LEGS; drift guard is the selftest's currency cell (rc 1 on two committed mutants, rc 0 on their parent); 16 cells 16 passed, sweep 22/22 + 20/20, doc_command 50 commands / 15 oracles

Browse Source
This commit is contained in:
volt committed 2026-10-20 14:51:39 +00:00
1 parent 723a852dcb
commit 9b2be3915e
5 files changed
+584 -10
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

WORK-QUEUE.md
+15
View File
@@ -10278,3 +10278,18 @@ Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-index-stale/` (`REPORT.m
- **Nothing operational.** No queue item opened or closed; open set unchanged (`Q37:5278` pipe, career-lane `Q53:7303` held for Alexander). Tracked files changed: `tools/doc_census_check.py`, this append, the index re-render, and this lane's own day's check-in (third pass). No another-party file edited, no secret printed, no issue/PR/release/deploy surface. Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-20-wrap-land/` (`REPORT.md` + correction, `check.sh`, `files.sha256`, `before/`, `after/`).
— Volt
**Volt tick 2026-10-20 14:5xZ — the instruments get a map, and the map is generated: every cell is derived from the tool it describes, so it cannot say what the tree does not.** (Clock read immediately before this edit — 14:51Z; ledger's last line at `495a591` was `— Volt`; guard run and index re-rendered in this same commit.)
- **The gap, stated as a count.** `tools/` holds seventeen scripts and no document states the set. The nearest thing is `verify.py`'s `LEGS` table, which names **five** of them (`checkin`, `reply-headings`, `index-current`, `append-audit`, `selftest-sweep`). A reader meets the other twelve one queue block at a time. A hand-written map would be the fourth notation of the defect this suite grades three times already — prose one step from the thing it describes, with no reader — so `docs/TOOLS.md` is **rendered**, not written: `tools/tool_map.py` derives every cell and `--check` re-derives it byte for byte.
- **The four derived columns, each from its own source.** *What it grades* — the module docstring's first sentence, with the first **paragraph** flattened before the cut, because the purpose sentence routinely wraps and a line-local cut is the defect `doc_census_check`'s wrap reader was given in the previous tick. *Flags* — the tool's own `--help` at `--rev`, run from a temporary copy of the blob, cut to the **`usage:` paragraph only**: argparse prints the module docstring above the usage line, and those docstrings discuss flags they do not accept — `citation_hash_class.py` names `--batch-all-objects`, `--objects`, `--unreachable` and `--no-reflogs` in prose, and the first render listed all four as if they were its interface. Measured on the fix: that cell went from 16 tokens to 12, and the four that left are exactly the prose mentions. *Probes* — `selftest_sweep.py`'s three tables, read with `ast.literal_eval`, never copied. *Gate leg* — `verify.py`'s `LEGS`, likewise.
- **What it reads at the commit that carries it.** `docs/TOOLS.md` **5,175 B / 17 tools / 17 with a `--help` oracle / 5 gate legs**, every row a one-line claim about one script.
- **The drift guard is a leg, not a convention.** The selftest's last cell re-renders the map at `HEAD` and compares bytes, and `selftest_sweep.py` runs every tool's `--selftest` — so a commit that changes a tool's purpose or its flags and does not re-render turns the sweep **RED**. That is the point of shipping the document inside the instrument rather than beside it.
- **RED/GREEN, on committed mutants in a scratch clone (nothing in this repo was mutated).** `check_lint.py` given a new `--mutant-flag` and a new sentence, map not re-rendered → `--check` **rc 1**, `DRIFT at line 17` printing both rows; `--selftest` **15 of 16**, the currency cell the one that failed. The same clone **before** the mutants → `--check` **rc 0**, `--selftest` **16 of 16** — so "current" is a reachable answer produced by the same comparison, not a detector that always fires. A **second** mutant adds the flag and touches no docstring: the flags cell grows `--mutant-flag` while the purpose cell does not move, which is the two columns being graded independently rather than one standing in for the other.
- **One honest limit of the guard, measured rather than asserted.** The map is derived at a revision, so an **uncommitted** worktree edit to a tool drifts nothing: the mutant above read **rc 0** in the worktree and **rc 1** only once committed. That is the revision discipline every other instrument here uses (`doc_census_check` reads docstrings at `HEAD`, `doc_command_check` adjudicates a command against the tool at `--rev`), stated here because a reader could otherwise expect a dirty worktree to show.
- **Instruments, at the carrying commit.** `--selftest` **16 cells, 16 passed, 0 failed**. `selftest_sweep.py` **17 tools, 17 OK, 0 fatal**; read-refusal **22 probes, 22 ok** (21 before), malformed-input **20 probes, 20 ok** (19 before) — the two added rows are this tool's own refusal contract, not an imposed one: `--repo` a path that does not exist → **rc 2** `cannot read repo`, and a directory that is not a git work tree → **rc 2** `cannot read repo: … is not a git work tree`. `doc_command_check.py` **50 commands over 15 oracles, OK 50** (46 over 14 before — this tool's four documented commands and its oracle join the scan). `doc_census_check.py --tools tools/tool_map.py --fail` **1 claim, `MATCHES`, rc 0** (the `16 cells` usage line).
- **The gate, and the fault in it that is not mine.** `verify.py --repo .` at this commit: **5 legs, 4 pass, 1 fault, rc 1** — `checkin` rc 1 on the same two files (`2026-10-19-zephyrus.md` `DUPLICATE`; `2026-10-19-zephyrus-sovereignty.md` `BAD_NAME,DUPLICATE`). A tree carrying none of this change reads the same two faults, measured in the same scratch clone at its parent, so the leg was red before this tick and stays the check-in ledger's to close. `reply-headings`, `index-current`, `append-audit` and `selftest-sweep` (17 tools, 17 OK) all pass.
- **What it does not do.** No leg was added to `verify.py`: that would move every reading of "five legs" in this record to six, so the guard rides `selftest-sweep` instead, which already runs every tracked tool. It reads module docstrings only, so a function-level summary is out of scope. It uses `--help` to *enumerate* flags and never grades the help text as prose; a tool that parses argv by hand is marked `no --help oracle` in its cell and is not accused of anything. It does not check that a flag does what its name says.
- **Nothing operational.** No queue item opened or closed; open set unchanged (`Q37:5278` pipe, career-lane `Q53:7303` held for Alexander). Tracked files changed: `tools/tool_map.py` (new), `tools/selftest_sweep.py` (two probe rows, one word in a comment), `docs/TOOLS.md` (new, generated), this append, the index re-render. No another-party file, no secret printed, no issue/PR/release/deploy surface. Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-20-tool-map/`.
— Volt
docs/TOOLS.md
+48
View File
@@ -0,0 +1,48 @@
# The instruments
Generated by `tools/tool_map.py` from the tree it is asked to read. Do not
edit by hand: `python3 tools/tool_map.py --repo . --write docs/TOOLS.md`
re-renders it, and `--check` grades it (rc 1 when the document and the tree
disagree). Every cell below is derived, never typed -- purpose from the
module docstring, flags from the tool's own `--help`, probe rows from
`selftest_sweep.py`'s tables, the gate leg from `verify.py`'s `LEGS`.
**17 tools, 17 with a `--help` oracle.** Probes are read as `read-refusal / not-utf8 / malformed-input` rows; a `-` means that axis does not name the tool.
| tool | what it grades | flags | probes | gate leg |
|---|---|---|---|---|
| `append_only_audit.py` | measure the repo-wide append-only invariant from git objects. | `--repo` `--json` `--selftest` `--progress` | 1 / 0 / 1 | - |
| `append_only_owners.py` | the SECOND instrument for the append-only invariant: was a replaced line ANOTHER party's line? | `--json` `--selftest` | 1 / 1 / 2 | - |
| `census_row_delta.py` | row-level diff of two doc_census_check.py censuses. | `--key` `--keys` `--verbose` `--selftest` `--json` | 1 / 1 / 2 | - |
| `check_lint.py` | grade a Ralph-loop `CHECK:` line before it is run. | `--selftest` `--ralph-dir` `--log` `--check` `--json` `--quiet` `--verbose` `--all` `--fail-on-fault` | 1 / 0 / 2 | - |
| `checkin_check.py` | the gate on the check-in contract, which nothing measured. | `--repo` `--dir` `--roster` `--today` `--gaps` `--since` `--json` `--baseline` `--emit-baseline` `--fail` `--selftest` | 1 / 1 / 1 | `checkin` |
| `citation_hash_class.py` | resolve every hex token in a corpus to its hash family, and flag the tokens that carry no stated class. | `--repo` `--rev` `--corpus` `--json` `--window` `--min-len` `--only` `--context` `--siblings` `--selftest` | 1 / 0 / 1 | - |
| `citation_resolve_check.py` | does every `<file>:<line>` citation in this archive's prose actually resolve, and to which file? | `--repo` `--rev` `--corpus` `--json` `--only` `--context` `--selftest` | 1 / 0 / 1 | - |
| `doc_census_check.py` | does a tool's own docstring census match the census that tool PRINTS? | `--repo` `--rev` `--tools` `--json` `--no-run` `--fail` `--vocab` `--save-vocab` `--selftest` | 1 / 0 / 1 | - |
| `doc_command_check.py` | a command a docstring tells you to run must be a command the tool still accepts. | `--repo` `--rev` `--tools` `--json` `--fail` `--selftest` | 1 / 0 / 1 | - |
| `landing_check.py` | is every file at this revision actually served by the forge? | `--repo` `--rev` `--branch` `--base-url` `--path` `--max-bytes` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
| `queue_append_audit.py` | was the queue's index current IN THE TREE of every commit that touched the queue? | `--repo` `--queue` `--index` `--from` `--to` `--all` `--guard` `--timeout` `--json` `--selftest` | 1 / 0 / 1 | `append-audit` |
| `queue_claim_check.py` | the claim check for `WORK-QUEUE.md`, shipped with its tests. | `--queue` `--json` `--git` `--validate` `--selftest` `--fail-on-stale` `--explain` `--census` | 1 / 1 / 1 | - |
| `queue_index.py` | the work queue's table of contents, and the gate that it is current. | `--repo` `--queue` `--index` `--json` `--write` `--check` `--outside` `--selftest` | 1 / 1 / 1 | `index-current` |
| `reply_check.py` | does every reply file's own header agree with its own name? | `--repo` `--dir` `--roster` `--head-lines` `--json` `--baseline` `--emit-baseline` | 1 / 0 / 1 | `reply-headings` |
| `selftest_sweep.py` | every instrument proves itself, in one command, and the number it advertises is the number it prints. | `--repo` `--tools-dir` `--timeout` `--json` `--quiet` `--verbose` `--selftest` | 1 / 0 / 1 | `selftest-sweep` |
| `tool_map.py` | the instruments get a map, and the map is generated from the tree. | `--repo` `--rev` `--write` `--check` `--json` `--selftest` | 1 / 0 / 1 | - |
| `verify.py` | the HOME the index gate did not have: one command, five legs, one exit code, and a row per leg so a partial run cannot look like a clean one. | `--repo` `--queue` `--index` `--checkin-day` `--fast` `--timeout` `--json` `--quiet` `--selftest` | 1 / 0 / 1 | - |
## The gate
`python3 tools/verify.py --repo .` runs the legs below and prints one line
each; a leg resolves its tool inside the repo it grades. The `slow` legs are
dropped by `--fast`.
| leg | tool | slow | what it grades |
|---|---|---|---|
| `checkin` | `checkin_check.py` | no | grade the newest day's check-ins against the naming and duplicate rules |
| `reply-headings` | `reply_check.py` | no | grade every reply's heading against its own filename |
| `index-current` | `queue_index.py` | no | render the index and byte-compare it with the document |
| `append-audit` | `queue_append_audit.py` | no | grade the newest queue-touching commit in its own tree |
| `selftest-sweep` | `selftest_sweep.py` | yes | every tool's --selftest and both read-refusal axes |
A tool that appears in no leg is still graded by `selftest-sweep`, which runs
every tracked tool's `--selftest` and both read-refusal axes -- so the
`probes` column above is also the sweep's cover of that tool.
docs/queue-index.md
+9 -9
View File
@@ -56,7 +56,7 @@
| 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot |
| 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree |
| 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 |
| 49 | Q45 | 7594-10280 | 455748 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | -- |
| 49 | Q45 | 7594-10295 | 461809 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | -- |
## Ids with no `## ` heading
@@ -71,7 +71,7 @@ Opened as records inside another section. `line` is the record's own opening lin
| Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q53 | 7292 | 7246 | 138 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q53 | 7292 | 7246 | 139 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD |
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
@@ -217,7 +217,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
"form": "BOLD-LEAD",
"id": "Q53",
"line": 7292,
"refs": 138,
"refs": 139,
"section": "Q49",
"shape": "BULLET+BOLD"
},
@@ -236,9 +236,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
"end_line": 8,
"line": 1
},
"queue_bytes": 1348009,
"queue_lines": 10280,
"queue_sha256": "cd51315f3a4ee29ccbbf915dad2c79771c5ed49ecbe145fa431777ebe253c5e0",
"queue_bytes": 1354070,
"queue_lines": 10295,
"queue_sha256": "2efbcb8224a3f0d96d24296fe0af9668926bb11a2b85f940831a5d55dc0b1bc6",
"sections": [
{
"bytes": 3139,
@@ -817,11 +817,11 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "The runtime-resolved root's live write path: name it, reproduce it, then close it"
},
{
"bytes": 455748,
"end_line": 10280,
"bytes": 461809,
"end_line": 10295,
"id": "Q45",
"line": 7594,
"lines": 2687,
"lines": 2702,
"role": "opening",
"role_detail": "",
"status": null,
tools/selftest_sweep.py
+4 -1
View File
@@ -169,6 +169,7 @@ READ_PROBES = (
("queue_append_audit.py", ["--repo", "{nope}"]),
("reply_check.py", ["--dir", "{nope}"]),
("selftest_sweep.py", ["--repo", "{nope}"]),
("tool_map.py", ["--repo", "{nope}"]),
("verify.py", ["--repo", "{nope}"])
)
NOT_UTF8_PROBES = (
@@ -182,7 +183,7 @@ NOT_UTF8_PROBES = (
# THE THIRD CLASS (2026-10-18): input that decodes as UTF-8 but cannot be made
# sense of. Each entry carries the answer the TOOL's own contract states --
# (tool, label, argv, expected rc, a substring the output must contain) -- so
# the axis measures the documented behaviour instead of imposing one. Thirteen
# the axis measures the documented behaviour instead of imposing one. Fourteen
# tools must refuse at rc 2; the three that grade a malformed input are listed
# with the rc and the word their grade prints, which is what makes a silent
# rc 0 (the doc_census_check --tools bug class) fail here.
@@ -230,6 +231,8 @@ MALFORMED_PROBES = (
2, "not a directory"),
("selftest_sweep.py", "not-a-repo", ["--repo", "{notrepo}"],
2, "no tools discovered"),
("tool_map.py", "not-a-repo", ["--repo", "{notrepo}"],
2, "cannot read repo"),
# verify.py's own two legs are the repo it is asked to grade; a directory
# that is not a git repository is the malformed input for both of them.
("verify.py", "not-a-repo", ["--repo", "{notrepo}"],
tools/tool_map.py
+508
View File
@@ -0,0 +1,508 @@
#!/usr/bin/env python3
"""tool_map.py -- the instruments get a map, and the map is generated from the tree.
WHY THIS EXISTS
`tools/` holds sixteen scripts and there is no document that says what they
are. A reader meets them one at a time -- in a `verify.py` leg, in a queue
block, in a reply -- and nothing states the whole set: which tool grades what,
which flags each one actually accepts, which of them the two probe axes cover,
and which leg of the operator's gate a failure shows up in. The closest thing
is `verify.py`'s `LEGS` table, which names five of the sixteen.
A hand-written map would be a fourth notation of the defect this suite keeps
grading: prose about the tree, one step from the thing it describes, with no
reader. So this document is not written, it is DERIVED -- every cell comes
from the artefact it describes, at a stated revision:
what it grades the module docstring's first line, after `<name> -- `
flags the tool's own `--help`, run from a temporary copy of the
blob at `--rev` (the only statement of an interface that
cannot drift from the interface)
probes `selftest_sweep.py`'s READ_PROBES / NOT_UTF8_PROBES /
MALFORMED_PROBES tables, read with `ast.literal_eval`
gate leg `verify.py`'s `LEGS` table, likewise
Nothing is inferred and nothing is typed by hand, so the document cannot say
something the tree does not: `--check` re-derives it and compares bytes.
WHAT IT MEASURES -- one verdict per invocation
CURRENT the document on disk is byte-identical to the render
DRIFT the document differs; the first differing line is printed. A
FAULT: the map is a claim about the tree and it is no longer true.
UNREADABLE the document does not exist, or the revision will not
materialise, or a tool's `--help` outlived its timeout. rc 2,
never rc 1 -- an ungradable map is not a stale one.
`--selftest` ends with the currency cell: it re-renders `docs/TOOLS.md` at
HEAD and compares. That cell is the reason the map stays true -- the tool
sweep runs every tool's `--selftest`, so a commit that changes a tool's
docstring, flags or probe rows without re-rendering the map turns the sweep
RED instead of leaving a stale document that reads as current.
READ-ONLY except for the one path it is told to write. The only process it
starts is `<tool> --help`, which does no work by construction; it never runs a
documented command, so it cannot write, post, or deploy anything.
EXIT CODES (the tools/ contract)
0 the map is current (or was written)
1 the map has drifted from the tree
2 could not grade: not a git work tree, a revision that will not
materialise, or a document that cannot be read
USAGE
python3 tool_map.py --repo . --write docs/TOOLS.md # render
python3 tool_map.py --repo . --check docs/TOOLS.md # grade drift, rc 1 on drift
python3 tool_map.py --repo . --json out.json
python3 tool_map.py --selftest # 16 cells
"""
from __future__ import annotations
import argparse
import ast
import json
import os
import re
import subprocess
import sys
import tempfile
DEFAULT_REPO = "/root/troupe-standup"
DEFAULT_DOC = "docs/TOOLS.md"
HELP_TIMEOUT = 30
# A long option, or a short one, as argparse prints them in the usage block.
# `-h`/`--help` are dropped: every tool has them, and a column of sixteen
# identical pairs is noise, not information.
OPT_RE = re.compile(r"(?<![\w-])(--[a-z0-9][a-z0-9-]*|-[A-Za-z])(?![\w-])")
SKIP_OPTS = {"-h", "--help"}
def git(repo, *args, inp=None):
r = subprocess.run(["git", "-C", repo, *args], input=inp,
capture_output=True, text=True)
return r.returncode, r.stdout, r.stderr
def read_blob(repo, rev, path):
rc, out, err = git(repo, "show", f"{rev}:{path}")
return out if rc == 0 else None
def tracked_tools(repo, rev):
"""Sorted `tools/*.py` basenames tracked at `rev`."""
rc, out, _ = git(repo, "ls-tree", "-r", "--name-only", rev, "--", "tools/")
if rc != 0:
return None
names = [p.split("/")[-1] for p in out.split("\n")
if p.startswith("tools/") and p.endswith(".py")]
return sorted(names)
def literal_assignments(repo, rev, path, names):
"""{name: value} for the module-level assignments whose value is a literal."""
src = read_blob(repo, rev, path)
if src is None:
return {}
try:
tree = ast.parse(src)
except SyntaxError:
return {}
found = {}
for node in tree.body:
if not isinstance(node, ast.Assign) or len(node.targets) != 1:
continue
tgt = node.targets[0]
if not isinstance(tgt, ast.Name) or tgt.id not in names:
continue
try:
found[tgt.id] = ast.literal_eval(node.value)
except (ValueError, SyntaxError):
continue
return found
def usage_block(text):
"""Only the `usage:` paragraph of a `--help` page.
argparse prints the module docstring as the description ABOVE the usage
line, and those docstrings discuss flags in prose -- `citation_hash_class.py`
mentions `--batch-all-objects` while explaining why it does not use it. A
flag that prose mentions is not a flag the tool accepts, so the oracle is
cut to the usage paragraph and stop: it ends at the first blank line after
the line that opens with `usage:`. No usage line means no oracle.
"""
lines = text.splitlines()
start = None
for i, line in enumerate(lines):
if line.lower().startswith("usage:"):
start = i
break
if start is None:
return ""
block = []
for line in lines[start:]:
if not line.strip():
break
block.append(line)
return "\n".join(block)
def purpose_of(src):
"""The module docstring's first sentence as a one-clause summary.
The house shape is `<name> -- <what it grades>.`; documents written without
the separator fall back to the docstring's opening sentence. The whole first
PARAGRAPH is flattened before the sentence cut, because the sentence that
states a tool's purpose routinely wraps across two or three source lines --
a line-local cut is what this suite grades as a defect everywhere else.
Returns "" when there is no module docstring, which the row prints as `-`.
"""
if not src:
return ""
try:
doc = ast.get_docstring(ast.parse(src)) or ""
except SyntaxError:
return ""
first_para = re.split(r"\n\s*\n", doc.strip())[0] if doc.strip() else ""
flat = " ".join(line.strip() for line in first_para.splitlines()).strip()
if not flat:
return ""
flat = re.sub(r"^[a-z0-9_]+\.[a-z]+ *(?:--|—|–|-) *", "", flat).strip()
m = re.search(r"^(.+?[.?!])(?:\s|$)", flat)
out = (m.group(1) if m else flat).rstrip()
if len(out) > 180:
out = out[:177].rstrip() + "..."
return out
def flags_of(repo, rev, name):
"""The tool's accepted options, from its own `--help`.
The blob is written to a temporary directory and run there so a tool is
adjudicated at `--rev` against the parser it will actually use, never
against a worktree copy. Returns (flags, note) where note is "" when the
oracle answered and a short reason when it did not.
"""
src = read_blob(repo, rev, f"tools/{name}")
if src is None:
return [], "no such file at rev"
with tempfile.TemporaryDirectory() as td:
path = os.path.join(td, name)
with open(path, "w", encoding="utf-8") as fh:
fh.write(src)
try:
proc = subprocess.run([sys.executable, path, "--help"],
capture_output=True, text=True,
timeout=HELP_TIMEOUT, cwd=td)
except subprocess.TimeoutExpired:
return [], "help timed out"
text = (proc.stdout or "") + (proc.stderr or "")
usage = usage_block(text)
seen, out = set(), []
for m in OPT_RE.finditer(usage):
opt = m.group(1)
if opt in SKIP_OPTS or opt in seen:
continue
seen.add(opt)
out.append(opt)
if not out:
return [], "no --help oracle"
return out, ""
def collect(repo, rev):
"""The whole map as data: one dict per tool plus the gate's legs."""
names = tracked_tools(repo, rev)
if names is None:
raise SystemExit(f"tool_map: cannot read repo: {repo} at {rev}")
sweep = literal_assignments(repo, rev, "tools/selftest_sweep.py",
{"READ_PROBES", "NOT_UTF8_PROBES",
"MALFORMED_PROBES"})
read_n, utf8_n, mal_n = {}, {}, {}
for name, _argv in sweep.get("READ_PROBES", ()): # (tool, argv)
read_n[name] = read_n.get(name, 0) + 1
for name, _argv in sweep.get("NOT_UTF8_PROBES", ()):
utf8_n[name] = utf8_n.get(name, 0) + 1
for name, *_rest in sweep.get("MALFORMED_PROBES", ()): # (tool, label, ...)
mal_n[name] = mal_n.get(name, 0) + 1
legs = literal_assignments(repo, rev, "tools/verify.py", {"LEGS"}).get("LEGS", ())
leg_of = {}
for leg in legs:
if isinstance(leg, dict) and "tool" in leg and "key" in leg:
leg_of.setdefault(leg["tool"], []).append(leg["key"])
rows = []
for name in names:
src = read_blob(repo, rev, f"tools/{name}")
flags, note = flags_of(repo, rev, name)
rows.append({
"tool": name,
"purpose": purpose_of(src),
"flags": flags,
"no_oracle": note,
"read": read_n.get(name, 0),
"not_utf8": utf8_n.get(name, 0),
"malformed": mal_n.get(name, 0),
"legs": leg_of.get(name, []),
})
return {"rev": rev, "rows": rows,
"legs": [{"key": l.get("key"), "tool": l.get("tool"),
"slow": bool(l.get("slow")), "what": l.get("what", "")}
for l in legs if isinstance(l, dict)]}
def render(data):
"""The document, byte-deterministic for a given tree."""
rows, legs = data["rows"], data["legs"]
with_oracle = sum(1 for r in rows if not r["no_oracle"])
lines = [
"# The instruments",
"",
"Generated by `tools/tool_map.py` from the tree it is asked to read. Do not",
"edit by hand: `python3 tools/tool_map.py --repo . --write docs/TOOLS.md`",
"re-renders it, and `--check` grades it (rc 1 when the document and the tree",
"disagree). Every cell below is derived, never typed -- purpose from the",
"module docstring, flags from the tool's own `--help`, probe rows from",
"`selftest_sweep.py`'s tables, the gate leg from `verify.py`'s `LEGS`.",
"",
f"**{len(rows)} tools, {with_oracle} with a `--help` oracle.** "
f"Probes are read as `read-refusal / not-utf8 / malformed-input` rows; a "
f"`-` means that axis does not name the tool.",
"",
"| tool | what it grades | flags | probes | gate leg |",
"|---|---|---|---|---|",
]
for r in rows:
purpose = r["purpose"] or "-"
if r["no_oracle"]:
flags = f"- *({r['no_oracle']})*"
else:
flags = " ".join(f"`{f}`" for f in r["flags"]) or "-"
probes = (f"{r['read']} / {r['not_utf8']} / {r['malformed']}"
if (r["read"] or r["not_utf8"] or r["malformed"]) else "-")
gate = ", ".join(f"`{k}`" for k in r["legs"]) or "-"
lines.append(f"| `{r['tool']}` | {purpose} | {flags} | {probes} | {gate} |")
lines += [
"",
"## The gate",
"",
"`python3 tools/verify.py --repo .` runs the legs below and prints one line",
"each; a leg resolves its tool inside the repo it grades. The `slow` legs are",
"dropped by `--fast`.",
"",
"| leg | tool | slow | what it grades |",
"|---|---|---|---|",
]
for leg in legs:
lines.append(f"| `{leg['key']}` | `{leg['tool']}` | "
f"{'yes' if leg['slow'] else 'no'} | {leg['what']} |")
lines += [
"",
"A tool that appears in no leg is still graded by `selftest-sweep`, which runs",
"every tracked tool's `--selftest` and both read-refusal axes -- so the",
"`probes` column above is also the sweep's cover of that tool.",
"",
]
return "\n".join(lines)
def selftest(repo):
cells = []
def cell(label, ok, detail=""):
cells.append({"label": label, "ok": bool(ok), "detail": detail})
# -- the derivation, on fixtures -------------------------------------
cell("purpose: house `<name> -- ...` shape is stripped",
purpose_of('"""fake_tool.py -- grades the fake contract, at one line.\n\n'
'More prose here."""\n') ==
"grades the fake contract, at one line.")
cell("purpose: no separator falls back to the first sentence",
purpose_of('"""A map of the tree. Second sentence follows."""\n') ==
"A map of the tree.")
cell("purpose: absent docstring is empty, not invented",
purpose_of("import os\n") == "")
cell("flags: `-h`/`--help` are dropped, real options kept",
[o for o in OPT_RE.findall("usage: t.py [-h] [--repo R] [--rev V]")
if o not in SKIP_OPTS] == ["--repo", "--rev"])
cell("flags: a positional-looking token is not an option",
OPT_RE.findall("usage: t.py [-h] [--repo REPO] filename") ==
["-h", "--repo"])
cell("flags: a flag the description only MENTIONS is not an oracle",
usage_block("desc mentions --nope\n\noptions:\n") == "")
with tempfile.TemporaryDirectory() as td:
fixture = os.path.join(td, "fx")
os.makedirs(os.path.join(fixture, "tools"))
os.makedirs(os.path.join(fixture, "docs"))
env = dict(os.environ, GIT_AUTHOR_NAME="t", GIT_AUTHOR_EMAIL="t@e",
GIT_COMMITTER_NAME="t", GIT_COMMITTER_EMAIL="t@e")
subprocess.run(["git", "init", "-q", fixture], check=True, env=env)
open(os.path.join(fixture, "tools", "fake_tool.py"), "w").write(
'#!/usr/bin/env python3\n'
'"""fake_tool.py -- grades the fake contract, which nothing measured.\n\n'
'Prose."""\n'
'import argparse\n'
'ap = argparse.ArgumentParser()\n'
'ap.add_argument("--repo")\n'
'ap.add_argument("--fail", action="store_true")\n'
'if "--help" in __import__("sys").argv:\n'
' ap.print_help()\n')
open(os.path.join(fixture, "tools", "quiet_tool.py"), "w").write(
'#!/usr/bin/env python3\n'
'"""quiet_tool.py -- parses argv by hand, so it prints no usage."""\n'
'import sys\n'
'print("ran")\n')
open(os.path.join(fixture, "tools", "selftest_sweep.py"), "w").write(
'READ_PROBES = (("fake_tool.py", ["--repo", "{nope}"]),)\n'
'NOT_UTF8_PROBES = ()\n'
'MALFORMED_PROBES = (("fake_tool.py", "not-a-repo",\n'
' ["--repo", "{notrepo}"], 2, "cannot read"),)\n')
open(os.path.join(fixture, "tools", "verify.py"), "w").write(
'LEGS = ({"key": "fake", "tool": "fake_tool.py", "slow": False,\n'
' "what": "grade the fake contract"},)\n')
def commit(msg):
subprocess.run(["git", "-C", fixture, "add", "-A"], check=True, env=env)
subprocess.run(["git", "-C", fixture, "commit", "-qm", msg],
check=True, env=env)
commit("fixture")
data = collect(fixture, "HEAD")
by = {r["tool"]: r for r in data["rows"]}
cell("collect: every tracked .py is a row, oracle or not",
len(data["rows"]) == 4 and by["fake_tool.py"]["no_oracle"] == ""
and by["quiet_tool.py"]["no_oracle"] == "no --help oracle",
str([(r["tool"], r["no_oracle"]) for r in data["rows"]]))
cell("collect: probe rows and the gate leg are joined by name",
by["fake_tool.py"]["read"] == 1
and by["fake_tool.py"]["malformed"] == 1
and by["fake_tool.py"]["legs"] == ["fake"],
str({k: (v["read"], v["malformed"], v["legs"]) for k, v in by.items()}))
cell("collect: only printed flags are listed, and --help is not",
by["fake_tool.py"]["flags"] == ["--repo", "--fail"],
str(by["fake_tool.py"]["flags"]))
doc = os.path.join(fixture, "docs", "TOOLS.md")
text = render(data)
open(doc, "w").write(text)
cell("render: deterministic (two renders of one tree are identical)",
text == render(collect(fixture, "HEAD")))
cell("render: the tool without an oracle is marked, not accused",
"quiet_tool.py" in text and "no --help oracle" in text)
rc_ok = compare(doc, text)
cell("--check: rc 0 on the document the same tree renders", rc_ok == 0,
f"rc {rc_ok}")
open(doc, "a").write("stale line\n")
rc_drift = compare(doc, render(collect(fixture, "HEAD")), quiet=True)
cell("--check: rc 1 once a byte drifts",
rc_drift == 1, f"rc {rc_drift}")
open(doc, "w").write(text)
rc_missing = compare(os.path.join(fixture, "docs", "nope.md"), text,
quiet=True)
cell("--check: rc 2, not rc 1, when the document does not exist",
rc_missing == 2, f"rc {rc_missing}")
rc_norepo = graderc(td)
cell("--repo a directory that is not a git work tree: rc 2",
rc_norepo == 2, f"rc {rc_norepo}")
# -- the currency cell: the reason this document stays true ----------
here = os.path.join(repo, "docs", "TOOLS.md")
try:
want = render(collect(repo, "HEAD"))
rc = compare(here, want, quiet=True)
except SystemExit as exc:
rc = 2
cell("HEAD: the shipped map renders", False, str(exc))
cell("HEAD: docs/TOOLS.md is the render of this tree (the drift guard)",
rc == 0, f"rc {rc}")
passed = sum(1 for c in cells if c["ok"])
for c in cells:
print(f" {'ok ' if c['ok'] else 'FAIL'} {c['label']}"
+ (f" [{c['detail']}]" if c["detail"] and not c["ok"] else ""))
print(f"selftest: {len(cells)} cells, {passed} passed, {len(cells) - passed} failed")
return 0 if passed == len(cells) else 1
def graderc(td):
"""rc of a full `--check` run against a directory that is not a repo."""
proc = subprocess.run([sys.executable, os.path.abspath(__file__),
"--repo", td, "--check", os.path.join(td, "x.md")],
capture_output=True, text=True)
return proc.returncode
def compare(path, want, quiet=False):
"""0 current, 1 drift (first differing line printed), 2 unreadable."""
try:
with open(path, encoding="utf-8") as fh:
have = fh.read()
except OSError as exc:
if not quiet:
print(f"tool_map: cannot read document: {exc}")
return 2
if have == want:
return 0
if not quiet:
a, b = have.splitlines(), want.splitlines()
for i in range(max(len(a), len(b))):
got = a[i] if i < len(a) else "<end of document>"
exp = b[i] if i < len(b) else "<end of document>"
if got != exp:
print(f"DRIFT at line {i + 1}:")
print(f" document: {got}")
print(f" tree: {exp}")
break
return 1
def main():
ap = argparse.ArgumentParser(
description="Render the map of tools/ from the tree, and grade its drift.")
ap.add_argument("--repo", default=DEFAULT_REPO)
ap.add_argument("--rev", default="HEAD")
ap.add_argument("--write", metavar="PATH",
help=f"render the map to PATH (default target {DEFAULT_DOC})")
ap.add_argument("--check", metavar="PATH",
help="grade PATH against a fresh render; rc 1 on drift")
ap.add_argument("--json", metavar="PATH", help="write the collected data as JSON")
ap.add_argument("--selftest", action="store_true")
args = ap.parse_args()
if args.selftest:
return selftest(args.repo)
rc, _, _ = git(args.repo, "rev-parse", "--show-toplevel")
if rc != 0:
print(f"tool_map: cannot read repo: {args.repo} is not a git work tree")
return 2
rc, _, err = git(args.repo, "rev-parse", "--verify", f"{args.rev}^{{commit}}")
if rc != 0:
print(f"tool_map: cannot read rev {args.rev}: {err.strip().splitlines()[0] if err.strip() else 'no such object'}")
return 2
data = collect(args.repo, args.rev)
text = render(data)
if args.json:
with open(args.json, "w", encoding="utf-8") as fh:
json.dump(data, fh, indent=2, sort_keys=True)
fh.write("\n")
print(f"tool_map: wrote {args.json}")
if args.write:
with open(args.write, "w", encoding="utf-8") as fh:
fh.write(text)
print(f"tool_map: wrote {args.write} ({len(text)} B, "
f"{len(data['rows'])} tools, {len(data['legs'])} gate legs)")
return 0
if args.check:
return compare(args.check, text)
print(text, end="")
return 0
if __name__ == "__main__":
sys.exit(main())