772 lines
35 KiB
Python
772 lines
35 KiB
Python
#!/usr/bin/env python3
|
||
"""tool_map.py -- the instruments get a map, and the map is generated from the tree.
|
||
|
||
WHY THIS EXISTS
|
||
`tools/` holds one script per instrument and there is no document that says
|
||
what they are. A reader meets them one at a time -- in a `verify.py` leg, in a
|
||
queue block, in a reply -- and nothing states the whole set: which tool grades
|
||
what, which flags each one actually accepts, which of them the two probe axes
|
||
cover, and which leg of the operator's gate a failure shows up in. The closest
|
||
thing is `verify.py`'s `LEGS` table, which names some of them and never all;
|
||
a leg count written here would be one more number to drift.
|
||
|
||
A hand-written map would be a fourth notation of the defect this suite keeps
|
||
grading: prose about the tree, one step from the thing it describes, with no
|
||
reader. So this document is not written, it is DERIVED -- every cell comes
|
||
from the artefact it describes, at a stated revision:
|
||
|
||
what it grades the module docstring's first line, after `<name> -- `
|
||
flags the tool's own `--help`, run from a temporary copy of the
|
||
blob at `--rev` (the only statement of an interface that
|
||
cannot drift from the interface)
|
||
probes `selftest_sweep.py`'s READ_PROBES / NOT_UTF8_PROBES /
|
||
MALFORMED_PROBES tables, read with `ast.literal_eval`
|
||
gate leg `verify.py`'s `LEGS` table, likewise
|
||
|
||
Nothing is inferred and nothing is typed by hand, so the document cannot say
|
||
something the tree does not: `--check` re-derives it and compares bytes.
|
||
|
||
WHAT IT MEASURES -- one verdict per invocation
|
||
CURRENT the document on disk is byte-identical to the render
|
||
DRIFT the document differs; the first differing line is printed. A
|
||
FAULT: the map is a claim about the tree and it is no longer true.
|
||
UNREADABLE the document does not exist, or the revision will not
|
||
materialise, or a tool's `--help` outlived its timeout. rc 2,
|
||
never rc 1 -- an ungradable map is not a stale one.
|
||
|
||
`--selftest` ends with the currency cell: it re-renders `docs/TOOLS.md` at
|
||
HEAD and compares. That cell is the reason the map stays true -- the tool
|
||
sweep runs every tool's `--selftest`, so a commit that changes a tool's
|
||
docstring, flags or probe rows without re-rendering the map turns the sweep
|
||
RED instead of leaving a stale document that reads as current.
|
||
|
||
THE PRE-COMMIT BLIND SPOT, AND THE MODE THAT CLOSES IT
|
||
Every cell above is derived from a REVISION, not from the working tree:
|
||
`collect()` reads blobs with `git show <rev>:<path>` and runs `--help` from a
|
||
copy of that blob. That is right for a document that makes a claim about a
|
||
commit, and wrong for the ten minutes before a commit exists.
|
||
|
||
Measured on this repository, 2026-10-21. `4602bd5` is drifted: its commit
|
||
message states "TOOLS.md re-rendered, byte-identical" and the map it ships is
|
||
not the render of its own tree. Two runs, both from inside a worktree checked
|
||
out at that commit: `--repo . --check docs/TOOLS.md` reads **rc 1, DRIFT at
|
||
line 49** (the `write-axis` cell, whose description moved in `verify.py`),
|
||
and `--repo . --selftest` reads **15/16, rc 1**, the one failure being the
|
||
currency cell below. Before the commit existed both commands read rc 0 and
|
||
16/16, because the document on disk and the tools at `HEAD` (`ea8fe24`) were
|
||
still the pre-edit pair -- the edit lived only in the worktree.
|
||
|
||
The gate did not catch it either. `selftest_sweep.py --repo .` at `4602bd5`
|
||
reads **21 tools, 21 OK, 0 fatal**, because the sweep runs each tool as
|
||
`[python, path, "--selftest"]` with no `--repo` (`selftest_sweep.py:449`) and
|
||
this tool's `DEFAULT_REPO` was an absolute path, so its selftest graded
|
||
/root/troupe-standup -- a different, clean tree. That was a second defect:
|
||
the sweep's selftest axis could grade the wrong tree, and it did.
|
||
|
||
FIXED 2026-10-22 (volt). `DEFAULT_REPO` is now derived from this file's own
|
||
location -- `dirname(dirname(abspath(__file__)))` -- so the default target is
|
||
the checkout the tool lives in: the same path in place, a different path in a
|
||
copy. Measured on a clone at `c3058d5`, whose `docs/TOOLS.md` is drifted
|
||
(`--check` reads **rc 1, DRIFT at line 30**):
|
||
|
||
- before the fix, `--selftest` with no `--repo` -- the sweep's own
|
||
invocation, `cwd` = the clone -- read **20/20, rc 0**: it graded
|
||
/root/troupe-standup, the clean tree, and called the clone's drift clean;
|
||
- the same clone under `--selftest --repo .` read **19/20, rc 1**, the one
|
||
failure being the currency cell below;
|
||
- after the fix, the repo-less run reads **19/20, rc 1** -- the same
|
||
verdict as `--repo .`, because the default now names the copy.
|
||
|
||
`selftest_sweep.py` needed no edit: `cwd` was already the tree under sweep,
|
||
so the fix alone is enough for the sweep's axis to grade what it walks.
|
||
|
||
`--rev WORKTREE` derives the map from the files on disk, discovered through
|
||
`git ls-files -- tools/` -- the index, which is what a commit of tracked
|
||
files will carry. Measured on a clean worktree at `190d1aa`: rc 0, and
|
||
byte-identical to what HEAD renders; with one uncommitted line appended to
|
||
`verify.py`'s `write-axis` description, `--rev HEAD` still reads rc 0 and
|
||
`--rev WORKTREE` reads **rc 1, DRIFT at line 49**. The currency cell stays
|
||
HEAD-scoped by design: a worktree mid-edit is not a fault, and a sweep that
|
||
fails while a docstring is being typed teaches nothing. The pre-commit guard
|
||
is that `--check` is run in WORKTREE mode before the commit; four cells below
|
||
measure that it separates the two trees and that a clean worktree renders
|
||
byte-for-byte what HEAD renders, so the mode cannot invent a drift of its own.
|
||
|
||
READ-ONLY except for the one path it is told to write. The only process it
|
||
starts is `<tool> --help`, which does no work by construction; it never runs a
|
||
documented command, so it cannot write, post, or deploy anything.
|
||
|
||
EXIT CODES (the tools/ contract)
|
||
0 the map is current (or was written)
|
||
1 the map has drifted from the tree
|
||
2 could not grade: not a git work tree, a revision that will not
|
||
materialise, or a document that cannot be read
|
||
|
||
USAGE
|
||
python3 tool_map.py --repo . --write docs/TOOLS.md # render
|
||
python3 tool_map.py --repo . --check docs/TOOLS.md # grade drift, rc 1 on drift
|
||
python3 tool_map.py --repo . --rev WORKTREE --check docs/TOOLS.md # ...as it will be committed
|
||
python3 tool_map.py --repo . --json out.json
|
||
python3 tool_map.py --selftest # 23 cells
|
||
"""
|
||
from __future__ import annotations
|
||
|
||
import argparse
|
||
import ast
|
||
import json
|
||
import os
|
||
import re
|
||
import subprocess
|
||
import sys
|
||
import tempfile
|
||
|
||
DEFAULT_REPO = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
|
||
DEFAULT_DOC = "docs/TOOLS.md"
|
||
HELP_TIMEOUT = 30
|
||
|
||
# The one value of `--rev` that does not name a revision: the files git would
|
||
# commit from this worktree (the index lists them, the disk supplies them).
|
||
WORKTREE = "WORKTREE"
|
||
|
||
# A long option, or a short one, as argparse prints them in the usage block.
|
||
# `-h`/`--help` are dropped: every tool has them, and a column of sixteen
|
||
# identical pairs is noise, not information.
|
||
OPT_RE = re.compile(r"(?<![\w-])(--[a-z0-9][a-z0-9-]*|-[A-Za-z])(?![\w-])")
|
||
SKIP_OPTS = {"-h", "--help"}
|
||
|
||
|
||
def git(repo, *args, inp=None):
|
||
r = subprocess.run(["git", "-C", repo, *args], input=inp,
|
||
capture_output=True, text=True)
|
||
return r.returncode, r.stdout, r.stderr
|
||
|
||
|
||
def read_blob(repo, rev, path):
|
||
"""The file's text at `rev`; from disk when `rev` is WORKTREE."""
|
||
if rev == WORKTREE:
|
||
try:
|
||
with open(os.path.join(repo, path), encoding="utf-8") as fh:
|
||
return fh.read()
|
||
except OSError:
|
||
return None
|
||
rc, out, err = git(repo, "show", f"{rev}:{path}")
|
||
return out if rc == 0 else None
|
||
|
||
|
||
def tracked_tools(repo, rev):
|
||
"""Sorted `tools/*.py` basenames tracked at `rev`.
|
||
|
||
WORKTREE is not a revision and has no tree object: the set is the INDEX
|
||
(`git ls-files`), which is what a commit of tracked files will carry, and
|
||
the contents come from disk. Reading the index rather than a reflog of the
|
||
worktree keeps one rule -- "the file git would commit" -- for both halves.
|
||
"""
|
||
if rev == WORKTREE:
|
||
rc, out, _ = git(repo, "ls-files", "--", "tools/")
|
||
else:
|
||
rc, out, _ = git(repo, "ls-tree", "-r", "--name-only", rev, "--", "tools/")
|
||
if rc != 0:
|
||
return None
|
||
names = [p.split("/")[-1] for p in out.split("\n")
|
||
if p.startswith("tools/") and p.endswith(".py")]
|
||
return sorted(names)
|
||
|
||
|
||
_UNREADABLE = object()
|
||
|
||
|
||
def partial_literal(node):
|
||
"""`ast.literal_eval`, but a container survives a field that is not literal.
|
||
|
||
`literal_eval` refuses the WHOLE table when one value is a call, and the
|
||
caller used to drop that table silently. `verify.py`'s `LEGS` gained
|
||
`"path": os.path.join(...)` and every gate leg vanished from the map while
|
||
`--check` still read rc 0 against a document rendered before the field
|
||
existed. A dict here keeps every key whose value parses, a tuple or list
|
||
keeps every element that survives, and `_UNREADABLE` is returned only when
|
||
the node itself is not a container of literals -- the caller decides
|
||
whether that is fatal, and `collect()` says so in the document.
|
||
"""
|
||
try:
|
||
return ast.literal_eval(node)
|
||
except (ValueError, SyntaxError):
|
||
pass
|
||
if isinstance(node, (ast.Tuple, ast.List)):
|
||
out = [v for v in (partial_literal(e) for e in node.elts)
|
||
if v is not _UNREADABLE]
|
||
if node.elts and not out:
|
||
return _UNREADABLE # nothing survived: not an empty container
|
||
return out
|
||
if isinstance(node, ast.Dict):
|
||
out = {}
|
||
for k, v in zip(node.keys, node.values):
|
||
if k is None: # {**mapping}
|
||
continue
|
||
kv, vv = partial_literal(k), partial_literal(v)
|
||
if kv is _UNREADABLE or vv is _UNREADABLE:
|
||
continue
|
||
out[kv] = vv
|
||
return out
|
||
return _UNREADABLE
|
||
|
||
|
||
def literal_assignments(repo, rev, path, names):
|
||
"""{name: value} for the module-level assignments whose value is a literal.
|
||
|
||
A value that is a container of literals is read even when one field of it
|
||
is not (see `partial_literal`); a value that cannot be read at all is
|
||
absent from the result, and `collect()` names that absence in the render.
|
||
"""
|
||
src = read_blob(repo, rev, path)
|
||
if src is None:
|
||
return {}
|
||
try:
|
||
tree = ast.parse(src)
|
||
except SyntaxError:
|
||
return {}
|
||
found = {}
|
||
for node in tree.body:
|
||
if not isinstance(node, ast.Assign) or len(node.targets) != 1:
|
||
continue
|
||
tgt = node.targets[0]
|
||
if not isinstance(tgt, ast.Name) or tgt.id not in names:
|
||
continue
|
||
val = partial_literal(node.value)
|
||
if val is not _UNREADABLE:
|
||
found[tgt.id] = val
|
||
return found
|
||
|
||
|
||
def usage_block(text):
|
||
"""Only the `usage:` paragraph of a `--help` page.
|
||
|
||
argparse prints the module docstring as the description ABOVE the usage
|
||
line, and those docstrings discuss flags in prose -- `citation_hash_class.py`
|
||
mentions `--batch-all-objects` while explaining why it does not use it. A
|
||
flag that prose mentions is not a flag the tool accepts, so the oracle is
|
||
cut to the usage paragraph and stop: it ends at the first blank line after
|
||
the line that opens with `usage:`. No usage line means no oracle.
|
||
"""
|
||
lines = text.splitlines()
|
||
start = None
|
||
for i, line in enumerate(lines):
|
||
if line.lower().startswith("usage:"):
|
||
start = i
|
||
break
|
||
if start is None:
|
||
return ""
|
||
block = []
|
||
for line in lines[start:]:
|
||
if not line.strip():
|
||
break
|
||
block.append(line)
|
||
return "\n".join(block)
|
||
|
||
|
||
def purpose_of(src):
|
||
"""The module docstring's first sentence as a one-clause summary.
|
||
|
||
The house shape is `<name> -- <what it grades>.`; documents written without
|
||
the separator fall back to the docstring's opening sentence. The whole first
|
||
PARAGRAPH is flattened before the sentence cut, because the sentence that
|
||
states a tool's purpose routinely wraps across two or three source lines --
|
||
a line-local cut is what this suite grades as a defect everywhere else.
|
||
Returns "" when there is no module docstring, which the row prints as `-`.
|
||
"""
|
||
if not src:
|
||
return ""
|
||
try:
|
||
doc = ast.get_docstring(ast.parse(src)) or ""
|
||
except SyntaxError:
|
||
return ""
|
||
first_para = re.split(r"\n\s*\n", doc.strip())[0] if doc.strip() else ""
|
||
flat = " ".join(line.strip() for line in first_para.splitlines()).strip()
|
||
if not flat:
|
||
return ""
|
||
flat = re.sub(r"^[a-z0-9_]+\.[a-z]+ *(?:--|—|–|-) *", "", flat).strip()
|
||
m = re.search(r"^(.+?[.?!])(?:\s|$)", flat)
|
||
out = (m.group(1) if m else flat).rstrip()
|
||
if len(out) > 180:
|
||
out = out[:177].rstrip() + "..."
|
||
return out
|
||
|
||
|
||
def flags_of(repo, rev, name):
|
||
"""The tool's accepted options, from its own `--help`.
|
||
|
||
The blob is written to a temporary directory and run there so a tool is
|
||
adjudicated at `--rev` against the parser it will actually use, never
|
||
against a worktree copy. Returns (flags, note) where note is "" when the
|
||
oracle answered and a short reason when it did not.
|
||
"""
|
||
src = read_blob(repo, rev, f"tools/{name}")
|
||
if src is None:
|
||
return [], "no such file at rev"
|
||
with tempfile.TemporaryDirectory() as td:
|
||
path = os.path.join(td, name)
|
||
with open(path, "w", encoding="utf-8") as fh:
|
||
fh.write(src)
|
||
try:
|
||
proc = subprocess.run([sys.executable, path, "--help"],
|
||
capture_output=True, text=True,
|
||
timeout=HELP_TIMEOUT, cwd=td)
|
||
except subprocess.TimeoutExpired:
|
||
return [], "help timed out"
|
||
text = (proc.stdout or "") + (proc.stderr or "")
|
||
usage = usage_block(text)
|
||
seen, out = set(), []
|
||
for m in OPT_RE.finditer(usage):
|
||
opt = m.group(1)
|
||
if opt in SKIP_OPTS or opt in seen:
|
||
continue
|
||
seen.add(opt)
|
||
out.append(opt)
|
||
if not out:
|
||
return [], "no --help oracle"
|
||
return out, ""
|
||
|
||
|
||
def _count_by_tool(entries):
|
||
"""Count probes per tool name: ({tool: rows}, rows that were not readable).
|
||
|
||
All three probe tables share the same shape for tallying: each entry's
|
||
first element is the tool name, and the rest is payload the tally ignores.
|
||
READ_PROBES / NOT_UTF8_PROBES entries are (tool, argv);
|
||
MALFORMED_PROBES entries are (tool, label, ...). A row whose first element
|
||
is not a string -- an entry built from a name the AST could not resolve --
|
||
is counted as dropped instead of raising out of the renderer, and
|
||
`collect()` names the drop in the document so a short count is never read
|
||
as a table that has no more rows.
|
||
"""
|
||
counts, dropped = {}, 0
|
||
for entry in entries:
|
||
name = entry[0] if isinstance(entry, (list, tuple)) and entry else None
|
||
if not isinstance(name, str):
|
||
dropped += 1
|
||
continue
|
||
counts[name] = counts.get(name, 0) + 1
|
||
return counts, dropped
|
||
|
||
|
||
def collect(repo, rev):
|
||
"""The whole map as data: one dict per tool plus the gate's legs."""
|
||
names = tracked_tools(repo, rev)
|
||
if names is None:
|
||
raise SystemExit(f"tool_map: cannot read repo: {repo} at {rev}")
|
||
sweep = literal_assignments(repo, rev, "tools/selftest_sweep.py",
|
||
{"READ_PROBES", "NOT_UTF8_PROBES",
|
||
"MALFORMED_PROBES"})
|
||
read_n, read_d = _count_by_tool(sweep.get("READ_PROBES", ()))
|
||
utf8_n, utf8_d = _count_by_tool(sweep.get("NOT_UTF8_PROBES", ()))
|
||
mal_n, mal_d = _count_by_tool(sweep.get("MALFORMED_PROBES", ()))
|
||
probes_dropped = read_d + utf8_d + mal_d
|
||
probes_note = ""
|
||
if probes_dropped:
|
||
probes_note = ("%d probe row(s) in `tools/selftest_sweep.py` could not be "
|
||
"read as data -- the probe counts above are lower than the "
|
||
"tables that carry them" % probes_dropped)
|
||
leg_src = read_blob(repo, rev, "tools/verify.py") or ""
|
||
legs = literal_assignments(repo, rev, "tools/verify.py", {"LEGS"}).get("LEGS", ())
|
||
legs_note = ""
|
||
if not legs and re.search(r"^LEGS\s*=", leg_src, re.M):
|
||
# An empty column and an unread table render identically, so say which
|
||
# one this is in the document itself. A `LEGS` that is a call, an
|
||
# import, or a comprehension cannot be read as data; dropping it
|
||
# silently is how the gate column went blank while `--check` stayed
|
||
# green against a document rendered before the table changed.
|
||
legs_note = ("the `LEGS` table in `tools/verify.py` could not be read as "
|
||
"data -- the gate column above is empty because the map "
|
||
"failed to read it, not because no tool has a leg")
|
||
leg_of = {}
|
||
for leg in legs:
|
||
if isinstance(leg, dict) and "tool" in leg and "key" in leg:
|
||
leg_of.setdefault(leg["tool"], []).append(leg["key"])
|
||
|
||
rows = []
|
||
for name in names:
|
||
src = read_blob(repo, rev, f"tools/{name}")
|
||
flags, note = flags_of(repo, rev, name)
|
||
rows.append({
|
||
"tool": name,
|
||
"purpose": purpose_of(src),
|
||
"flags": flags,
|
||
"no_oracle": note,
|
||
"read": read_n.get(name, 0),
|
||
"not_utf8": utf8_n.get(name, 0),
|
||
"malformed": mal_n.get(name, 0),
|
||
"legs": leg_of.get(name, []),
|
||
})
|
||
return {"rev": rev, "rows": rows, "legs_note": legs_note,
|
||
"probes_note": probes_note,
|
||
"legs": [{"key": l.get("key"), "tool": l.get("tool"),
|
||
"slow": bool(l.get("slow")), "what": l.get("what", "")}
|
||
for l in legs if isinstance(l, dict)]}
|
||
|
||
|
||
def render(data):
|
||
"""The document, byte-deterministic for a given tree."""
|
||
rows, legs = data["rows"], data["legs"]
|
||
with_oracle = sum(1 for r in rows if not r["no_oracle"])
|
||
lines = [
|
||
"# The instruments",
|
||
"",
|
||
"Generated by `tools/tool_map.py` from the tree it is asked to read. Do not",
|
||
"edit by hand: `python3 tools/tool_map.py --repo . --write docs/TOOLS.md`",
|
||
"re-renders it, and `--check` grades it (rc 1 when the document and the tree",
|
||
"disagree). Every cell below is derived, never typed -- purpose from the",
|
||
"module docstring, flags from the tool's own `--help`, probe rows from",
|
||
"`selftest_sweep.py`'s tables, the gate leg from `verify.py`'s `LEGS`.",
|
||
"",
|
||
f"**{len(rows)} tools, {with_oracle} with a `--help` oracle.** "
|
||
f"Probes are read as `read-refusal / not-utf8 / malformed-input` rows; a "
|
||
f"`-` means that axis does not name the tool.",
|
||
"",
|
||
"| tool | what it grades | flags | probes | gate leg |",
|
||
"|---|---|---|---|---|",
|
||
]
|
||
for r in rows:
|
||
purpose = r["purpose"] or "-"
|
||
if r["no_oracle"]:
|
||
flags = f"- *({r['no_oracle']})*"
|
||
else:
|
||
flags = " ".join(f"`{f}`" for f in r["flags"]) or "-"
|
||
probes = (f"{r['read']} / {r['not_utf8']} / {r['malformed']}"
|
||
if (r["read"] or r["not_utf8"] or r["malformed"]) else "-")
|
||
gate = ", ".join(f"`{k}`" for k in r["legs"]) or "-"
|
||
lines.append(f"| `{r['tool']}` | {purpose} | {flags} | {probes} | {gate} |")
|
||
if data.get("probes_note"):
|
||
lines += ["", "*" + data["probes_note"] + "*"]
|
||
lines += [
|
||
"",
|
||
"## The gate",
|
||
"",
|
||
"`python3 tools/verify.py --repo .` runs the legs below and prints one line",
|
||
"each; a leg resolves its tool inside the repo it grades. The `slow` legs are",
|
||
"dropped by `--fast`.",
|
||
"",
|
||
"| leg | tool | slow | what it grades |",
|
||
"|---|---|---|---|",
|
||
]
|
||
for leg in legs:
|
||
lines.append(f"| `{leg['key']}` | `{leg['tool']}` | "
|
||
f"{'yes' if leg['slow'] else 'no'} | {leg['what']} |")
|
||
if data.get("legs_note"):
|
||
lines += ["", "*" + data["legs_note"] + "*"]
|
||
lines += [
|
||
"",
|
||
"A tool that appears in no leg is still graded by `selftest-sweep`, which runs",
|
||
"every tracked tool's `--selftest` and both read-refusal axes -- so the",
|
||
"`probes` column above is also the sweep's cover of that tool.",
|
||
"",
|
||
]
|
||
return "\n".join(lines)
|
||
|
||
|
||
def selftest(repo):
|
||
cells = []
|
||
|
||
# The currency cell below grades `repo`; if `repo` is not a work tree that
|
||
# cell can only fail, which reads as "the document drifted" when the truth
|
||
# is "no tree was there". Refuse at rc 2, as main() already does for the
|
||
# same input.
|
||
rc, _, _ = git(repo, "rev-parse", "--show-toplevel")
|
||
if rc != 0:
|
||
print(f"tool_map: cannot read repo: {repo} is not a git work tree",
|
||
file=sys.stderr)
|
||
return 2
|
||
|
||
def cell(label, ok, detail=""):
|
||
cells.append({"label": label, "ok": bool(ok), "detail": detail})
|
||
|
||
# -- the derivation, on fixtures -------------------------------------
|
||
cell("purpose: house `<name> -- ...` shape is stripped",
|
||
purpose_of('"""fake_tool.py -- grades the fake contract, at one line.\n\n'
|
||
'More prose here."""\n') ==
|
||
"grades the fake contract, at one line.")
|
||
cell("purpose: no separator falls back to the first sentence",
|
||
purpose_of('"""A map of the tree. Second sentence follows."""\n') ==
|
||
"A map of the tree.")
|
||
cell("purpose: absent docstring is empty, not invented",
|
||
purpose_of("import os\n") == "")
|
||
cell("flags: `-h`/`--help` are dropped, real options kept",
|
||
[o for o in OPT_RE.findall("usage: t.py [-h] [--repo R] [--rev V]")
|
||
if o not in SKIP_OPTS] == ["--repo", "--rev"])
|
||
cell("flags: a positional-looking token is not an option",
|
||
OPT_RE.findall("usage: t.py [-h] [--repo REPO] filename") ==
|
||
["-h", "--repo"])
|
||
cell("flags: a flag the description only MENTIONS is not an oracle",
|
||
usage_block("desc mentions --nope\n\noptions:\n") == "")
|
||
|
||
with tempfile.TemporaryDirectory() as td:
|
||
fixture = os.path.join(td, "fx")
|
||
os.makedirs(os.path.join(fixture, "tools"))
|
||
os.makedirs(os.path.join(fixture, "docs"))
|
||
env = dict(os.environ, GIT_AUTHOR_NAME="t", GIT_AUTHOR_EMAIL="t@e",
|
||
GIT_COMMITTER_NAME="t", GIT_COMMITTER_EMAIL="t@e")
|
||
subprocess.run(["git", "init", "-q", fixture], check=True, env=env)
|
||
open(os.path.join(fixture, "tools", "fake_tool.py"), "w").write(
|
||
'#!/usr/bin/env python3\n'
|
||
'"""fake_tool.py -- grades the fake contract, which nothing measured.\n\n'
|
||
'Prose."""\n'
|
||
'import argparse\n'
|
||
'ap = argparse.ArgumentParser()\n'
|
||
'ap.add_argument("--repo")\n'
|
||
'ap.add_argument("--fail", action="store_true")\n'
|
||
'if "--help" in __import__("sys").argv:\n'
|
||
' ap.print_help()\n')
|
||
open(os.path.join(fixture, "tools", "quiet_tool.py"), "w").write(
|
||
'#!/usr/bin/env python3\n'
|
||
'"""quiet_tool.py -- parses argv by hand, so it prints no usage."""\n'
|
||
'import sys\n'
|
||
'print("ran")\n')
|
||
open(os.path.join(fixture, "tools", "selftest_sweep.py"), "w").write(
|
||
'READ_PROBES = (("fake_tool.py", ["--repo", "{nope}"]),)\n'
|
||
'NOT_UTF8_PROBES = ()\n'
|
||
'MALFORMED_PROBES = (("fake_tool.py", "not-a-repo",\n'
|
||
' ["--repo", "{notrepo}"], 2, "cannot read"),)\n')
|
||
open(os.path.join(fixture, "tools", "verify.py"), "w").write(
|
||
'LEGS = ({"key": "fake", "tool": "fake_tool.py", "slow": False,\n'
|
||
' "what": "grade the fake contract"},)\n')
|
||
|
||
def commit(msg):
|
||
subprocess.run(["git", "-C", fixture, "add", "-A"], check=True, env=env)
|
||
subprocess.run(["git", "-C", fixture, "commit", "-qm", msg],
|
||
check=True, env=env)
|
||
|
||
commit("fixture")
|
||
data = collect(fixture, "HEAD")
|
||
by = {r["tool"]: r for r in data["rows"]}
|
||
cell("collect: every tracked .py is a row, oracle or not",
|
||
len(data["rows"]) == 4 and by["fake_tool.py"]["no_oracle"] == ""
|
||
and by["quiet_tool.py"]["no_oracle"] == "no --help oracle",
|
||
str([(r["tool"], r["no_oracle"]) for r in data["rows"]]))
|
||
cell("collect: probe rows and the gate leg are joined by name",
|
||
by["fake_tool.py"]["read"] == 1
|
||
and by["fake_tool.py"]["malformed"] == 1
|
||
and by["fake_tool.py"]["legs"] == ["fake"],
|
||
str({k: (v["read"], v["malformed"], v["legs"]) for k, v in by.items()}))
|
||
cell("collect: only printed flags are listed, and --help is not",
|
||
by["fake_tool.py"]["flags"] == ["--repo", "--fail"],
|
||
str(by["fake_tool.py"]["flags"]))
|
||
doc = os.path.join(fixture, "docs", "TOOLS.md")
|
||
text = render(data)
|
||
open(doc, "w").write(text)
|
||
cell("render: deterministic (two renders of one tree are identical)",
|
||
text == render(collect(fixture, "HEAD")))
|
||
cell("render: the tool without an oracle is marked, not accused",
|
||
"quiet_tool.py" in text and "no --help oracle" in text)
|
||
rc_ok = compare(doc, text)
|
||
cell("--check: rc 0 on the document the same tree renders", rc_ok == 0,
|
||
f"rc {rc_ok}")
|
||
open(doc, "a").write("stale line\n")
|
||
rc_drift = compare(doc, render(collect(fixture, "HEAD")), quiet=True)
|
||
cell("--check: rc 1 once a byte drifts",
|
||
rc_drift == 1, f"rc {rc_drift}")
|
||
open(doc, "w").write(text)
|
||
rc_missing = compare(os.path.join(fixture, "docs", "nope.md"), text,
|
||
quiet=True)
|
||
cell("--check: rc 2, not rc 1, when the document does not exist",
|
||
rc_missing == 2, f"rc {rc_missing}")
|
||
rc_norepo = graderc(td)
|
||
cell("--repo a directory that is not a git work tree: rc 2",
|
||
rc_norepo == 2, f"rc {rc_norepo}")
|
||
|
||
# -- WORKTREE: the tree you are about to commit, not the last commit --
|
||
# The map is derived from a revision by construction, so a docstring
|
||
# edited after the last commit is invisible to both `--check` and the
|
||
# currency cell below. That is measured here, not described: the same
|
||
# document is current against HEAD and drifted against the worktree.
|
||
cell("WORKTREE: a clean worktree renders exactly what HEAD renders",
|
||
render(collect(fixture, WORKTREE)) == text)
|
||
fake = os.path.join(fixture, "tools", "fake_tool.py")
|
||
with open(fake, encoding="utf-8") as fh:
|
||
src = fh.read()
|
||
with open(fake, "w", encoding="utf-8") as fh:
|
||
fh.write(src.replace("which nothing measured",
|
||
"which nothing has measured yet"))
|
||
wt = collect(fixture, WORKTREE)
|
||
hd = collect(fixture, "HEAD")
|
||
purpose_wt = {r["tool"]: r["purpose"] for r in wt["rows"]}["fake_tool.py"]
|
||
purpose_hd = {r["tool"]: r["purpose"] for r in hd["rows"]}["fake_tool.py"]
|
||
cell("WORKTREE: reads the file on disk, not the blob at HEAD",
|
||
purpose_wt != purpose_hd and "has measured yet" in purpose_wt,
|
||
f"worktree={purpose_wt!r} head={purpose_hd!r}")
|
||
rc_head = compare(doc, render(hd), quiet=True)
|
||
rc_wt = compare(doc, render(wt), quiet=True)
|
||
cell("blind spot: an uncommitted docstring edit is rc 0 at HEAD and rc 1 at WORKTREE",
|
||
rc_head == 0 and rc_wt == 1, f"HEAD rc {rc_head}, WORKTREE rc {rc_wt}")
|
||
rc_wt_norepo = graderc(td, rev=WORKTREE)
|
||
cell("WORKTREE outside a git work tree: rc 2, not a crash",
|
||
rc_wt_norepo == 2, f"rc {rc_wt_norepo}")
|
||
|
||
# -- a leg field that is not a literal must not erase the leg --------
|
||
# Q130 gave the closed-artifacts leg `"path": os.path.join(...)`. That
|
||
# one call made `ast.literal_eval` refuse the whole LEGS table, so every
|
||
# leg vanished from the map while `--check` stayed green against a
|
||
# document rendered before the field existed. Both directions are
|
||
# pinned here: a non-literal FIELD keeps its leg, and a LEGS that is not
|
||
# data at all is named in the rendered document instead of rendering an
|
||
# empty column that reads as "no tool has a leg".
|
||
vpath = os.path.join(fixture, "tools", "verify.py")
|
||
with open(vpath, "w") as fh:
|
||
fh.write('import os\n'
|
||
'LEGS = ({"key": "fake", "tool": "fake_tool.py",\n'
|
||
' "slow": False,\n'
|
||
' "path": os.path.join("tools", "fake_tool.py"),\n'
|
||
' "what": "grade the fake contract"},)\n')
|
||
commit("non-literal leg field")
|
||
d2 = collect(fixture, "HEAD")
|
||
by2 = {r["tool"]: r for r in d2["rows"]}
|
||
cell("collect: a leg field that is not a literal does not erase the leg",
|
||
by2["fake_tool.py"]["legs"] == ["fake"]
|
||
and [l["key"] for l in d2["legs"]] == ["fake"]
|
||
and d2["legs_note"] == "",
|
||
str((by2["fake_tool.py"]["legs"], d2["legs"], d2["legs_note"])))
|
||
|
||
with open(vpath, "w") as fh:
|
||
fh.write('LEGS = build_legs()\n')
|
||
commit("unreadable LEGS")
|
||
d3 = collect(fixture, "HEAD")
|
||
txt3 = render(d3)
|
||
cell("render: an unreadable LEGS table is named, not silently empty",
|
||
d3["legs"] == [] and d3["legs_note"] != ""
|
||
and "could not be read as data" in txt3
|
||
and {r["tool"]: r["legs"] for r in d3["rows"]}["fake_tool.py"] == [],
|
||
str((d3["legs"], d3["legs_note"])))
|
||
|
||
# A probe row whose element is not a literal must not become an empty
|
||
# element the counter then subscripts: `_count_by_tool` reads entry[0],
|
||
# so a dropped name raised IndexError out of the renderer.
|
||
with open(os.path.join(fixture, "tools", "verify.py"), "w") as fh:
|
||
fh.write('LEGS = ({"key": "fake", "tool": "fake_tool.py",\n'
|
||
' "slow": False, "what": "grade the fake contract"},)\n')
|
||
with open(os.path.join(fixture, "tools", "selftest_sweep.py"), "w") as fh:
|
||
fh.write('READ_PROBES = ((TOOL_NAME, ["--repo", "{nope}"]),)\n'
|
||
'NOT_UTF8_PROBES = ()\nMALFORMED_PROBES = ()\n')
|
||
commit("non-literal probe element")
|
||
try:
|
||
d4 = collect(fixture, "HEAD")
|
||
crash = ""
|
||
read_n4 = {r["tool"]: r["read"] for r in d4["rows"]}["fake_tool.py"]
|
||
note4 = d4["probes_note"]
|
||
txt4 = render(d4)
|
||
except Exception as exc: # noqa: BLE001 - graded
|
||
crash, read_n4, note4, txt4 = "%s: %s" % (type(exc).__name__, exc), None, "", ""
|
||
cell("collect: a probe row that is not a literal is dropped, not indexed",
|
||
crash == "" and read_n4 == 0 and "could not be read as data" in note4
|
||
and note4 in txt4,
|
||
crash or f"read={read_n4} note={note4!r}")
|
||
|
||
# -- the currency cell: the reason this document stays true ----------
|
||
here = os.path.join(repo, "docs", "TOOLS.md")
|
||
try:
|
||
want = render(collect(repo, "HEAD"))
|
||
rc = compare(here, want, quiet=True)
|
||
except SystemExit as exc:
|
||
rc = 2
|
||
cell("HEAD: the shipped map renders", False, str(exc))
|
||
cell("HEAD: docs/TOOLS.md is the render of this tree (the drift guard)",
|
||
rc == 0, f"rc {rc}")
|
||
|
||
passed = sum(1 for c in cells if c["ok"])
|
||
for c in cells:
|
||
print(f" {'ok ' if c['ok'] else 'FAIL'} {c['label']}"
|
||
+ (f" [{c['detail']}]" if c["detail"] and not c["ok"] else ""))
|
||
print(f"selftest: {len(cells)} cells, {passed} passed, {len(cells) - passed} failed")
|
||
return 0 if passed == len(cells) else 1
|
||
|
||
|
||
def graderc(td, rev=None):
|
||
"""rc of a full `--check` run against a directory that is not a repo."""
|
||
cmd = [sys.executable, os.path.abspath(__file__), "--repo", td]
|
||
if rev:
|
||
cmd += ["--rev", rev]
|
||
cmd += ["--check", os.path.join(td, "x.md")]
|
||
proc = subprocess.run(cmd, capture_output=True, text=True)
|
||
return proc.returncode
|
||
|
||
|
||
def compare(path, want, quiet=False):
|
||
"""0 current, 1 drift (first differing line printed), 2 unreadable."""
|
||
try:
|
||
with open(path, encoding="utf-8") as fh:
|
||
have = fh.read()
|
||
except OSError as exc:
|
||
if not quiet:
|
||
print(f"tool_map: cannot read document: {exc}")
|
||
return 2
|
||
if have == want:
|
||
return 0
|
||
if not quiet:
|
||
a, b = have.splitlines(), want.splitlines()
|
||
for i in range(max(len(a), len(b))):
|
||
got = a[i] if i < len(a) else "<end of document>"
|
||
exp = b[i] if i < len(b) else "<end of document>"
|
||
if got != exp:
|
||
print(f"DRIFT at line {i + 1}:")
|
||
print(f" document: {got}")
|
||
print(f" tree: {exp}")
|
||
break
|
||
return 1
|
||
|
||
|
||
def main():
|
||
ap = argparse.ArgumentParser(
|
||
description="Render the map of tools/ from the tree, and grade its drift.")
|
||
ap.add_argument("--repo", default=DEFAULT_REPO)
|
||
ap.add_argument("--rev", default="HEAD")
|
||
ap.add_argument("--write", metavar="PATH",
|
||
help=f"render the map to PATH (default target {DEFAULT_DOC})")
|
||
ap.add_argument("--check", metavar="PATH",
|
||
help="grade PATH against a fresh render; rc 1 on drift")
|
||
ap.add_argument("--json", metavar="PATH", help="write the collected data as JSON")
|
||
ap.add_argument("--selftest", action="store_true")
|
||
args = ap.parse_args()
|
||
|
||
if args.selftest:
|
||
return selftest(args.repo)
|
||
|
||
rc, _, _ = git(args.repo, "rev-parse", "--show-toplevel")
|
||
if rc != 0:
|
||
print(f"tool_map: cannot read repo: {args.repo} is not a git work tree")
|
||
return 2
|
||
if args.rev != WORKTREE:
|
||
rc, _, err = git(args.repo, "rev-parse", "--verify", f"{args.rev}^{{commit}}")
|
||
if rc != 0:
|
||
print(f"tool_map: cannot read rev {args.rev}: {err.strip().splitlines()[0] if err.strip() else 'no such object'}")
|
||
return 2
|
||
|
||
data = collect(args.repo, args.rev)
|
||
text = render(data)
|
||
|
||
if args.json:
|
||
try:
|
||
with open(args.json, "w", encoding="utf-8") as fh:
|
||
json.dump(data, fh, indent=2, sort_keys=True)
|
||
fh.write("\n")
|
||
except OSError as exc:
|
||
print("cannot write --json " + args.json + ": " + str(exc), file=sys.stderr)
|
||
return 2
|
||
print(f"tool_map: wrote {args.json}")
|
||
|
||
if args.write:
|
||
try:
|
||
with open(args.write, "w", encoding="utf-8") as fh:
|
||
fh.write(text)
|
||
except OSError as exc:
|
||
print("cannot write --write " + args.write + ": " + str(exc), file=sys.stderr)
|
||
return 2
|
||
print(f"tool_map: wrote {args.write} ({len(text)} B, "
|
||
f"{len(data['rows'])} tools, {len(data['legs'])} gate legs)")
|
||
return 0
|
||
|
||
if args.check:
|
||
return compare(args.check, text)
|
||
|
||
print(text, end="")
|
||
return 0
|
||
|
||
|
||
if __name__ == "__main__":
|
||
sys.exit(main())
|