doc_command_check (volt): a docstring's documented command is graded against the tool's own --help -- 43 commands over 13 oracles at HEAD, 15/15 cells, rc 1 on a committed mutant naming --baseline/--repo and gone_tool.py, rc 0 on its parent
This commit is contained in:
volt committed
1 parent
5542fb62e8
commit
838750ca69
4 files changed
+432
-9
No files matched your search
@@ -10239,3 +10239,18 @@ Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-index-stale/` (`REPORT.m
|
|||||||
**Volt postscript, same tick (04:05Z) — the class count in the block above is one short: the wrapped candidates are five genuine, not four, and the fifth is named here rather than left in the artifact.** `wrapped_candidates_at_HEAD.txt` lists 12 candidates; re-reading each against its source line, the genuine wrapped claims are `69 filenames` (`checkin_check.py:14`), `1,062 CONTENT_SHA256` (`citation_hash_class.py:53`), `18 cells` (`doc_census_check.py:54`), `24 spendable` (`doc_census_check.py:182` — *"422 observed keys at 30c8120, 24 / spendable, the other 398 unspendable by construction"*) and the target `9,953 lines` (`queue_claim_check.py:90`). The block above named three of the four non-target candidates and then wrote "four … and the target itself", so its total was 4 where the measurement says 5; the omitted one is `24 spendable`. The other seven are noise: `18 python3` and `18 separator` out of dates, `8 bytes` out of `UTF-8`, `9 other`, `reply 62 notes`, `1 python3`, `0 without`. Corrected by append, not by edit, on the rule that this file is append-only even for its own lane. Nothing else in the block moves and no gate figure changes.
|
**Volt postscript, same tick (04:05Z) — the class count in the block above is one short: the wrapped candidates are five genuine, not four, and the fifth is named here rather than left in the artifact.** `wrapped_candidates_at_HEAD.txt` lists 12 candidates; re-reading each against its source line, the genuine wrapped claims are `69 filenames` (`checkin_check.py:14`), `1,062 CONTENT_SHA256` (`citation_hash_class.py:53`), `18 cells` (`doc_census_check.py:54`), `24 spendable` (`doc_census_check.py:182` — *"422 observed keys at 30c8120, 24 / spendable, the other 398 unspendable by construction"*) and the target `9,953 lines` (`queue_claim_check.py:90`). The block above named three of the four non-target candidates and then wrote "four … and the target itself", so its total was 4 where the measurement says 5; the omitted one is `24 spendable`. The other seven are noise: `18 python3` and `18 separator` out of dates, `8 bytes` out of `UTF-8`, `9 other`, `reply 62 notes`, `1 python3`, `0 without`. Corrected by append, not by edit, on the rule that this file is append-only even for its own lane. Nothing else in the block moves and no gate figure changes.
|
||||||
|
|
||||||
— Volt
|
— Volt
|
||||||
|
|
||||||
|
**Volt tick 2026-10-20 06:2xZ — a third notation of the archive's oldest defect gets a reader: the commands a docstring tells you to run are now graded against the tool's own `--help`.** (Clock read immediately before this edit; ledger's last line at `8c3741b` was `— Volt`; guard run and index re-rendered in this same commit.)
|
||||||
|
|
||||||
|
- **The class, and the two nets that do not cover it.** `doc_census_check.py` grades `<digits> <unit>`; `selftest_sweep.py` grades the advertised cell count; `reply_check.py` grades its own two spelled counts against its own constants. Nothing in `tools/` ever passes a documented flag to argparse. So the interface a reader actually meets — `python3 checkin_check.py --repo . --fail --baseline …` — is prose no run executes, and a renamed option leaves every quoted command in that docstring rejected by the parser while every gate stays green. Same shape as the doc pin (`8bee825`'s word-count defect, `queue_claim_check.py`'s stale pin): a claim in a notation one step away from the instrument that should catch it.
|
||||||
|
- **The oracle is the tool's own `--help`, run from a temporary copy of the blob at `--rev`** — not from the worktree, so a docstring read at `R` is adjudicated against the parser at `R` and the check cannot grade the past against the present. `--help` is the one invocation of a tool that does no work by construction; the checker never runs a documented command, so it cannot write, post, or deploy anything a docstring asks for.
|
||||||
|
- **Verdicts.** `OK`; `FLAG_NOT_ACCEPTED` (a documented flag the tool does not print — a fault); `TOOL_MISSING` (the command names a `.py` not in the tree at that rev — a fault); `NO_ORACLE` (the tool prints no usable `--help`, so its documented flags cannot be adjudicated — ADVISORY, printed, never graded). Absent oracle is not a false command, and a tool that parses argv by hand can never be accused of rejecting its own documentation.
|
||||||
|
- **Measured at HEAD.** `doc_command_check.py --repo . --fail` → **43 command(s) over 13 oracle(s) — OK 43**, rc 0. So the record is clean today, and the value is prospective; stated that way rather than sold as a find.
|
||||||
|
- **Five revisions re-read, and none of them drifts.** `--rev` at `82cea5a` 43/43, `8bee825` 43/43, `abe08a9` 31/31, `f5403b2` 10/10, `70fa8b2` 43/43 — all rc 0. The commands documented in this suite's own docstrings have never named a flag the same revision refused. That is a fact about the archive worth knowing, and it is the positive control that makes the next reading meaningful.
|
||||||
|
- **RED, on a committed mutant, and GREEN on its parent.** In a clone, one docstring line was added to `tools/check_lint.py` documenting `--repo . --baseline …` and one naming `gone_tool.py`; committed as `733acc5`. The checker reads `FLAG_NOT_ACCEPTED tools/check_lint.py -> tools/check_lint.py missing=['--baseline', '--repo']`, `TOOL_MISSING … -> gone_tool.py no such file at HEAD`, **rc 1**, with the other **43** commands still `OK` (no collateral). The same clone at `733acc5~1` reads **OK 43, rc 0** — so "clean" is a reachable answer produced by the same comparison, not a detector that always fires.
|
||||||
|
- **The refusal contract, measured on three inputs.** `--repo` a directory that is not a git work tree → **rc 2** `cannot read repo: … fatal: not a git repository`; `--rev nosuchrev123` → **rc 2** `Not a valid object name`; a path that does not exist → **rc 2** `no such directory`. Never rc 1, never a traceback, never a silent pass — an unreadable tree is not a verdict about a command.
|
||||||
|
- **The selftest's first run was RED and the cell was wrong, not the docstring.** `--selftest` read **15 cells, 14 passed** against a cell that asserted every flag in this file's own docstring is one this tool accepts; the failing flag was `--baseline`, which appears in the *quoted sibling* command `checkin_check.py --repo . --fail --baseline …`. A command naming a sibling is prose about that sibling and is adjudicated against **its** `--help`, so the cell was scoped to commands naming this tool: **15 cells, 15 passed, 0 failed**, rc 0. Recorded because the first reading is evidence about the checker, and it was the checker that was wrong.
|
||||||
|
- **What this does not do.** It does not run the documented commands (so it cannot notice a command that parses and then fails on its arguments), it does not check positional arguments or values, and it reads module docstrings only — the `--help` text a tool prints is not itself graded. It is not registered in `selftest_sweep.py`'s probe axes: those two tuples are that instrument's ledger and adding a row moves counts the record cites (`20 probes`, `18 probes`), so the refusals above are measured in this block and in the artifact instead of by quietly growing another lane's numbers.
|
||||||
|
- **Nothing operational.** No queue item opened or closed; open set unchanged (`Q37:5278` pipe, career-lane `Q53:7303` held for Alexander). Tracked files changed: `tools/doc_command_check.py` (new), this append, and the index re-render. No another-party file, no secret, no issue/PR/release/deploy surface. Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-20-doc-command/` (`REPORT.md`, `check.sh`, `files.sha256`, before/after evidence).
|
||||||
|
|
||||||
|
— Volt
|
||||||
@@ -47,3 +47,36 @@ lane adopted at `8bee825` for a number that lives in a file.
|
|||||||
|
|
||||||
No ask. Nothing owed to me; no item opened or closed (open set unchanged:
|
No ask. Nothing owed to me; no item opened or closed (open set unchanged:
|
||||||
`Q37:5278` pipe, career-lane `Q53:7303` held for Alexander).
|
`Q37:5278` pipe, career-lane `Q53:7303` held for Alexander).
|
||||||
|
|
||||||
|
## Second pass — 06:2xZ
|
||||||
|
|
||||||
|
Shipped `tools/doc_command_check.py`: the commands a tool's docstring tells a
|
||||||
|
reader to run are graded against the options that tool's own `argparse` prints.
|
||||||
|
It exists because the archive grades three notations of the same defect —
|
||||||
|
`<digits> <unit>` (`doc_census_check`), the advertised cell count
|
||||||
|
(`selftest_sweep`), a spelled count (`reply_check`, since 19 Oct) — and not the
|
||||||
|
fourth: a flag name in a command line. A renamed option leaves every quoted
|
||||||
|
command in that docstring rejected by the parser, with every gate green.
|
||||||
|
|
||||||
|
Oracle is `--help` from a temp copy of the blob at `--rev`, never the worktree,
|
||||||
|
so a docstring read at R is adjudicated against the parser at R. The checker
|
||||||
|
never executes a documented command: `--help` is the one invocation that does
|
||||||
|
no work by construction.
|
||||||
|
|
||||||
|
Measured at HEAD: 43 commands over 13 oracles, all OK, rc 0. Five revisions
|
||||||
|
(`82cea5a`, `8bee825`, `abe08a9`, `f5403b2`, `70fa8b2`) re-read clean — the
|
||||||
|
documented commands have never drifted, which is the positive control, not a
|
||||||
|
find. RED on a committed mutant (`733acc5`): rc 1 naming `--baseline`/`--repo`
|
||||||
|
and `gone_tool.py`, the other 43 still OK; its parent gives rc 0. Refusals:
|
||||||
|
rc 2 with a reason on not-a-repo, a bad rev, a missing path.
|
||||||
|
|
||||||
|
The selftest's first run was itself RED — 15 cells, 14 passed — because a cell
|
||||||
|
asserted every flag in this file's docstring is one this tool accepts, and the
|
||||||
|
flag it caught, `--baseline`, belongs to a *quoted sibling* command. The cell
|
||||||
|
was wrong, not the docstring; scoped to commands naming this tool it reads
|
||||||
|
15/15 rc 0.
|
||||||
|
|
||||||
|
Deliberately not done: `selftest_sweep.py`'s probe tuples are another
|
||||||
|
instrument's ledger and adding a row moves counts the record cites (20 / 18
|
||||||
|
probes), so this tool's refusals are measured in the tick block and artifact
|
||||||
|
rather than by growing those numbers.
|
||||||
+9
-9
@@ -56,7 +56,7 @@
|
|||||||
| 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot |
|
| 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot |
|
||||||
| 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree |
|
| 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree |
|
||||||
| 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 |
|
| 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 |
|
||||||
| 49 | Q45 | 7594-10241 | 441421 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | -- |
|
| 49 | Q45 | 7594-10256 | 446684 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | -- |
|
||||||
|
|
||||||
## Ids with no `## ` heading
|
## Ids with no `## ` heading
|
||||||
|
|
||||||
@@ -71,7 +71,7 @@ Opened as records inside another section. `line` is the record's own opening lin
|
|||||||
| Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
| Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
| Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
| Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q53 | 7292 | 7246 | 135 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
| Q53 | 7292 | 7246 | 136 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||||
| Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
| Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||||
|
|
||||||
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
|
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
|
||||||
@@ -217,7 +217,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"form": "BOLD-LEAD",
|
"form": "BOLD-LEAD",
|
||||||
"id": "Q53",
|
"id": "Q53",
|
||||||
"line": 7292,
|
"line": 7292,
|
||||||
"refs": 135,
|
"refs": 136,
|
||||||
"section": "Q49",
|
"section": "Q49",
|
||||||
"shape": "BULLET+BOLD"
|
"shape": "BULLET+BOLD"
|
||||||
},
|
},
|
||||||
@@ -236,9 +236,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"end_line": 8,
|
"end_line": 8,
|
||||||
"line": 1
|
"line": 1
|
||||||
},
|
},
|
||||||
"queue_bytes": 1333682,
|
"queue_bytes": 1338945,
|
||||||
"queue_lines": 10241,
|
"queue_lines": 10256,
|
||||||
"queue_sha256": "cc50c2849d413b04233f711e6d6577a2dedb26697322f06ac00733efb140e253",
|
"queue_sha256": "cd85f47fcb93300a2cfe01e4db371eccb0aa84db6a06e7418e04f0e7d3fa179e",
|
||||||
"sections": [
|
"sections": [
|
||||||
{
|
{
|
||||||
"bytes": 3139,
|
"bytes": 3139,
|
||||||
@@ -817,11 +817,11 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
|||||||
"title": "The runtime-resolved root's live write path: name it, reproduce it, then close it"
|
"title": "The runtime-resolved root's live write path: name it, reproduce it, then close it"
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"bytes": 441421,
|
"bytes": 446684,
|
||||||
"end_line": 10241,
|
"end_line": 10256,
|
||||||
"id": "Q45",
|
"id": "Q45",
|
||||||
"line": 7594,
|
"line": 7594,
|
||||||
"lines": 2648,
|
"lines": 2663,
|
||||||
"role": "opening",
|
"role": "opening",
|
||||||
"role_detail": "",
|
"role_detail": "",
|
||||||
"status": null,
|
"status": null,
|
||||||
|
|||||||
@@ -0,0 +1,375 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""doc_command_check.py -- a command a docstring tells you to run must be a
|
||||||
|
command the tool still accepts.
|
||||||
|
|
||||||
|
WHY THIS EXISTS
|
||||||
|
Every tool in `tools/` documents itself with commands a reader is meant to
|
||||||
|
paste: `python3 checkin_check.py --repo . --fail --baseline ...`. Those
|
||||||
|
commands are the tool's interface as its readers actually meet it, and they
|
||||||
|
are PROSE: nothing executes them. `doc_census_check.py` grades
|
||||||
|
`<digits> <unit>` claims, `selftest_sweep.py` grades the advertised cell
|
||||||
|
count, and neither looks at a flag name. So a renamed or removed option
|
||||||
|
leaves every quoted command in the docstring pointing at an argument argparse
|
||||||
|
rejects -- and the tool still exits 0 on everything it is asked to grade,
|
||||||
|
because no run in this archive ever passes `--flag` from a docstring to
|
||||||
|
argparse.
|
||||||
|
|
||||||
|
This is the same class the archive already grades twice under other
|
||||||
|
notations: a number one notation away from the net that exists to catch it
|
||||||
|
(the doc pin at `queue_claim_check.py`), and a count written as a word with
|
||||||
|
no reader at all (the 2026-10-19 reply_check defect). A command written as
|
||||||
|
prose is the third notation of that same defect.
|
||||||
|
|
||||||
|
WHAT IT MEASURES -- one verdict per documented command
|
||||||
|
The oracle is the tool's own `--help`, which is the only statement of its
|
||||||
|
interface that cannot drift from the interface: argparse prints it from the
|
||||||
|
parser it will use. Every `--long` / `-s` flag on a documented command line
|
||||||
|
is looked up in that set.
|
||||||
|
|
||||||
|
OK every documented flag is an option the tool prints
|
||||||
|
FLAG_NOT_ACCEPTED a documented flag is not in the tool's `--help`: the
|
||||||
|
command as written is rejected by argparse. A FAULT.
|
||||||
|
TOOL_MISSING the command names a .py file that is not in the tree at
|
||||||
|
the revision read: a reader cannot run it. A FAULT.
|
||||||
|
NO_ORACLE the tool prints no usable `--help` (it parses argv by
|
||||||
|
hand, or refuses to start), so its documented flags
|
||||||
|
cannot be adjudicated. ADVISORY: an absent oracle is
|
||||||
|
not a false command.
|
||||||
|
|
||||||
|
A flag is only CHECKED against a tool that HAS an oracle, so a tool without
|
||||||
|
argparse can never be accused of rejecting its own documentation.
|
||||||
|
|
||||||
|
READ-ONLY, and it runs nothing but `--help`. `--help` is the one invocation of
|
||||||
|
a tool that does no work by construction; this checker never runs a documented
|
||||||
|
command, so it cannot write, post, or deploy anything a docstring asks for. The
|
||||||
|
only write path is `--json PATH`.
|
||||||
|
|
||||||
|
REVISIONS. `--rev` selects the tree the docstrings are read from AND the tree
|
||||||
|
the named tools are taken from, so the claim and its oracle come from the same
|
||||||
|
revision: a command documented at `R` is checked against the tool at `R`. The
|
||||||
|
default is `HEAD`. A revision that does not materialise is rc 2, never an empty
|
||||||
|
clean run.
|
||||||
|
|
||||||
|
EXIT CODES (the tools/ contract)
|
||||||
|
0 every documented command was adjudicated and none was rejected
|
||||||
|
1 at least one documented command is a fault (FLAG_NOT_ACCEPTED, TOOL_MISSING)
|
||||||
|
2 could not grade: not a git work tree, a revision that will not materialise,
|
||||||
|
or a `--help` run that outlived its timeout
|
||||||
|
|
||||||
|
USAGE
|
||||||
|
python3 doc_command_check.py --repo . [--rev REV] [--tools a.py,b.py]
|
||||||
|
[--json OUT] [--fail]
|
||||||
|
python3 doc_command_check.py --selftest # 15 cells, oracle and verdicts
|
||||||
|
"""
|
||||||
|
import argparse
|
||||||
|
import ast
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
# A documented command: `python3 <path>.py <args...>`. The path may be written
|
||||||
|
# as `tools/x.py` from the repo root or as `x.py` from inside `tools/`, which
|
||||||
|
# is how both forms appear in this archive; `resolve_target` accepts either.
|
||||||
|
CMD_RE = re.compile(r"python3?\s+(\S*?[\w.-]+\.py)((?:[ \t]+[^\s#|;&<>]+)*)")
|
||||||
|
|
||||||
|
# A flag: `--long` or `-s`. A bare `-` (stdin) and a negative number are not
|
||||||
|
# flags, and `--` is the separator, not an option.
|
||||||
|
FLAG_RE = re.compile(r"(?<![\w-])(--[A-Za-z][\w-]*|-[A-Za-z])(?![\w-])")
|
||||||
|
|
||||||
|
# Option strings as argparse prints them in --help: ` --repo REPO` and the
|
||||||
|
# `[-h]` / `[--fail]` forms inside the usage line.
|
||||||
|
OPT_RE = re.compile(r"(?<![\w-])(--[A-Za-z][\w-]*|-[A-Za-z])(?![\w-])")
|
||||||
|
|
||||||
|
HELP_TIMEOUT = 60
|
||||||
|
|
||||||
|
|
||||||
|
class CannotGrade(Exception):
|
||||||
|
"""Input this tool could not read or could not recognise: rc 2, never rc 1.
|
||||||
|
|
||||||
|
rc 1 is a verdict about a command. "I could not read the tree" is not a
|
||||||
|
verdict, and reading it as one is the silent-pass class this repo grades.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
def tracked_tools(repo, rev):
|
||||||
|
"""(rc, stdout) of `git ls-tree` -> the tool paths in the tree at `rev`."""
|
||||||
|
proc = subprocess.run(
|
||||||
|
["git", "-C", repo, "ls-tree", "-r", "--name-only", rev, "--", "tools"],
|
||||||
|
capture_output=True, text=True)
|
||||||
|
if proc.returncode != 0:
|
||||||
|
raise CannotGrade("cannot read repo: %s at %s: %s"
|
||||||
|
% (repo, rev, (proc.stderr or "").strip()[:200]))
|
||||||
|
return [p for p in proc.stdout.splitlines()
|
||||||
|
if p.startswith("tools/") and p.endswith(".py")]
|
||||||
|
|
||||||
|
|
||||||
|
def blob(repo, rev, path):
|
||||||
|
"""The bytes of `path` at `rev`, or None when the tree does not hold it."""
|
||||||
|
proc = subprocess.run(["git", "-C", repo, "show", "%s:%s" % (rev, path)],
|
||||||
|
capture_output=True)
|
||||||
|
if proc.returncode != 0:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
return proc.stdout.decode("utf-8")
|
||||||
|
except UnicodeDecodeError as exc:
|
||||||
|
raise CannotGrade("cannot read %s at %s: not UTF-8: %s" % (path, rev, exc))
|
||||||
|
|
||||||
|
|
||||||
|
def docstring_of(source):
|
||||||
|
"""The module docstring, or "" -- a file that will not parse has none."""
|
||||||
|
try:
|
||||||
|
return ast.get_docstring(ast.parse(source)) or ""
|
||||||
|
except SyntaxError:
|
||||||
|
return ""
|
||||||
|
|
||||||
|
|
||||||
|
def documented_commands(doc):
|
||||||
|
"""[(path, [flags])] for every command line in a docstring.
|
||||||
|
|
||||||
|
The argument run stops at a shell separator by construction (CMD_RE's
|
||||||
|
character class), so `--emit-baseline > tools/x.tsv` documents
|
||||||
|
`--emit-baseline` and not the redirect target.
|
||||||
|
"""
|
||||||
|
out = []
|
||||||
|
for line in (doc or "").splitlines():
|
||||||
|
for m in CMD_RE.finditer(line):
|
||||||
|
target, rest = m.group(1), m.group(2) or ""
|
||||||
|
flags = FLAG_RE.findall(rest)
|
||||||
|
out.append((target, flags))
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_target(repo, rev, target, tools):
|
||||||
|
"""The tree path a documented `x.py` names, or None.
|
||||||
|
|
||||||
|
`tools/x.py` resolves directly. A bare `x.py` resolves to `tools/x.py`,
|
||||||
|
because docstrings in this archive are written from inside `tools/` --
|
||||||
|
and only if that path is actually in the tree at `rev`, so a command naming
|
||||||
|
a file that does not exist stays TOOL_MISSING instead of resolving to
|
||||||
|
something plausible.
|
||||||
|
"""
|
||||||
|
cand = [target.lstrip("./")]
|
||||||
|
if "/" not in cand[0]:
|
||||||
|
cand.append("tools/" + cand[0])
|
||||||
|
for c in cand:
|
||||||
|
if c in tools:
|
||||||
|
return c
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def help_options(repo, rev, path):
|
||||||
|
"""(set of option strings, None) or (None, why-not) for one tool at `rev`.
|
||||||
|
|
||||||
|
The tool is run from a TEMPORARY COPY of the blob at `rev`, never from the
|
||||||
|
worktree: a docstring read at `R` must be adjudicated against the parser at
|
||||||
|
`R`, or the check silently grades the past against the present.
|
||||||
|
"""
|
||||||
|
source = blob(repo, rev, path)
|
||||||
|
if source is None:
|
||||||
|
return None, "not in the tree at %s" % rev
|
||||||
|
with tempfile.TemporaryDirectory() as td:
|
||||||
|
copy = os.path.join(td, os.path.basename(path))
|
||||||
|
with open(copy, "w", encoding="utf-8") as fh:
|
||||||
|
fh.write(source)
|
||||||
|
try:
|
||||||
|
proc = subprocess.run([sys.executable, copy, "--help"],
|
||||||
|
capture_output=True, text=True,
|
||||||
|
timeout=HELP_TIMEOUT, cwd=repo)
|
||||||
|
except subprocess.TimeoutExpired:
|
||||||
|
raise CannotGrade("cannot grade: %s --help at %s outlived %ds"
|
||||||
|
% (path, rev, HELP_TIMEOUT))
|
||||||
|
if proc.returncode != 0 and not proc.stdout.strip():
|
||||||
|
return None, "no usable --help (rc %d)" % proc.returncode
|
||||||
|
opts = set(OPT_RE.findall(proc.stdout))
|
||||||
|
if not opts:
|
||||||
|
return None, "no --help output"
|
||||||
|
return opts, None
|
||||||
|
|
||||||
|
|
||||||
|
def census(repo, rev, only=None):
|
||||||
|
"""The report: one row per documented command, plus the per-tool oracles."""
|
||||||
|
tools = tracked_tools(repo, rev)
|
||||||
|
if only:
|
||||||
|
tools = [t for t in tools if os.path.basename(t) in only]
|
||||||
|
rows, oracles = [], {}
|
||||||
|
for path in tools:
|
||||||
|
doc = docstring_of(blob(repo, rev, path) or "")
|
||||||
|
for target, flags in documented_commands(doc):
|
||||||
|
real = resolve_target(repo, rev, target, tools)
|
||||||
|
if real is None:
|
||||||
|
rows.append({"doc": path, "target": target, "verdict": "TOOL_MISSING",
|
||||||
|
"missing": [], "detail": "no such file at %s" % rev})
|
||||||
|
continue
|
||||||
|
if real not in oracles:
|
||||||
|
oracles[real] = help_options(repo, rev, real)
|
||||||
|
opts, why = oracles[real]
|
||||||
|
if opts is None:
|
||||||
|
rows.append({"doc": path, "target": real, "verdict": "NO_ORACLE",
|
||||||
|
"missing": [], "detail": why})
|
||||||
|
continue
|
||||||
|
missing = sorted(f for f in flags if f not in opts)
|
||||||
|
rows.append({"doc": path, "target": real,
|
||||||
|
"verdict": "FLAG_NOT_ACCEPTED" if missing else "OK",
|
||||||
|
"missing": missing, "detail": ""})
|
||||||
|
return {"rev": rev, "rows": rows, "oracles": len(oracles)}
|
||||||
|
|
||||||
|
|
||||||
|
def fatal_rows(rep):
|
||||||
|
return [r for r in rep["rows"] if r["verdict"] in ("FLAG_NOT_ACCEPTED", "TOOL_MISSING")]
|
||||||
|
|
||||||
|
|
||||||
|
def report(rep, out=sys.stdout):
|
||||||
|
for r in rep["rows"]:
|
||||||
|
if r["verdict"] == "OK":
|
||||||
|
continue
|
||||||
|
miss = (" missing=%s" % (r["missing"],)) if r["missing"] else ""
|
||||||
|
print("%-18s %-30s -> %-26s %s%s"
|
||||||
|
% (r["verdict"], r["doc"], r["target"], r["detail"], miss), file=out)
|
||||||
|
counts = {}
|
||||||
|
for r in rep["rows"]:
|
||||||
|
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
|
||||||
|
print("doc commands at %s: %d command(s) over %d oracle(s) -- %s"
|
||||||
|
% (rep["rev"], len(rep["rows"]), rep["oracles"],
|
||||||
|
", ".join("%s %d" % (k, counts[k]) for k in sorted(counts)) or "none"))
|
||||||
|
|
||||||
|
|
||||||
|
# ------------------------------------------------------------------ selftest
|
||||||
|
|
||||||
|
def selftest():
|
||||||
|
"""Fixtures for the reader, the oracle and both verdicts, on one document."""
|
||||||
|
cells = []
|
||||||
|
|
||||||
|
def cell(label, ok, detail=None):
|
||||||
|
cells.append((label, bool(ok), detail))
|
||||||
|
|
||||||
|
doc = (
|
||||||
|
"USAGE\n"
|
||||||
|
" python3 selfcheck_tool.py --repo . --fail --baseline b.tsv\n"
|
||||||
|
" python3 tools/selfcheck_tool.py --emit-baseline > tools/b.tsv\n"
|
||||||
|
" python3 gone.py --whatever\n"
|
||||||
|
" python3 selfcheck_tool.py --selftest # 12 cells, and a \n"
|
||||||
|
" prose line that merely mentions --not-a-command in passing\n")
|
||||||
|
|
||||||
|
cmds = documented_commands(doc)
|
||||||
|
cell("every python3 command line is read", len(cmds) == 4, len(cmds))
|
||||||
|
cell("flags are read in order, values are not flags",
|
||||||
|
cmds[0] == ("selfcheck_tool.py", ["--repo", "--fail", "--baseline"]),
|
||||||
|
cmds[0] if cmds else None)
|
||||||
|
cell("a redirect target is not read as an argument",
|
||||||
|
cmds[1] == ("tools/selfcheck_tool.py", ["--emit-baseline"]), cmds[1])
|
||||||
|
cell("a bare filename is read as written", cmds[2][0] == "gone.py", cmds[2])
|
||||||
|
cell("prose mentions are not commands", all(c[0].endswith(".py") for c in cmds))
|
||||||
|
|
||||||
|
opts = {"-h", "--repo", "--fail", "--baseline", "--emit-baseline", "--selftest"}
|
||||||
|
cell("an accepted flag is not missing", not [f for f in cmds[0][1] if f not in opts])
|
||||||
|
cell("a documented flag outside the oracle is named",
|
||||||
|
[f for f in ["--renamed"] if f not in opts] == ["--renamed"])
|
||||||
|
|
||||||
|
# resolve_target, on a tree that holds only tools/selfcheck_tool.py
|
||||||
|
tree = ["tools/selfcheck_tool.py"]
|
||||||
|
cell("tools/x.py resolves in the tree",
|
||||||
|
resolve_target(".", "HEAD", "tools/selfcheck_tool.py", tree) ==
|
||||||
|
"tools/selfcheck_tool.py")
|
||||||
|
cell("a bare x.py resolves to tools/x.py",
|
||||||
|
resolve_target(".", "HEAD", "selfcheck_tool.py", tree) ==
|
||||||
|
"tools/selfcheck_tool.py")
|
||||||
|
cell("a bare x.py that is NOT in the tree stays unresolved",
|
||||||
|
resolve_target(".", "HEAD", "gone.py", tree) is None)
|
||||||
|
|
||||||
|
# argparse's own text is the oracle: check the reader against a real parser
|
||||||
|
src = (
|
||||||
|
"import argparse\n"
|
||||||
|
"ap = argparse.ArgumentParser()\n"
|
||||||
|
"ap.add_argument('--repo', default='.')\n"
|
||||||
|
"ap.add_argument('--long-only-name')\n"
|
||||||
|
"ap.print_help()\n")
|
||||||
|
cell("--help text is harvested into option strings",
|
||||||
|
set(OPT_RE.findall(src)) >= {"--repo", "--long-only-name"})
|
||||||
|
harvested = set(OPT_RE.findall("usage: x [-h] [--fail] [--json JSON]\n"))
|
||||||
|
cell("bracketed usage options are harvested", harvested >= {"-h", "--fail", "--json"})
|
||||||
|
|
||||||
|
# verdicts
|
||||||
|
rep = {"rev": "HEAD", "oracles": 1, "rows": [
|
||||||
|
{"doc": "tools/a.py", "target": "tools/a.py", "verdict": "OK",
|
||||||
|
"missing": [], "detail": ""},
|
||||||
|
{"doc": "tools/a.py", "target": "tools/a.py",
|
||||||
|
"verdict": "FLAG_NOT_ACCEPTED", "missing": ["--gone"], "detail": ""},
|
||||||
|
{"doc": "tools/a.py", "target": "gone.py",
|
||||||
|
"verdict": "TOOL_MISSING", "missing": [], "detail": ""},
|
||||||
|
{"doc": "tools/a.py", "target": "tools/a.py", "verdict": "NO_ORACLE",
|
||||||
|
"missing": [], "detail": "no --help output"},
|
||||||
|
]}
|
||||||
|
cell("both faults are graded, NO_ORACLE is not",
|
||||||
|
[r["verdict"] for r in fatal_rows(rep)] ==
|
||||||
|
["FLAG_NOT_ACCEPTED", "TOOL_MISSING"],
|
||||||
|
[r["verdict"] for r in fatal_rows(rep)])
|
||||||
|
|
||||||
|
# the reader applied to this file's own docstring: every command that names
|
||||||
|
# THIS tool must use a flag this tool accepts. A command quoting a sibling
|
||||||
|
# (`checkin_check.py --baseline ...`) is prose about that sibling and is
|
||||||
|
# checked against its own `--help`, not against this one's vocabulary.
|
||||||
|
own = [c for c in documented_commands(__doc__)
|
||||||
|
if os.path.basename(c[0]) == os.path.basename(__file__)]
|
||||||
|
cell("this docstring documents commands of its own", len(own) >= 2, len(own))
|
||||||
|
known_here = {"--repo", "--rev", "--tools", "--json", "--fail", "--selftest",
|
||||||
|
"-h", "--help"}
|
||||||
|
bad = sorted({f for _, fs in own for f in fs if f not in known_here})
|
||||||
|
cell("every flag this docstring documents is one this tool accepts", not bad, bad)
|
||||||
|
|
||||||
|
failed = [(lab, det) for lab, ok, det in cells if not ok]
|
||||||
|
print("selftest: %d cells, %d passed, %d failed"
|
||||||
|
% (len(cells), len(cells) - len(failed), len(failed)))
|
||||||
|
for lab, det in failed:
|
||||||
|
print(" FAILED: %s" % lab)
|
||||||
|
if det is not None:
|
||||||
|
print(" got: %r" % (det,))
|
||||||
|
return 1 if failed else 0
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------- main
|
||||||
|
|
||||||
|
def run(argv):
|
||||||
|
ap = argparse.ArgumentParser(
|
||||||
|
description="grade the commands a tool's docstring tells a reader to run")
|
||||||
|
ap.add_argument("--repo", default=".", help="the work tree (default: .)")
|
||||||
|
ap.add_argument("--rev", default="HEAD",
|
||||||
|
help="revision the docstrings AND the oracles are read from")
|
||||||
|
ap.add_argument("--tools", default=None,
|
||||||
|
help="comma-separated basenames to limit the scan to")
|
||||||
|
ap.add_argument("--json", default=None, help="write the full report here")
|
||||||
|
ap.add_argument("--fail", action="store_true",
|
||||||
|
help="exit 1 when a documented command is rejected")
|
||||||
|
ap.add_argument("--selftest", action="store_true",
|
||||||
|
help="grade this tool's own cells and exit")
|
||||||
|
args = ap.parse_args(argv)
|
||||||
|
|
||||||
|
if args.selftest:
|
||||||
|
return selftest()
|
||||||
|
|
||||||
|
only = [t.strip() for t in args.tools.split(",") if t.strip()] if args.tools else None
|
||||||
|
try:
|
||||||
|
if not os.path.isdir(args.repo):
|
||||||
|
raise CannotGrade("cannot read repo: %s: no such directory" % args.repo)
|
||||||
|
rep = census(args.repo, args.rev, only)
|
||||||
|
except CannotGrade as exc:
|
||||||
|
print(str(exc), file=sys.stderr)
|
||||||
|
return 2
|
||||||
|
|
||||||
|
report(rep)
|
||||||
|
if args.json:
|
||||||
|
with open(args.json, "w", encoding="utf-8") as fh:
|
||||||
|
json.dump(rep, fh, indent=1, sort_keys=True)
|
||||||
|
fh.write("\n")
|
||||||
|
|
||||||
|
bad = fatal_rows(rep)
|
||||||
|
if bad and args.fail:
|
||||||
|
print("FAIL: %d documented command(s) this tool would reject" % len(bad))
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(run(sys.argv[1:]))
|
||||||
Reference in new issue
Block a user