volt
c3d70567a6
The gate had six legs and none of them read a docstring's own command line.
doc_command_check.py existed since 838750c but only as a tool: nothing on the
gate ran it, so a documented flag its parser never learned could land and
nobody read it. This is that leg.
doc-commands tools/doc_command_check.py --rev WORKTREE --fail
WORKTREE, not HEAD: the commit window is where a docstring is
edited. Measured at the worktree this lands in: 67 commands
over 19 oracle tools, 67 OK, rc 0.
doc_command_check.py learns --rev WORKTREE: tracked paths from `git ls-files`
(an untracked tools/x.py is not part of the commit this mode grades), bytes
from disk. Outside a work tree it is rc 2, not an empty clean run. Selftest
15 -> 18 cells: one document edited on disk but not committed is rc 0 at HEAD
and rc 1 at WORKTREE, and the flag it names is the edited one.
Three tools stopped defaulting to a fixed path on this host. verify.py,
selftest_sweep.py, write_probe.py, citation_resolve_check.py,
path_claim_check.py and doc_census_check.py now resolve DEFAULT_REPO from the
file's own location, so a bare run in a clone grades the clone.
Three selftests refuse unreadable input at rc 2 instead of reporting it as
failed content: tool_map (not a work tree), write_probe (no table), and
citation_resolve_check (no corpus). Each cell below them graded `repo`, so an
absent tree read as "your table is wrong".
Evidence, all measured on this tree before the commit:
verify.py --selftest 55 cells, 55 passed (docstring says 55)
doc_command_check.py --selftest 18 cells, 18 passed (baseline pinned 18)
tool_map.py --repo . --rev WORKTREE --check docs/TOOLS.md rc 0
verify.py --repo . 7 legs, 6 pass, 1 fault -- selftest-sweep,
tool_map's drift cell, which renders HEAD
and so cannot pass inside a commit window
tools/selftest_baseline.tsv re-pins doc_command_check.py 15 -> 18 cells.
964 lines
50 KiB
Python
964 lines
50 KiB
Python
#!/usr/bin/env python3
|
|
"""verify.py -- the HOME the index gate did not have: one command, seven legs,
|
|
one exit code, and a row per leg so a partial run cannot look like a clean one.
|
|
|
|
WHY THIS EXISTS
|
|
On 2026-10-18 this lane told Alexander, three ticks running, to "wire
|
|
`queue_index --check` into the tick's check.sh". There is no `check.sh` in
|
|
this repo and no `scripts/` directory at all -- `git ls-files` shows `tools/`
|
|
and nothing else. The advice named a target that does not exist, which is why
|
|
it was never adopted. The gate did not need a decision about policy; it needed
|
|
a home. This is that home, and it is a task, not a policy call: nothing here
|
|
obliges any lane to run it, and nothing is enforced until a lane does.
|
|
|
|
The second reason is narrower and this lane's own: three separate counts of
|
|
"which commits left the index stale" came out wrong in one week, each from a
|
|
hand-assembled probe loop. A gate whose legs are remembered is a gate that
|
|
gets run selectively. Fixed order, one rc -- that is the whole design.
|
|
|
|
The third reason is measured, not argued. On 2026-10-19T09:42Z this lane put a
|
|
contract-breaking check-in into a clone of HEAD -- `checkins/2026-10-19-volt-02.md`,
|
|
a second same-day file carrying a suffix -- and ran both instruments on it:
|
|
`checkin_check.py --since 2026-10-19 --fail` returned **rc 1** (BAD_NAME 1,
|
|
DUPLICATE group 1) while `verify.py --fast` returned **rc 0** on the same tree,
|
|
"gate: OK -- 2 of 2 graded leg(s) passed". Three legs that grade the queue, its
|
|
append discipline and the tools' selftests said nothing about the one artifact
|
|
every lane writes every tick. A fourth leg closes that hole.
|
|
|
|
The fifth is the same shape one directory over, and it is here for a reason
|
|
that was measured before it was written. On 2026-10-19 this lane built
|
|
`reply_check.py` and deliberately did NOT wire it: the archive has one live
|
|
fault -- `replies/2026-10-15-to-timmy-58.md` is named for Timmy and headed
|
|
"to Merlin" -- it is Merlin's file and Merlin's to repair, and that tick's
|
|
reasoning was that a gate wired to a defect its owner has not fixed yet is a
|
|
gate that gets switched off in a week. That reasoning is right about a gate
|
|
with no ledger and wrong about this repo, which already solved the same
|
|
problem for check-ins: `tools/reply_baseline.tsv` states the one tolerance,
|
|
the fault stays visible in every report as known-faulting, and a second
|
|
mis-addressed reply -- or a new fault class on the known file -- is rc 1 the
|
|
day it lands. Waiting for another lane to edit its own file was the only
|
|
alternative, and it had no date on it.
|
|
|
|
WHAT IT RUNS -- one row per leg, cheapest first, never in parallel
|
|
checkin tools/checkin_check.py --fail --baseline tools/checkin_baseline.tsv
|
|
the WHOLE check-in archive against F1-F5 (names, duplicates,
|
|
unknown handles, dates, emptiness), where the ledger states the
|
|
faults the archive already carried; only a class it does not
|
|
name is a fault. `--checkin-day DATE` adds a `--since` floor
|
|
reply-headings tools/reply_check.py --baseline tools/reply_baseline.tsv
|
|
every reply's heading against its own filename (addressee and
|
|
number), on the same ledger rule: `replies/` is the second
|
|
largest artifact directory and, until 2026-10-19, nothing here
|
|
read it
|
|
index-current tools/queue_index.py --check docs/queue-index.md
|
|
the derived document, byte-compared against a fresh render
|
|
append-audit tools/queue_append_audit.py --guard --to HEAD
|
|
the newest queue-touching commit, graded in ITS OWN tree
|
|
doc-commands tools/doc_command_check.py --rev WORKTREE --fail
|
|
every command a tool's docstring tells a reader to run: each
|
|
flag must be one that tool's own `--help` prints. READ-ONLY --
|
|
it runs `--help` and nothing else. WORKTREE, not HEAD, so the
|
|
document being written is the document graded; this leg did
|
|
not exist before 2026-10-22, and every docstring command in
|
|
the archive was prose until it did
|
|
selftest-sweep tools/selftest_sweep.py
|
|
every tool's --selftest plus its two read-refusal axes
|
|
(skip with --fast; the row is still printed and counted)
|
|
write-axis tools/write_probe.py
|
|
every tracked tool's `--json PATH` pointed at a target this
|
|
host cannot write: rc 2, one line naming the path, never a
|
|
traceback -- and, since 2026-10-21, every caller-path write
|
|
flag other than `--json` (`--write`, `--save-vocab`) on the
|
|
same rule. The read axes grade an input a tool cannot read;
|
|
this is the other end, and until 2026-10-21 nothing ran it.
|
|
|
|
The legs are ordered by cost, and a leg is never skipped because an earlier
|
|
one failed: a stale index and a broken selftest are different defects and the
|
|
run should name both in one pass.
|
|
|
|
WHY THE CHECK-IN LEG NOW GRADES THE ARCHIVE, NOT ONE DAY
|
|
Grading the whole archive was, until 2026-10-19, a permanently red gate: at
|
|
`e98a34b` the unbaselined run exits 1 on every one of the archive's pre-rule
|
|
rows (25 BAD_NAME, 10 DUPLICATE groups, all dated 2026-09-24 through
|
|
2026-10-07) -- that list is now a measured artifact, `tools/checkin_baseline.tsv`,
|
|
one row per file rather than a number in this docstring. The first version of this
|
|
leg therefore graded the archive's **newest day** only, on the reasoning that
|
|
it is the only day on which a new fault can appear. That reasoning is false,
|
|
and it was measured false rather than argued: a check-in added today under a
|
|
**backdated** filename -- `checkins/2026-10-02-volt-b.md`, which is what a typo
|
|
in the date half produces -- is not on the newest day, so the floor excluded
|
|
it. On a clone of `e98a34b` carrying that one file, `verify.py --fast` printed
|
|
`checkin_check: 1 files, 0 with a fault` and exited **0**, while the same
|
|
tree's whole-archive run exited **1** naming the file.
|
|
|
|
The fix keeps both properties -- history is not re-litigated, nothing is
|
|
invisible. `tools/checkin_baseline.tsv` lists the faults the archive already
|
|
carried, one `<filename>\t<CLASS[,CLASS]>` row each, and `--baseline` grades
|
|
every file against it: a listed class is KNOWN, anything else is a fault. A
|
|
file that is not in the ledger at all is a fault wherever its date falls, and a
|
|
file that gains a class it did not have is a fault too -- the ledger states what
|
|
was tolerated, not that a path is exempt. A missing or malformed ledger is
|
|
rc 2 (cannot grade), never a silent pass, and stale entries (file gone, class
|
|
cleared) are reported and never make rc 1. `--checkin-day DATE` survives as an
|
|
explicit narrowing floor for an operator who wants one.
|
|
|
|
THE EXIT CONTRACT -- the house three codes
|
|
0 every graded leg passed
|
|
1 at least one leg ran and graded a fault
|
|
2 at least one leg could not grade -- a missing tool, a timeout, an
|
|
unexpected exit code, an unreadable repo -- or no leg was runnable at all
|
|
|
|
rc 2 OUTRANKS rc 1 deliberately. rc 1 means "I ran and I graded a fault";
|
|
a run with an ungradable leg has not made that claim about the whole gate,
|
|
and folding the two (as the 15:16Z correction caught `queue_append_audit`
|
|
doing with ungradable commits) is how a "3 stale" reading hides a "2 could
|
|
not be read". The summary always prints both counts, so precedence never
|
|
erases a fault -- it only refuses to certify the run that found it.
|
|
|
|
A skipped leg is not a pass and not a fault: it is counted as `skipped` and
|
|
printed as SKIP. `--fast` on a green tree exits 0 with one row SKIP, and the
|
|
summary says so rather than pretending the sweep ran.
|
|
|
|
TOOLS ARE RESOLVED FROM --repo, NOT FROM THIS FILE'S DIRECTORY
|
|
`verify.py --repo /path/to/worktree` runs THAT tree's `queue_index.py`,
|
|
`queue_append_audit.py` and `selftest_sweep.py`. That is the instrument
|
|
lesson the 15:16Z correction paid for: a gate run in a working tree at HEAD
|
|
says nothing about a cited revision, so grading a revision means grading its
|
|
own tools from its own checkout.
|
|
|
|
--queue and --index stay REPO-RELATIVE (defaults WORK-QUEUE.md and
|
|
docs/queue-index.md) because `queue_append_audit` reads them with
|
|
`git show <rev>:<path>`, which has no absolute-path form.
|
|
|
|
--repo IS A WORK TREE, ASKED OF GIT
|
|
Until 2026-10-19 the repo test was `os.path.isdir(<repo>/.git)`, and it was
|
|
wrong in both directions. A LINKED worktree's `.git` is a 47-byte file holding
|
|
`gitdir: ...`, so from `git worktree add --detach <path>` the gate returned
|
|
**rc 2 "not a git repository"** against a checkout whose every leg passes --
|
|
measured that day on `cf5101f`: the old test rc 2, the test that follows rc 0
|
|
with every leg then wired green (`selftest-sweep` included). That made the gate
|
|
unrunnable in the one sandbox a lane gets for free and forced the choice
|
|
between grading unsandboxed in the shared clone and not running it at all.
|
|
An empty directory named `.git` was the same test's false positive: it passed
|
|
while git itself refused the path. `resolve_repo` now asks
|
|
`git rev-parse --show-toplevel`, which succeeds in every work tree (main,
|
|
linked, submodule) and fails in a bare repository or a bare `.git` directory.
|
|
The path must still be the ROOT: a subdirectory is rc 2 naming the root,
|
|
because widening `--repo` silently would grade a tree nobody named.
|
|
|
|
`--selftest` fixture corpus, 55 cells
|
|
"""
|
|
|
|
import argparse
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
import time
|
|
|
|
QUEUE_DEFAULT = "WORK-QUEUE.md"
|
|
INDEX_DEFAULT = "docs/queue-index.md"
|
|
CHECKINS_DEFAULT = "checkins"
|
|
|
|
PASS, FAULT, UNGRADABLE, SKIP = "PASS", "FAULT", "UNGRADABLE", "SKIP"
|
|
|
|
# Legs, in the fixed order they are printed and run. `tool` is resolved inside
|
|
# the repo being graded. `slow` marks the leg --fast drops.
|
|
LEGS = (
|
|
{"key": "checkin", "tool": "checkin_check.py", "slow": False,
|
|
"what": "grade the newest day's check-ins against the naming and "
|
|
"duplicate rules"},
|
|
{"key": "reply-headings", "tool": "reply_check.py", "slow": False,
|
|
"what": "grade every reply's heading against its own filename"},
|
|
{"key": "index-current", "tool": "queue_index.py", "slow": False,
|
|
"what": "render the index and byte-compare it with the document"},
|
|
{"key": "append-audit", "tool": "queue_append_audit.py", "slow": False,
|
|
"what": "grade the newest queue-touching commit in its own tree"},
|
|
{"key": "doc-commands", "tool": "doc_command_check.py", "slow": False,
|
|
"what": "every command a tool's docstring tells a reader to run: its "
|
|
"flags must be options that tool's --help accepts"},
|
|
{"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True,
|
|
"what": "every tool's --selftest and both read-refusal axes"},
|
|
{"key": "write-axis", "tool": "write_probe.py", "slow": False,
|
|
"what": "a --json target this host cannot write: rc 2, one line naming "
|
|
"it, never a traceback -- and the same rule for every caller-path "
|
|
"write flag (--write, --save-vocab)"},
|
|
)
|
|
|
|
# A detail line is a courtesy, not the verdict -- but a verdict with no line
|
|
# beside it is what the read-refusal axis calls SILENT, so it is graded too.
|
|
#
|
|
# STRONG outranks WEAK, and both outrank a per-file SKIP note. That tiering is
|
|
# not cosmetic: `checkin_check.py` prints " SKIP README.md (the directory's own
|
|
# README: not a check-in)" as its second line on every real run, "not a " is a
|
|
# WEAK hint, and before the tiering that note was carried onto a FAULT row --
|
|
# the row said FAULT beside a line about the README while the tool's own
|
|
# "FAIL: 2 file(s) violate the check-in contract" sat further down, unpicked.
|
|
STRONG_HINTS = ("STALE:", "FAIL:", "FAILED", "fatal", "TIMEOUT", "NO_",
|
|
"cannot", "error", "Error", "refus",
|
|
# write_probe's fault block (2026-10-21): each line begins
|
|
# "FAULT <tool> ...", and that line sits above the probe table
|
|
# precisely so this function reaches it before the first healthy
|
|
# row. A SILENT fault's own detail carries no other marker, so
|
|
# without this token the block would be skipped for the row that
|
|
# names a tool with nothing wrong with it.
|
|
"FAULT")
|
|
WEAK_HINTS = ("FAIL", "not a ")
|
|
|
|
# A tool's concluding line: `reply_check: 192 file(s), 0 new fault(s) ...`.
|
|
# Hyphenated labels (`read-refusal:`, `malformed-input:`) are deliberately not
|
|
# summaries -- they are axis lines inside a tool's report.
|
|
SUMMARY_RE = re.compile(r"^[a-z][a-z0-9_]*: ")
|
|
|
|
# The selftest size this file's docstring claims. Cell 27 grades that claim
|
|
# against the count that actually ran, because on 2026-10-19 the count left
|
|
# this exact defect behind: the docstring and the run said 50 while the
|
|
# argparse help string still said 42, and nothing failed.
|
|
_DOC_CLAIM = re.search(r"fixture corpus, (\d+) cells", __doc__ or "")
|
|
DOC_CELLS = int(_DOC_CLAIM.group(1)) if _DOC_CLAIM else 0
|
|
|
|
|
|
def pick_detail(text, limit=150, summary=False):
|
|
"""First line that names a reason, else the last line that says anything.
|
|
|
|
A summary line that reports ZERO faults is not a reason: `selftest_sweep`
|
|
prints `sweep: 15 tools, 15 OK, 0 fatal` even on the run where its input
|
|
axes went red, and that line was being carried onto a FAULT row -- the row
|
|
said FAULT beside the words "0 fatal". Zero-count summaries are therefore
|
|
only reachable on the second pass (a clean run still gets its sweep line),
|
|
and any line that names a real reason outranks them.
|
|
|
|
`summary=True` is the clean-run answer, and it is a different question. On a
|
|
PASS row the useful detail is the tool's own concluding line, not the first
|
|
line in its report that happens to contain "refus" -- `reply_check` prints
|
|
one heading excerpt per advisory row, and one of those headings contains the
|
|
word "refuses", which was reaching the row as if it were the reason. On a
|
|
fault row the reason still wins, because that is what a reader needs there.
|
|
"""
|
|
lines = [ln.strip() for ln in text.splitlines() if ln.strip()]
|
|
kept = [ln for ln in lines if not ln.startswith("SKIP ")] or lines
|
|
if summary:
|
|
sums = [ln for ln in kept if SUMMARY_RE.match(ln)]
|
|
if sums:
|
|
return sums[-1][:limit]
|
|
hinted = [ln for ln in kept if any(h in ln for h in STRONG_HINTS + WEAK_HINTS)]
|
|
# A class row that printed zero is not a reason and not a count of anything:
|
|
# `reply_check` prints its whole class table on every run, so `NO_HEADER 0`
|
|
# was reaching the detail line of a PASS row ahead of the summary. Same
|
|
# failure the " 0 fatal" filter catches, one shape further out.
|
|
class_row = re.compile(r"^[A-Z][A-Z0-9_]+ +(\d+)$")
|
|
zero_row = re.compile(r"^[A-Z][A-Z0-9_]+ +0$")
|
|
reasons = [ln for ln in hinted
|
|
if " 0 fatal" not in ln and " 0 fault" not in ln
|
|
and not zero_row.match(ln)]
|
|
# Nor is a nonzero class row a "reason": `NO_RECIPIENT_IN_HEADING 7` is a
|
|
# count, and `NO_` in its name is not a diagnostic marker. A hint counts
|
|
# only on a line that is not a class table row.
|
|
strong = [ln for ln in reasons
|
|
if any(h in ln for h in STRONG_HINTS) and not class_row.match(ln)]
|
|
# A tool's own verdict line outranks every other reason in its report: when
|
|
# `reply_check` names a mis-addressed file it also prints one heading excerpt
|
|
# per row, and an excerpt containing "refuses" was beating the FAIL line.
|
|
verdict = [ln for ln in strong if ln.startswith(("FAIL:", "STALE:"))]
|
|
if verdict:
|
|
return verdict[0][:limit]
|
|
if strong:
|
|
return strong[0][:limit]
|
|
if reasons:
|
|
return reasons[0][:limit]
|
|
if hinted:
|
|
return hinted[0][:limit]
|
|
return kept[-1][:limit] if kept else ""
|
|
|
|
|
|
def resolve_repo(path):
|
|
"""`--repo` is a git WORK TREE ROOT. Return (repo, why-not), asking git.
|
|
|
|
Until 2026-10-19 this test was `os.path.isdir(<repo>/.git)`, which is the
|
|
wrong question in both directions, and both were measured on that day:
|
|
|
|
* a LINKED worktree's `.git` is a 47-byte FILE holding `gitdir: ...`, so
|
|
`git worktree add --detach` produced a checkout every leg would have
|
|
graded and the gate refused with rc 2 "not a git repository" -- the one
|
|
sandbox a lane gets for free was the one place the operator gate could
|
|
not run, and the alternatives were grading unsandboxed in the shared
|
|
clone or not running the gate at all;
|
|
* an empty directory named `.git` passed the same test on a path git
|
|
itself refuses, so the refusal was also not worth the name.
|
|
|
|
`git rev-parse --show-toplevel` answers the real question: it succeeds in
|
|
every work tree (main, linked, submodule) and fails in a bare repository or
|
|
a directory that merely holds a `.git` entry. The path must still be the
|
|
ROOT -- a subdirectory is rc 2 with the root named, because widening
|
|
`--repo` silently would grade a different tree than the caller named.
|
|
"""
|
|
repo = os.path.abspath(path)
|
|
if not os.path.isdir(repo):
|
|
return None, "cannot read repo: %s is not a directory" % repo
|
|
try:
|
|
proc = subprocess.run(["git", "-C", repo, "rev-parse", "--show-toplevel"],
|
|
capture_output=True, text=True)
|
|
except OSError as exc:
|
|
return None, ("cannot read repo: %s (git is not runnable: %s)"
|
|
% (repo, exc))
|
|
top = (proc.stdout or "").strip()
|
|
if proc.returncode != 0 or not top:
|
|
why = [l for l in (proc.stderr or "").strip().splitlines() if l.strip()]
|
|
return None, ("cannot read repo: %s is not a git work tree (%s)"
|
|
% (repo, why[-1] if why else
|
|
"git rev-parse rc %d" % proc.returncode))
|
|
if os.path.realpath(top) != os.path.realpath(repo):
|
|
return None, ("cannot read repo: %s is inside the work tree rooted at "
|
|
"%s; --repo takes the root" % (repo, top))
|
|
return repo, None
|
|
|
|
|
|
def newest_checkin_day(repo):
|
|
"""The newest `<YYYY-MM-DD>` prefix in checkins/ -> (day, why-not).
|
|
|
|
Not the graded window any more -- the leg grades the whole archive against
|
|
its own ledger -- but still the cheapest way to answer "is there anything
|
|
here to grade at all", which is what makes a repo with no `checkins/`
|
|
directory (or no dated file in it) ungradable with a message naming which of
|
|
the two it is, rather than a spawn and an rc 2 from further down.
|
|
"""
|
|
d = os.path.join(repo, CHECKINS_DEFAULT)
|
|
if not os.path.isdir(d):
|
|
return None, "no check-in directory: %s" % d
|
|
days = sorted(m.group(1) for m in
|
|
(re.match(r"^(\d{4}-\d{2}-\d{2})-", n) for n in os.listdir(d))
|
|
if m)
|
|
if not days:
|
|
return None, "no dated check-in files in %s" % d
|
|
return days[-1], None
|
|
|
|
|
|
def leg_argv(key, repo, queue, index, checkin_day=None):
|
|
if key == "checkin":
|
|
# The ledger is a file in the tree being graded, so a revision is graded
|
|
# with the tolerance it committed -- not with this host's idea of one.
|
|
argv = [sys.executable, os.path.join(repo, "tools", "checkin_check.py"),
|
|
"--repo", repo, "--fail",
|
|
"--baseline", os.path.join(repo, "tools", "checkin_baseline.tsv")]
|
|
if checkin_day:
|
|
argv += ["--since", checkin_day]
|
|
return argv
|
|
if key == "reply-headings":
|
|
# The ledger comes from the tree being graded, for the same reason the
|
|
# check-in ledger does: a revision is graded with the tolerance it
|
|
# committed, not with this host's idea of one.
|
|
return [sys.executable, os.path.join(repo, "tools", "reply_check.py"),
|
|
"--repo", repo, "--baseline",
|
|
os.path.join(repo, "tools", "reply_baseline.tsv")]
|
|
if key == "index-current":
|
|
return [sys.executable, os.path.join(repo, "tools", "queue_index.py"),
|
|
"--repo", repo, "--queue", queue, "--check", index]
|
|
if key == "append-audit":
|
|
return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"),
|
|
"--repo", repo, "--queue", queue, "--index", index,
|
|
"--guard", "--to", "HEAD"]
|
|
if key == "doc-commands":
|
|
# WORKTREE, not HEAD: the commit window is where a docstring is edited,
|
|
# and a leg that read HEAD would grade the last commit while the lane
|
|
# was writing the next one. The mode is the tool's own, measured there.
|
|
return [sys.executable, os.path.join(repo, "tools",
|
|
"doc_command_check.py"),
|
|
"--repo", repo, "--rev", "WORKTREE", "--fail"]
|
|
if key == "selftest-sweep":
|
|
return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"),
|
|
"--repo", repo, "--quiet"]
|
|
if key == "write-axis":
|
|
# The probe writes only under its own TemporaryDirectory; the repo is
|
|
# the tree it discovers tools from, so a revision is graded against the
|
|
# guards it committed rather than this host's copy.
|
|
return [sys.executable, os.path.join(repo, "tools", "write_probe.py"),
|
|
"--repo", repo]
|
|
raise KeyError(key)
|
|
|
|
|
|
def _elapsed_ms(started):
|
|
"""Milliseconds elapsed since `started`; the integer clock every row carries.
|
|
|
|
Extracted so run_leg() does not recompute `int((time.time() - started) * 1000)`
|
|
at each of its exit paths -- the DRY collapse that previously ran the same
|
|
expression three times (timeout, OSError, normal verdict).
|
|
"""
|
|
return int((time.time() - started) * 1000)
|
|
|
|
|
|
def run_leg(leg, repo, queue, index, timeout, checkin_day=None):
|
|
"""One leg -> one row. rc 0/1/2 are graded; anything else is ungradable."""
|
|
tool = os.path.join(repo, "tools", leg["tool"])
|
|
started = time.time()
|
|
row = {"key": leg["key"], "tool": leg["tool"], "verdict": None,
|
|
"rc": None, "detail": "", "what": leg["what"]}
|
|
if not os.path.exists(tool):
|
|
row.update(verdict=UNGRADABLE, detail="no such tool: %s" % tool)
|
|
row["ms"] = 0
|
|
return row
|
|
day = checkin_day
|
|
if leg["key"] == "checkin":
|
|
# Presence is still pre-checked here: the tool's own rc-2 message names
|
|
# the directory, but this row can say so without spawning a process.
|
|
newest, why = newest_checkin_day(repo)
|
|
if newest is None:
|
|
row.update(verdict=UNGRADABLE, detail=why)
|
|
row["ms"] = 0
|
|
return row
|
|
try:
|
|
proc = subprocess.run(leg_argv(leg["key"], repo, queue, index, day),
|
|
cwd=repo, capture_output=True, text=True,
|
|
timeout=timeout)
|
|
except (subprocess.TimeoutExpired, OSError) as exc:
|
|
# Both transport failures collapse to the same ungradable shape; only the
|
|
# detail differs -- timeout names the deadline, OSError names the error.
|
|
if isinstance(exc, subprocess.TimeoutExpired):
|
|
detail = "TIMEOUT after %ss" % timeout
|
|
else:
|
|
detail = "could not run: %s" % exc
|
|
row.update(verdict=UNGRADABLE, detail=detail)
|
|
else:
|
|
out = (proc.stdout or "") + (proc.stderr or "")
|
|
row["rc"] = proc.returncode
|
|
row["detail"] = pick_detail(out, summary=(proc.returncode == 0))
|
|
if proc.returncode == 0:
|
|
row["verdict"] = PASS
|
|
elif proc.returncode == 1:
|
|
row["verdict"] = FAULT
|
|
elif proc.returncode == 2:
|
|
row["verdict"] = UNGRADABLE
|
|
else:
|
|
# rc 3 is retired in this repo (one meaning, two codes); any other
|
|
# code is a tool that left the contract, and the run cannot grade
|
|
# through it.
|
|
row["verdict"] = UNGRADABLE
|
|
row["detail"] = "unexpected rc %d: %s" % (proc.returncode, row["detail"])
|
|
row["ms"] = _elapsed_ms(started)
|
|
return row
|
|
|
|
|
|
def counts(rows):
|
|
c = {PASS: 0, FAULT: 0, UNGRADABLE: 0, SKIP: 0}
|
|
for r in rows:
|
|
c[r["verdict"]] += 1
|
|
return c
|
|
|
|
|
|
def decide(rows):
|
|
"""The precedence, in one place: ungradable outranks fault outranks clean."""
|
|
c = counts(rows)
|
|
if c[UNGRADABLE]:
|
|
return 2
|
|
if c[FAULT]:
|
|
return 1
|
|
return 0
|
|
|
|
|
|
def report(rows, repo, quiet=False):
|
|
c = counts(rows)
|
|
if not quiet:
|
|
print("verify -- repo: %s" % repo)
|
|
for r in rows:
|
|
rc = "-" if r["rc"] is None else str(r["rc"])
|
|
print(" %-16s %-10s rc=%-2s %-6s %s"
|
|
% (r["key"], r["verdict"], rc, "%dms" % r["ms"], r["detail"]))
|
|
print()
|
|
print("verify: %d legs, %d pass, %d fault, %d ungradable, %d skipped"
|
|
% (len(rows), c[PASS], c[FAULT], c[UNGRADABLE], c[SKIP]))
|
|
faulted = [r["key"] for r in rows if r["verdict"] == FAULT]
|
|
ungraded = [r["key"] for r in rows if r["verdict"] == UNGRADABLE]
|
|
if faulted:
|
|
print("fault: %s" % " ".join(faulted))
|
|
if ungraded:
|
|
print("ungradable: %s" % " ".join(ungraded))
|
|
rc = decide(rows)
|
|
if rc == 0:
|
|
print("gate: OK -- %d of %d graded leg(s) passed%s"
|
|
% (c[PASS], c[PASS] + c[FAULT],
|
|
"" if not c[SKIP] else
|
|
" (%d skipped: not a pass, not a fault)" % c[SKIP]))
|
|
elif rc == 1:
|
|
print("gate: FAULT -- %d leg(s) ran and graded a fault: %s"
|
|
% (c[FAULT], " ".join(faulted)))
|
|
else:
|
|
print("gate: NOT CERTIFIED -- %d leg(s) could not grade: %s"
|
|
% (c[UNGRADABLE], " ".join(ungraded)))
|
|
return rc
|
|
|
|
|
|
def main(argv=None):
|
|
ap = argparse.ArgumentParser(
|
|
description="Run the repo's gates as one command: the newest day's "
|
|
"check-ins, index currency, the append audit, and the "
|
|
"selftest sweep.")
|
|
ap.add_argument("--repo", default=".")
|
|
ap.add_argument("--queue", default=QUEUE_DEFAULT,
|
|
help="repo-relative queue path (default %s)" % QUEUE_DEFAULT)
|
|
ap.add_argument("--index", default=INDEX_DEFAULT,
|
|
help="repo-relative index document (default %s)" % INDEX_DEFAULT)
|
|
ap.add_argument("--checkin-day", default=None, metavar="DATE",
|
|
help="add a --since DATE floor to the check-in leg, so only "
|
|
"check-ins dated DATE or later are graded (default: the "
|
|
"whole archive, against tools/checkin_baseline.tsv)")
|
|
ap.add_argument("--fast", action="store_true",
|
|
help="skip the selftest-sweep leg; the row is printed as SKIP")
|
|
ap.add_argument("--timeout", type=int, default=300, metavar="S",
|
|
help="per-leg timeout in seconds (default 300)")
|
|
ap.add_argument("--json", metavar="PATH", help="write the rows here")
|
|
ap.add_argument("--quiet", action="store_true")
|
|
ap.add_argument("--selftest", action="store_true",
|
|
help="run the fixture corpus (the run prints its cell count)")
|
|
a = ap.parse_args(argv)
|
|
|
|
if a.selftest:
|
|
return selftest()
|
|
|
|
repo, why = resolve_repo(a.repo)
|
|
if why:
|
|
print(why, file=sys.stderr)
|
|
return 2
|
|
|
|
rows = []
|
|
for leg in LEGS:
|
|
if a.fast and leg["slow"]:
|
|
rows.append({"key": leg["key"], "tool": leg["tool"], "verdict": SKIP,
|
|
"rc": None, "detail": "--fast: leg not run",
|
|
"what": leg["what"], "ms": 0})
|
|
continue
|
|
rows.append(run_leg(leg, repo, a.queue, a.index, a.timeout,
|
|
a.checkin_day))
|
|
|
|
rc = report(rows, repo, a.quiet)
|
|
if a.json:
|
|
c = counts(rows)
|
|
try:
|
|
with open(a.json, "w") as fh:
|
|
json.dump({"repo": repo, "queue": a.queue, "index": a.index,
|
|
"fast": a.fast, "rows": rows, "counts": c, "rc": rc},
|
|
fh, indent=1)
|
|
except OSError as exc:
|
|
print("cannot write --json " + a.json + ": " + str(exc), file=sys.stderr)
|
|
return 2
|
|
return rc
|
|
|
|
|
|
# ------------------------------------------------------------------- selftest
|
|
|
|
def _stub(path, rc, out="", sleep=0.0, marker=None, argv_marker=None):
|
|
"""A stand-in leg: it prints what the fixture needs and exits with its rc."""
|
|
body = ["#!/usr/bin/env python3", "import sys"]
|
|
if sleep:
|
|
body.append("import time; time.sleep(%r)" % sleep)
|
|
if marker:
|
|
body.append("open(%r, 'a').write('hit\\n')" % marker)
|
|
if argv_marker:
|
|
body.append("open(%r, 'w').write(' '.join(sys.argv[1:]))" % argv_marker)
|
|
if out:
|
|
body.append("sys.stdout.write(%r)" % out)
|
|
body.append("sys.exit(%d)" % rc)
|
|
with open(path, "w") as fh:
|
|
fh.write("\n".join(body) + "\n")
|
|
os.chmod(path, 0o755)
|
|
|
|
|
|
def _git_init(repo, commit=False):
|
|
"""Make `repo` a REAL work tree; with commit=True, one commit at HEAD.
|
|
|
|
The fixture used to create an empty `.git/` directory, which is not a
|
|
repository -- it only looked like one to the `isdir(.git)` test this gate
|
|
replaced. The fixture is now as real as what it grades, so cell 25 can grade
|
|
the empty directory as the refusal it always was.
|
|
"""
|
|
subprocess.run(["git", "init", "-q", repo], check=True,
|
|
capture_output=True, text=True)
|
|
if commit:
|
|
subprocess.run(["git", "-C", repo, "add", "-A"], check=True,
|
|
capture_output=True, text=True)
|
|
subprocess.run(["git", "-C", repo,
|
|
"-c", "user.name=verify-selftest",
|
|
"-c", "user.email=selftest@invalid",
|
|
"-c", "commit.gpgsign=false",
|
|
"commit", "-q", "-m", "fixture"], check=True,
|
|
capture_output=True, text=True)
|
|
|
|
|
|
def _fixture(base, rcs, outs=None, sleep=0.0, markers=None, argv_markers=None,
|
|
checkins=("2026-01-01-fixture.md",), commit=False):
|
|
"""A throwaway repo with stubbed legs. rcs: {tool: rc}.
|
|
|
|
`checkins` seeds `checkins/`: one dated file by default (so the check-in leg
|
|
has a day to grade), a tuple of names for cells that need a specific set,
|
|
and None for a repo with no `checkins/` directory at all. `commit` makes one
|
|
commit at HEAD, which a cell needs before it can `git worktree add`.
|
|
"""
|
|
repo = os.path.join(base, "repo")
|
|
os.makedirs(os.path.join(repo, "tools"))
|
|
with open(os.path.join(repo, QUEUE_DEFAULT), "w") as fh:
|
|
fh.write("## Q1 -- fixture\n")
|
|
os.makedirs(os.path.join(repo, "docs"))
|
|
with open(os.path.join(repo, INDEX_DEFAULT), "w") as fh:
|
|
fh.write("fixture\n")
|
|
if checkins is not None:
|
|
os.makedirs(os.path.join(repo, CHECKINS_DEFAULT))
|
|
for name in checkins:
|
|
with open(os.path.join(repo, CHECKINS_DEFAULT, name), "w") as fh:
|
|
fh.write("Did: fixture\nBlocked: nothing\nNext: nothing\n"
|
|
"Ask: nothing\n")
|
|
for tool, rc in rcs.items():
|
|
if rc is None: # tool absent on purpose
|
|
continue
|
|
_stub(os.path.join(repo, "tools", tool), rc,
|
|
out=(outs or {}).get(tool, ""), sleep=sleep,
|
|
marker=(markers or {}).get(tool),
|
|
argv_marker=(argv_markers or {}).get(tool))
|
|
_git_init(repo, commit=commit)
|
|
return repo
|
|
|
|
|
|
def _verify(*argv):
|
|
"""Run this file as the CLI, so the fixture grades the real entry point."""
|
|
proc = subprocess.run([sys.executable, os.path.abspath(__file__)] + list(argv),
|
|
capture_output=True, text=True)
|
|
return proc.returncode, (proc.stdout or "") + (proc.stderr or "")
|
|
|
|
|
|
GREEN_ALL = {"checkin_check.py": 0, "reply_check.py": 0, "queue_index.py": 0,
|
|
"queue_append_audit.py": 0, "doc_command_check.py": 0,
|
|
"selftest_sweep.py": 0, "write_probe.py": 0}
|
|
|
|
|
|
def selftest():
|
|
cells = 0
|
|
passed = 0
|
|
failures = []
|
|
|
|
def check(label, got, want):
|
|
nonlocal cells, passed
|
|
cells += 1
|
|
if got == want:
|
|
passed += 1
|
|
else:
|
|
failures.append("%s\n got: %r\n want: %r" % (label, got, want))
|
|
|
|
with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td:
|
|
# 1. seven green stubs -> rc 0
|
|
repo = _fixture(os.path.join(td, "c1"), GREEN_ALL)
|
|
rc, out = _verify("--repo", repo)
|
|
check("1 all seven legs green -> rc 0", rc, 0)
|
|
check("1b summary counts the seven legs", "verify: 7 legs, 7 pass, 0 fault, "
|
|
"0 ungradable, 0 skipped" in out, True)
|
|
|
|
# 2. a stale index is a FAULT, and its own line reaches the row
|
|
repo = _fixture(os.path.join(td, "c2"),
|
|
dict(GREEN_ALL, **{"queue_index.py": 1}),
|
|
outs={"queue_index.py":
|
|
"STALE: rendered document differs at line 7\n"})
|
|
rc, out = _verify("--repo", repo)
|
|
check("2 stale index -> rc 1", rc, 1)
|
|
check("2b the STALE line is carried into the row",
|
|
"index-current" in out and "differs at line 7" in out, True)
|
|
|
|
# 3. a refusal is UNGRADABLE, not a fault
|
|
repo = _fixture(os.path.join(td, "c3"),
|
|
dict(GREEN_ALL, **{"queue_index.py": 2}),
|
|
outs={"queue_index.py": "cannot read queue: nope\n"})
|
|
rc, out = _verify("--repo", repo)
|
|
check("3 refusal -> rc 2", rc, 2)
|
|
check("3b refusal is named in the summary", "ungradable: index-current" in out, True)
|
|
|
|
# 4. precedence: ungradable outranks fault, and the fault is NOT erased
|
|
repo = _fixture(os.path.join(td, "c4"),
|
|
dict(GREEN_ALL, **{"queue_index.py": 1,
|
|
"selftest_sweep.py": 2}),
|
|
outs={"queue_index.py": "STALE: line 3\n",
|
|
"selftest_sweep.py": "cannot read: no tools\n"})
|
|
rc, out = _verify("--repo", repo)
|
|
check("4 ungradable outranks fault -> rc 2", rc, 2)
|
|
check("4b both classes are still printed in one run",
|
|
"1 fault, 1 ungradable" in out and "fault: index-current" in out,
|
|
True)
|
|
|
|
# 5. a missing leg tool is ungradable, and says which path was missing
|
|
repo = _fixture(os.path.join(td, "c5"),
|
|
dict(GREEN_ALL, **{"queue_append_audit.py": None}))
|
|
rc, out = _verify("--repo", repo)
|
|
check("5 missing tool -> rc 2", rc, 2)
|
|
check("5b the missing path is named", "no such tool:" in out, True)
|
|
|
|
# 6. --fast skips the sweep and does not run it
|
|
marker = os.path.join(td, "c6-sweep-ran")
|
|
repo = _fixture(os.path.join(td, "c6"), GREEN_ALL, markers={"selftest_sweep.py": marker})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
check("6 --fast on a green tree -> rc 0", rc, 0)
|
|
check("6b the skipped leg is printed as SKIP", "selftest-sweep" in out
|
|
and "SKIP" in out, True)
|
|
check("6c the skipped stub never ran", os.path.exists(marker), False)
|
|
|
|
# 7. --json is a real machine-readable record
|
|
repo = _fixture(os.path.join(td, "c7"), GREEN_ALL)
|
|
jpath = os.path.join(td, "c7.json")
|
|
rc, out = _verify("--repo", repo, "--json", jpath)
|
|
obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
|
|
check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])],
|
|
["checkin", "reply-headings", "index-current", "append-audit",
|
|
"doc-commands", "selftest-sweep", "write-axis"])
|
|
check("7b --json carries the same rc the process returned",
|
|
obj.get("rc"), rc)
|
|
|
|
# 8. a repo that is not there is refused before any leg runs
|
|
rc, out = _verify("--repo", os.path.join(td, "no-such-repo"))
|
|
check("8 missing repo -> rc 2", rc, 2)
|
|
check("8b the refusal names the path", "cannot read repo:" in out, True)
|
|
|
|
# 9. rc 3 is retired: it is ungradable, never a fault and never a pass
|
|
repo = _fixture(os.path.join(td, "c9"),
|
|
dict(GREEN_ALL, **{"queue_index.py": 3}),
|
|
outs={"queue_index.py": "unknown state\n"})
|
|
rc, out = _verify("--repo", repo)
|
|
check("9 rc 3 -> rc 2 (retired code)", rc, 2)
|
|
check("9b the unexpected code is named", "unexpected rc 3" in out, True)
|
|
|
|
# 10. a hung leg is a timeout, not a pass and not a fault
|
|
repo = _fixture(os.path.join(td, "c10"),
|
|
dict(GREEN_ALL, **{"selftest_sweep.py": 0}), sleep=5.0)
|
|
rc, out = _verify("--repo", repo, "--timeout", "1")
|
|
check("10 timeout -> rc 2", rc, 2)
|
|
check("10b the timeout is named", "TIMEOUT after 1s" in out, True)
|
|
|
|
# 11. order is fixed, cheapest first: the table is not re-sorted by outcome
|
|
repo = _fixture(os.path.join(td, "c11"),
|
|
dict(GREEN_ALL, **{"queue_index.py": 1}),
|
|
outs={"queue_index.py": "STALE: line 1\n"})
|
|
rc, out = _verify("--repo", repo)
|
|
order = [out.find(k) for k in ("checkin", "reply-headings", "index-current",
|
|
"append-audit", "doc-commands",
|
|
"selftest-sweep", "write-axis")]
|
|
check("11 legs print in fixed order", order == sorted(order)
|
|
and -1 not in order, True)
|
|
|
|
# 12. a skip is counted as a skip, never folded into pass
|
|
repo = _fixture(os.path.join(td, "c12"), GREEN_ALL)
|
|
jpath = os.path.join(td, "c12.json")
|
|
rc, out = _verify("--repo", repo, "--fast", "--json", jpath)
|
|
obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
|
|
check("12 skipped leg is its own count in --json",
|
|
obj.get("counts", {}).get("SKIP"), 1)
|
|
check("12b and the skipped leg is not counted as a pass",
|
|
obj.get("counts", {}).get(PASS), 6)
|
|
|
|
# 13. the detail line must not contradict the verdict: a row graded
|
|
# FAULT beside "0 fatal" is the SILENT failure this gate exists to catch.
|
|
check("13 a zero-fault summary is not a reason line",
|
|
pick_detail("sweep: 15 tools, 15 OK, 0 fatal\n"
|
|
"read-refusal: 20 probes, 19 ok, 1 fault (NO_PROBE)\n"
|
|
"malformed-input: 18 probes, 17 ok, 1 fault (NO_PROBE)\n"),
|
|
"read-refusal: 20 probes, 19 ok, 1 fault (NO_PROBE)")
|
|
check("13b a clean run still gets its sweep line",
|
|
pick_detail("sweep: 14 tools, 14 OK, 0 fatal\n"
|
|
"read-refusal: 19 probes, 19 ok, 0 fault\n"),
|
|
"sweep: 14 tools, 14 OK, 0 fatal")
|
|
|
|
# 14. THE HOLE THIS LEG CLOSES: a contract-breaking check-in is rc 1,
|
|
# and the gate is no longer green on a tree that carries one.
|
|
repo = _fixture(os.path.join(td, "c14"),
|
|
dict(GREEN_ALL, **{"checkin_check.py": 1}),
|
|
outs={"checkin_check.py":
|
|
"FAIL: 2 file(s) violate the check-in contract\n"})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
check("14 a broken check-in -> rc 1 (was rc 0 with three legs)", rc, 1)
|
|
check("14b the fault row is named checkin and carries its reason",
|
|
"checkin" in out and "violate the check-in contract" in out
|
|
and "fault: checkin" in out, True)
|
|
|
|
# 15. no checkins/ directory is ungradable, not a pass and not a fault
|
|
repo = _fixture(os.path.join(td, "c15"), GREEN_ALL, checkins=None)
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
check("15 no checkins dir -> rc 2", rc, 2)
|
|
check("15b the ungradable row says which directory is missing",
|
|
"ungradable: checkin" in out and "no check-in directory:" in out, True)
|
|
|
|
# 16. the leg grades the WHOLE archive against its own ledger -- the
|
|
# fix for a newest-day floor that made a backdated fault invisible.
|
|
av = os.path.join(td, "c16-argv")
|
|
repo = _fixture(os.path.join(td, "c16"), GREEN_ALL,
|
|
argv_markers={"checkin_check.py": av},
|
|
checkins=("2026-01-01-fixture.md", "2026-03-04-fixture.md"))
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
seen = open(av).read() if os.path.exists(av) else ""
|
|
check("16 the check-in leg is run with --fail", "--fail" in seen, True)
|
|
check("16b and with the ledger resolved from --repo, so no day is excluded",
|
|
"--baseline" in seen
|
|
and os.path.join(repo, "tools", "checkin_baseline.tsv") in seen
|
|
and "--since" not in seen, True)
|
|
|
|
# 17. --checkin-day still narrows the window explicitly, in the open
|
|
av = os.path.join(td, "c17-argv")
|
|
repo = _fixture(os.path.join(td, "c17"), GREEN_ALL,
|
|
argv_markers={"checkin_check.py": av},
|
|
checkins=("2026-01-01-fixture.md", "2026-03-04-fixture.md"))
|
|
rc, out = _verify("--repo", repo, "--fast", "--checkin-day", "2026-01-01")
|
|
seen = open(av).read() if os.path.exists(av) else ""
|
|
check("17 --checkin-day adds a --since floor",
|
|
"--since 2026-01-01" in seen, True)
|
|
|
|
# 19. a ledger that cannot be read is UNGRADABLE (rc 2), never a pass:
|
|
# the tool's own reason has to reach the row, because "no ledger" and
|
|
# "clean archive" print nothing alike and must not grade alike.
|
|
repo = _fixture(os.path.join(td, "c19"),
|
|
dict(GREEN_ALL, **{"checkin_check.py": 2}),
|
|
outs={"checkin_check.py":
|
|
"cannot grade: baseline %s/tools/checkin_baseline.tsv: "
|
|
"[Errno 2] No such file or directory\n"
|
|
% os.path.join(td, "c19", "repo")})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
check("19 a missing ledger is rc 2 and names the ledger", rc, 2)
|
|
check("19b the ungradable row carries the tool's own reason",
|
|
"ungradable: checkin" in out and "checkin_baseline.tsv" in out, True)
|
|
|
|
# 18. the detail line must survive a real tool's noise: `checkin_check`
|
|
# prints a README SKIP note carrying "not a " on every run, and a WEAK
|
|
# hint must not beat the FAIL line that actually names the verdict.
|
|
noisy = ("== 2 check-in files in /tmp/x/checkins ==\n"
|
|
"roster: 9 handles; today 2026-10-19\n"
|
|
"since 2026-10-19: 2 graded, 72 older file(s) not graded\n"
|
|
" SKIP README.md (the directory's own README: not a check-in)\n"
|
|
"\n"
|
|
"2026-10-19-volt-02.md BAD_NAME,DUPLICATE 2 files for volt\n"
|
|
"checkin_check: 2 files, 2 with a fault (BAD_NAME 1)\n"
|
|
"FAIL: 2 file(s) violate the check-in contract\n")
|
|
check("18 a README skip note does not mask the FAIL line",
|
|
pick_detail(noisy), "FAIL: 2 file(s) violate the check-in contract")
|
|
|
|
# 18b. and on a clean run the same noise must not become the detail:
|
|
# the row reads PASS beside the summary, not beside a README note.
|
|
clean = noisy.replace("checkin_check: 2 files, 2 with a fault (BAD_NAME 1)\n"
|
|
"FAIL: 2 file(s) violate the check-in contract\n",
|
|
"checkin_check: 2 files, 0 with a fault, 1 advisory\n")
|
|
check("18b a clean check-in row carries its summary, not the skip note",
|
|
pick_detail(clean), "checkin_check: 2 files, 0 with a fault, 1 advisory")
|
|
|
|
# 20. THE FIFTH LEG: a reply whose heading names the wrong party is a
|
|
# FAULT on the gate and the row is named -- the class that was invisible
|
|
# for as long as `replies/` had no reader.
|
|
repo = _fixture(os.path.join(td, "c20"),
|
|
dict(GREEN_ALL, **{"reply_check.py": 1}),
|
|
outs={"reply_check.py":
|
|
"reply_check: 192 file(s), 1 new fault(s), "
|
|
"0 known-faulting in the baseline\n"})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
check("20 a new mis-addressed reply -> rc 1", rc, 1)
|
|
check("20b the fault row is named reply-headings and carries its reason",
|
|
"fault: reply-headings" in out and "1 new fault(s)" in out, True)
|
|
|
|
# 21. the ledger is read from the tree being graded, not from the host
|
|
av = os.path.join(td, "c21-argv")
|
|
repo = _fixture(os.path.join(td, "c21"), GREEN_ALL,
|
|
argv_markers={"reply_check.py": av})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
seen = open(av).read() if os.path.exists(av) else ""
|
|
check("21 the reply leg is run with the repo's own ledger",
|
|
"--baseline" in seen
|
|
and os.path.join(repo, "tools", "reply_baseline.tsv") in seen, True)
|
|
|
|
# 22. on a PASS row the detail is the tool's own concluding summary, not
|
|
# the first line that happens to contain a hint substring.
|
|
reply_noise = ("reply_check -- 192 file(s) under /tmp/r (roster: x)\n"
|
|
" NO_RECIPIENT_IN_HEADING 7\n"
|
|
" 2026-10-12-to-merlin-19.md | heading: # the precondition refuses\n"
|
|
"reply_check: 192 file(s), 0 new fault(s), 1 known-faulting in the baseline\n")
|
|
check("22 a clean row's detail is the tool's summary",
|
|
pick_detail(reply_noise, summary=True),
|
|
"reply_check: 192 file(s), 0 new fault(s), 1 known-faulting in the baseline")
|
|
check("22b a fault run still prefers the reason",
|
|
pick_detail("cannot grade: baseline x: No such file\n"
|
|
"tool: 1 new fault(s)\n"),
|
|
"cannot grade: baseline x: No such file")
|
|
|
|
# 23. THE SANDBOX THE GATE REFUSED. A linked worktree's `.git` is a FILE
|
|
# (`gitdir: ...`), not a directory, so `os.path.isdir(repo/.git)` -- the
|
|
# test this gate used until 2026-10-19 -- answered "not a git repository"
|
|
# (rc 2) on a checkout every leg would have graded. Measured RED against
|
|
# the old test on this exact fixture, GREEN here.
|
|
repo = _fixture(os.path.join(td, "c23"), GREEN_ALL, commit=True)
|
|
wt = os.path.join(td, "c23-wt")
|
|
subprocess.run(["git", "-C", repo, "worktree", "add", "--detach", wt,
|
|
"HEAD"], check=True, capture_output=True, text=True)
|
|
with open(os.path.join(wt, ".git")) as fh:
|
|
gitfile = fh.read()
|
|
check("23 the fixture worktree really carries a .git file, not a dir",
|
|
gitfile.startswith("gitdir: ") and not os.path.isdir(wt + "/.git"),
|
|
True)
|
|
rc, out = _verify("--repo", wt, "--fast")
|
|
check("23b a linked worktree is graded, not refused -> rc 0", rc, 0)
|
|
check("23c the row names the worktree, not the tree it links to",
|
|
"verify -- repo: %s" % os.path.abspath(wt) in out, True)
|
|
|
|
# 24. --repo stays the ROOT: a subdirectory is refused with the root
|
|
# named, because widening it silently would grade a tree nobody asked for
|
|
rc, out = _verify("--repo", os.path.join(repo, "tools"), "--fast")
|
|
check("24 a subdirectory is refused, not widened -> rc 2", rc, 2)
|
|
check("24b the refusal names the root it would have graded",
|
|
"--repo takes the root" in out and os.path.realpath(repo) in out,
|
|
True)
|
|
|
|
# 25. the OLD test's false positive: an empty `.git` directory is not a
|
|
# repository, and git says so where `isdir` said otherwise.
|
|
fake = os.path.join(td, "c25")
|
|
os.makedirs(os.path.join(fake, ".git"))
|
|
rc, out = _verify("--repo", fake, "--fast")
|
|
check("25 an empty '.git' directory is not a repo -> rc 2", rc, 2)
|
|
check("25b and the refusal says git refused it",
|
|
"is not a git work tree" in out and "cannot read repo:" in out, True)
|
|
|
|
# 26. a bare repository has no work tree to grade
|
|
bare = os.path.join(td, "c26.git")
|
|
subprocess.run(["git", "init", "-q", "--bare", bare], check=True,
|
|
capture_output=True, text=True)
|
|
rc, out = _verify("--repo", bare, "--fast")
|
|
check("26 a bare repository -> rc 2", rc, 2)
|
|
|
|
# 28. THE SEVENTH LEG, and the tree it must read. The tool is called
|
|
# with `--rev WORKTREE --fail`: the commit window is where a docstring is
|
|
# edited, and a leg reading HEAD would grade the last commit while the
|
|
# lane wrote the next one. Measured here as the argv, not as a promise.
|
|
av = os.path.join(td, "c28-argv")
|
|
repo = _fixture(os.path.join(td, "c28"), GREEN_ALL,
|
|
argv_markers={"doc_command_check.py": av})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
seen = open(av).read() if os.path.exists(av) else ""
|
|
check("28 the doc-commands leg runs --rev WORKTREE --fail",
|
|
"--rev WORKTREE" in seen and "--fail" in seen, True)
|
|
check("28b and the leg runs the tool inside --repo (this stub wrote it)",
|
|
os.path.exists(av), True)
|
|
|
|
# 29. a documented command its own tool would reject is a FAULT on the
|
|
# gate, named on the row -- the class that was prose until this leg.
|
|
repo = _fixture(os.path.join(td, "c29"),
|
|
dict(GREEN_ALL, **{"doc_command_check.py": 1}),
|
|
outs={"doc_command_check.py":
|
|
"FLAG_NOT_ACCEPTED tools/x.py -> tools/x.py "
|
|
"missing=['--gone']\n"})
|
|
rc, out = _verify("--repo", repo, "--fast")
|
|
check("29 a rejected docstring command -> rc 1", rc, 1)
|
|
check("29b the fault row is named doc-commands and carries its reason",
|
|
"fault: doc-commands" in out and "FLAG_NOT_ACCEPTED" in out, True)
|
|
|
|
# 27. the count in this file's own docstring is a CLAIM, so it is graded
|
|
# against the count that actually ran. On 2026-10-19 the docstring and
|
|
# the run said 50 while the argparse help string still said 42 and
|
|
# nothing failed -- the same count-word defect this repo keeps finding in
|
|
# documents, closed in the document that prints the number.
|
|
check("27 the docstring's cell count is the count that ran",
|
|
DOC_CELLS, cells + 1)
|
|
|
|
print("selftest: %d cells, %d passed, %d failed" % (cells, passed, len(failures)))
|
|
for f in failures:
|
|
print(" FAIL %s" % f)
|
|
return 0 if passed == cells else 1
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|