#!/usr/bin/env python3 """verify.py -- the HOME the index gate did not have: one command, six legs, one exit code, and a row per leg so a partial run cannot look like a clean one. WHY THIS EXISTS On 2026-10-18 this lane told Alexander, three ticks running, to "wire `queue_index --check` into the tick's check.sh". There is no `check.sh` in this repo and no `scripts/` directory at all -- `git ls-files` shows `tools/` and nothing else. The advice named a target that does not exist, which is why it was never adopted. The gate did not need a decision about policy; it needed a home. This is that home, and it is a task, not a policy call: nothing here obliges any lane to run it, and nothing is enforced until a lane does. The second reason is narrower and this lane's own: three separate counts of "which commits left the index stale" came out wrong in one week, each from a hand-assembled probe loop. A gate whose legs are remembered is a gate that gets run selectively. Fixed order, one rc -- that is the whole design. The third reason is measured, not argued. On 2026-10-19T09:42Z this lane put a contract-breaking check-in into a clone of HEAD -- `checkins/2026-10-19-volt-02.md`, a second same-day file carrying a suffix -- and ran both instruments on it: `checkin_check.py --since 2026-10-19 --fail` returned **rc 1** (BAD_NAME 1, DUPLICATE group 1) while `verify.py --fast` returned **rc 0** on the same tree, "gate: OK -- 2 of 2 graded leg(s) passed". Three legs that grade the queue, its append discipline and the tools' selftests said nothing about the one artifact every lane writes every tick. A fourth leg closes that hole. The fifth is the same shape one directory over, and it is here for a reason that was measured before it was written. On 2026-10-19 this lane built `reply_check.py` and deliberately did NOT wire it: the archive has one live fault -- `replies/2026-10-15-to-timmy-58.md` is named for Timmy and headed "to Merlin" -- it is Merlin's file and Merlin's to repair, and that tick's reasoning was that a gate wired to a defect its owner has not fixed yet is a gate that gets switched off in a week. That reasoning is right about a gate with no ledger and wrong about this repo, which already solved the same problem for check-ins: `tools/reply_baseline.tsv` states the one tolerance, the fault stays visible in every report as known-faulting, and a second mis-addressed reply -- or a new fault class on the known file -- is rc 1 the day it lands. Waiting for another lane to edit its own file was the only alternative, and it had no date on it. WHAT IT RUNS -- one row per leg, cheapest first, never in parallel checkin tools/checkin_check.py --fail --baseline tools/checkin_baseline.tsv the WHOLE check-in archive against F1-F5 (names, duplicates, unknown handles, dates, emptiness), where the ledger states the faults the archive already carried; only a class it does not name is a fault. `--checkin-day DATE` adds a `--since` floor reply-headings tools/reply_check.py --baseline tools/reply_baseline.tsv every reply's heading against its own filename (addressee and number), on the same ledger rule: `replies/` is the second largest artifact directory and, until 2026-10-19, nothing here read it index-current tools/queue_index.py --check docs/queue-index.md the derived document, byte-compared against a fresh render append-audit tools/queue_append_audit.py --guard --to HEAD the newest queue-touching commit, graded in ITS OWN tree selftest-sweep tools/selftest_sweep.py every tool's --selftest plus its two read-refusal axes (skip with --fast; the row is still printed and counted) write-axis tools/write_probe.py every tracked tool's `--json PATH` pointed at a target this host cannot write: rc 2, one line naming the path, never a traceback -- and, since 2026-10-21, every caller-path write flag other than `--json` (`--write`, `--save-vocab`) on the same rule. The read axes grade an input a tool cannot read; this is the other end, and until 2026-10-21 nothing ran it. The legs are ordered by cost, and a leg is never skipped because an earlier one failed: a stale index and a broken selftest are different defects and the run should name both in one pass. WHY THE CHECK-IN LEG NOW GRADES THE ARCHIVE, NOT ONE DAY Grading the whole archive was, until 2026-10-19, a permanently red gate: at `e98a34b` the unbaselined run exits 1 on every one of the archive's pre-rule rows (25 BAD_NAME, 10 DUPLICATE groups, all dated 2026-09-24 through 2026-10-07) -- that list is now a measured artifact, `tools/checkin_baseline.tsv`, one row per file rather than a number in this docstring. The first version of this leg therefore graded the archive's **newest day** only, on the reasoning that it is the only day on which a new fault can appear. That reasoning is false, and it was measured false rather than argued: a check-in added today under a **backdated** filename -- `checkins/2026-10-02-volt-b.md`, which is what a typo in the date half produces -- is not on the newest day, so the floor excluded it. On a clone of `e98a34b` carrying that one file, `verify.py --fast` printed `checkin_check: 1 files, 0 with a fault` and exited **0**, while the same tree's whole-archive run exited **1** naming the file. The fix keeps both properties -- history is not re-litigated, nothing is invisible. `tools/checkin_baseline.tsv` lists the faults the archive already carried, one `\t` row each, and `--baseline` grades every file against it: a listed class is KNOWN, anything else is a fault. A file that is not in the ledger at all is a fault wherever its date falls, and a file that gains a class it did not have is a fault too -- the ledger states what was tolerated, not that a path is exempt. A missing or malformed ledger is rc 2 (cannot grade), never a silent pass, and stale entries (file gone, class cleared) are reported and never make rc 1. `--checkin-day DATE` survives as an explicit narrowing floor for an operator who wants one. THE EXIT CONTRACT -- the house three codes 0 every graded leg passed 1 at least one leg ran and graded a fault 2 at least one leg could not grade -- a missing tool, a timeout, an unexpected exit code, an unreadable repo -- or no leg was runnable at all rc 2 OUTRANKS rc 1 deliberately. rc 1 means "I ran and I graded a fault"; a run with an ungradable leg has not made that claim about the whole gate, and folding the two (as the 15:16Z correction caught `queue_append_audit` doing with ungradable commits) is how a "3 stale" reading hides a "2 could not be read". The summary always prints both counts, so precedence never erases a fault -- it only refuses to certify the run that found it. A skipped leg is not a pass and not a fault: it is counted as `skipped` and printed as SKIP. `--fast` on a green tree exits 0 with one row SKIP, and the summary says so rather than pretending the sweep ran. TOOLS ARE RESOLVED FROM --repo, NOT FROM THIS FILE'S DIRECTORY `verify.py --repo /path/to/worktree` runs THAT tree's `queue_index.py`, `queue_append_audit.py` and `selftest_sweep.py`. That is the instrument lesson the 15:16Z correction paid for: a gate run in a working tree at HEAD says nothing about a cited revision, so grading a revision means grading its own tools from its own checkout. --queue and --index stay REPO-RELATIVE (defaults WORK-QUEUE.md and docs/queue-index.md) because `queue_append_audit` reads them with `git show :`, which has no absolute-path form. --repo IS A WORK TREE, ASKED OF GIT Until 2026-10-19 the repo test was `os.path.isdir(/.git)`, and it was wrong in both directions. A LINKED worktree's `.git` is a 47-byte file holding `gitdir: ...`, so from `git worktree add --detach ` the gate returned **rc 2 "not a git repository"** against a checkout whose every leg passes -- measured that day on `cf5101f`: the old test rc 2, the test that follows rc 0 with every leg then wired green (`selftest-sweep` included). That made the gate unrunnable in the one sandbox a lane gets for free and forced the choice between grading unsandboxed in the shared clone and not running it at all. An empty directory named `.git` was the same test's false positive: it passed while git itself refused the path. `resolve_repo` now asks `git rev-parse --show-toplevel`, which succeeds in every work tree (main, linked, submodule) and fails in a bare repository or a bare `.git` directory. The path must still be the ROOT: a subdirectory is rc 2 naming the root, because widening `--repo` silently would grade a tree nobody named. `--selftest` fixture corpus, 51 cells """ import argparse import json import os import re import subprocess import sys import tempfile import time QUEUE_DEFAULT = "WORK-QUEUE.md" INDEX_DEFAULT = "docs/queue-index.md" CHECKINS_DEFAULT = "checkins" PASS, FAULT, UNGRADABLE, SKIP = "PASS", "FAULT", "UNGRADABLE", "SKIP" # Legs, in the fixed order they are printed and run. `tool` is resolved inside # the repo being graded. `slow` marks the leg --fast drops. LEGS = ( {"key": "checkin", "tool": "checkin_check.py", "slow": False, "what": "grade the newest day's check-ins against the naming and " "duplicate rules"}, {"key": "reply-headings", "tool": "reply_check.py", "slow": False, "what": "grade every reply's heading against its own filename"}, {"key": "index-current", "tool": "queue_index.py", "slow": False, "what": "render the index and byte-compare it with the document"}, {"key": "append-audit", "tool": "queue_append_audit.py", "slow": False, "what": "grade the newest queue-touching commit in its own tree"}, {"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True, "what": "every tool's --selftest and both read-refusal axes"}, {"key": "write-axis", "tool": "write_probe.py", "slow": False, "what": "a --json target this host cannot write: rc 2, one line naming " "it, never a traceback -- and the same rule for every caller-path " "write flag (--write, --save-vocab)"}, ) # A detail line is a courtesy, not the verdict -- but a verdict with no line # beside it is what the read-refusal axis calls SILENT, so it is graded too. # # STRONG outranks WEAK, and both outrank a per-file SKIP note. That tiering is # not cosmetic: `checkin_check.py` prints " SKIP README.md (the directory's own # README: not a check-in)" as its second line on every real run, "not a " is a # WEAK hint, and before the tiering that note was carried onto a FAULT row -- # the row said FAULT beside a line about the README while the tool's own # "FAIL: 2 file(s) violate the check-in contract" sat further down, unpicked. STRONG_HINTS = ("STALE:", "FAIL:", "FAILED", "fatal", "TIMEOUT", "NO_", "cannot", "error", "Error", "refus", # write_probe's fault block (2026-10-21): each line begins # "FAULT ...", and that line sits above the probe table # precisely so this function reaches it before the first healthy # row. A SILENT fault's own detail carries no other marker, so # without this token the block would be skipped for the row that # names a tool with nothing wrong with it. "FAULT") WEAK_HINTS = ("FAIL", "not a ") # A tool's concluding line: `reply_check: 192 file(s), 0 new fault(s) ...`. # Hyphenated labels (`read-refusal:`, `malformed-input:`) are deliberately not # summaries -- they are axis lines inside a tool's report. SUMMARY_RE = re.compile(r"^[a-z][a-z0-9_]*: ") # The selftest size this file's docstring claims. Cell 27 grades that claim # against the count that actually ran, because on 2026-10-19 the count left # this exact defect behind: the docstring and the run said 50 while the # argparse help string still said 42, and nothing failed. _DOC_CLAIM = re.search(r"fixture corpus, (\d+) cells", __doc__ or "") DOC_CELLS = int(_DOC_CLAIM.group(1)) if _DOC_CLAIM else 0 def pick_detail(text, limit=150, summary=False): """First line that names a reason, else the last line that says anything. A summary line that reports ZERO faults is not a reason: `selftest_sweep` prints `sweep: 15 tools, 15 OK, 0 fatal` even on the run where its input axes went red, and that line was being carried onto a FAULT row -- the row said FAULT beside the words "0 fatal". Zero-count summaries are therefore only reachable on the second pass (a clean run still gets its sweep line), and any line that names a real reason outranks them. `summary=True` is the clean-run answer, and it is a different question. On a PASS row the useful detail is the tool's own concluding line, not the first line in its report that happens to contain "refus" -- `reply_check` prints one heading excerpt per advisory row, and one of those headings contains the word "refuses", which was reaching the row as if it were the reason. On a fault row the reason still wins, because that is what a reader needs there. """ lines = [ln.strip() for ln in text.splitlines() if ln.strip()] kept = [ln for ln in lines if not ln.startswith("SKIP ")] or lines if summary: sums = [ln for ln in kept if SUMMARY_RE.match(ln)] if sums: return sums[-1][:limit] hinted = [ln for ln in kept if any(h in ln for h in STRONG_HINTS + WEAK_HINTS)] # A class row that printed zero is not a reason and not a count of anything: # `reply_check` prints its whole class table on every run, so `NO_HEADER 0` # was reaching the detail line of a PASS row ahead of the summary. Same # failure the " 0 fatal" filter catches, one shape further out. class_row = re.compile(r"^[A-Z][A-Z0-9_]+ +(\d+)$") zero_row = re.compile(r"^[A-Z][A-Z0-9_]+ +0$") reasons = [ln for ln in hinted if " 0 fatal" not in ln and " 0 fault" not in ln and not zero_row.match(ln)] # Nor is a nonzero class row a "reason": `NO_RECIPIENT_IN_HEADING 7` is a # count, and `NO_` in its name is not a diagnostic marker. A hint counts # only on a line that is not a class table row. strong = [ln for ln in reasons if any(h in ln for h in STRONG_HINTS) and not class_row.match(ln)] # A tool's own verdict line outranks every other reason in its report: when # `reply_check` names a mis-addressed file it also prints one heading excerpt # per row, and an excerpt containing "refuses" was beating the FAIL line. verdict = [ln for ln in strong if ln.startswith(("FAIL:", "STALE:"))] if verdict: return verdict[0][:limit] if strong: return strong[0][:limit] if reasons: return reasons[0][:limit] if hinted: return hinted[0][:limit] return kept[-1][:limit] if kept else "" def resolve_repo(path): """`--repo` is a git WORK TREE ROOT. Return (repo, why-not), asking git. Until 2026-10-19 this test was `os.path.isdir(/.git)`, which is the wrong question in both directions, and both were measured on that day: * a LINKED worktree's `.git` is a 47-byte FILE holding `gitdir: ...`, so `git worktree add --detach` produced a checkout every leg would have graded and the gate refused with rc 2 "not a git repository" -- the one sandbox a lane gets for free was the one place the operator gate could not run, and the alternatives were grading unsandboxed in the shared clone or not running the gate at all; * an empty directory named `.git` passed the same test on a path git itself refuses, so the refusal was also not worth the name. `git rev-parse --show-toplevel` answers the real question: it succeeds in every work tree (main, linked, submodule) and fails in a bare repository or a directory that merely holds a `.git` entry. The path must still be the ROOT -- a subdirectory is rc 2 with the root named, because widening `--repo` silently would grade a different tree than the caller named. """ repo = os.path.abspath(path) if not os.path.isdir(repo): return None, "cannot read repo: %s is not a directory" % repo try: proc = subprocess.run(["git", "-C", repo, "rev-parse", "--show-toplevel"], capture_output=True, text=True) except OSError as exc: return None, ("cannot read repo: %s (git is not runnable: %s)" % (repo, exc)) top = (proc.stdout or "").strip() if proc.returncode != 0 or not top: why = [l for l in (proc.stderr or "").strip().splitlines() if l.strip()] return None, ("cannot read repo: %s is not a git work tree (%s)" % (repo, why[-1] if why else "git rev-parse rc %d" % proc.returncode)) if os.path.realpath(top) != os.path.realpath(repo): return None, ("cannot read repo: %s is inside the work tree rooted at " "%s; --repo takes the root" % (repo, top)) return repo, None def newest_checkin_day(repo): """The newest `` prefix in checkins/ -> (day, why-not). Not the graded window any more -- the leg grades the whole archive against its own ledger -- but still the cheapest way to answer "is there anything here to grade at all", which is what makes a repo with no `checkins/` directory (or no dated file in it) ungradable with a message naming which of the two it is, rather than a spawn and an rc 2 from further down. """ d = os.path.join(repo, CHECKINS_DEFAULT) if not os.path.isdir(d): return None, "no check-in directory: %s" % d days = sorted(m.group(1) for m in (re.match(r"^(\d{4}-\d{2}-\d{2})-", n) for n in os.listdir(d)) if m) if not days: return None, "no dated check-in files in %s" % d return days[-1], None def leg_argv(key, repo, queue, index, checkin_day=None): if key == "checkin": # The ledger is a file in the tree being graded, so a revision is graded # with the tolerance it committed -- not with this host's idea of one. argv = [sys.executable, os.path.join(repo, "tools", "checkin_check.py"), "--repo", repo, "--fail", "--baseline", os.path.join(repo, "tools", "checkin_baseline.tsv")] if checkin_day: argv += ["--since", checkin_day] return argv if key == "reply-headings": # The ledger comes from the tree being graded, for the same reason the # check-in ledger does: a revision is graded with the tolerance it # committed, not with this host's idea of one. return [sys.executable, os.path.join(repo, "tools", "reply_check.py"), "--repo", repo, "--baseline", os.path.join(repo, "tools", "reply_baseline.tsv")] if key == "index-current": return [sys.executable, os.path.join(repo, "tools", "queue_index.py"), "--repo", repo, "--queue", queue, "--check", index] if key == "append-audit": return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"), "--repo", repo, "--queue", queue, "--index", index, "--guard", "--to", "HEAD"] if key == "selftest-sweep": return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"), "--repo", repo, "--quiet"] if key == "write-axis": # The probe writes only under its own TemporaryDirectory; the repo is # the tree it discovers tools from, so a revision is graded against the # guards it committed rather than this host's copy. return [sys.executable, os.path.join(repo, "tools", "write_probe.py"), "--repo", repo] raise KeyError(key) def run_leg(leg, repo, queue, index, timeout, checkin_day=None): """One leg -> one row. rc 0/1/2 are graded; anything else is ungradable.""" tool = os.path.join(repo, "tools", leg["tool"]) started = time.time() row = {"key": leg["key"], "tool": leg["tool"], "verdict": None, "rc": None, "detail": "", "what": leg["what"]} if not os.path.exists(tool): row.update(verdict=UNGRADABLE, detail="no such tool: %s" % tool) row["ms"] = 0 return row day = checkin_day if leg["key"] == "checkin": # Presence is still pre-checked here: the tool's own rc-2 message names # the directory, but this row can say so without spawning a process. newest, why = newest_checkin_day(repo) if newest is None: row.update(verdict=UNGRADABLE, detail=why) row["ms"] = 0 return row try: proc = subprocess.run(leg_argv(leg["key"], repo, queue, index, day), cwd=repo, capture_output=True, text=True, timeout=timeout) except subprocess.TimeoutExpired: row.update(verdict=UNGRADABLE, detail="TIMEOUT after %ss" % timeout) row["ms"] = int((time.time() - started) * 1000) return row except OSError as exc: row.update(verdict=UNGRADABLE, detail="could not run: %s" % exc) row["ms"] = int((time.time() - started) * 1000) return row out = (proc.stdout or "") + (proc.stderr or "") row["rc"] = proc.returncode row["detail"] = pick_detail(out, summary=(proc.returncode == 0)) if proc.returncode == 0: row["verdict"] = PASS elif proc.returncode == 1: row["verdict"] = FAULT elif proc.returncode == 2: row["verdict"] = UNGRADABLE else: # rc 3 is retired in this repo (one meaning, two codes); any other code # is a tool that left the contract, and the run cannot grade through it. row["verdict"] = UNGRADABLE row["detail"] = "unexpected rc %d: %s" % (proc.returncode, row["detail"]) row["ms"] = int((time.time() - started) * 1000) return row def counts(rows): c = {PASS: 0, FAULT: 0, UNGRADABLE: 0, SKIP: 0} for r in rows: c[r["verdict"]] += 1 return c def decide(rows): """The precedence, in one place: ungradable outranks fault outranks clean.""" c = counts(rows) if c[UNGRADABLE]: return 2 if c[FAULT]: return 1 return 0 def report(rows, repo, quiet=False): c = counts(rows) if not quiet: print("verify -- repo: %s" % repo) for r in rows: rc = "-" if r["rc"] is None else str(r["rc"]) print(" %-16s %-10s rc=%-2s %-6s %s" % (r["key"], r["verdict"], rc, "%dms" % r["ms"], r["detail"])) print() print("verify: %d legs, %d pass, %d fault, %d ungradable, %d skipped" % (len(rows), c[PASS], c[FAULT], c[UNGRADABLE], c[SKIP])) faulted = [r["key"] for r in rows if r["verdict"] == FAULT] ungraded = [r["key"] for r in rows if r["verdict"] == UNGRADABLE] if faulted: print("fault: %s" % " ".join(faulted)) if ungraded: print("ungradable: %s" % " ".join(ungraded)) rc = decide(rows) if rc == 0: print("gate: OK -- %d of %d graded leg(s) passed%s" % (c[PASS], c[PASS] + c[FAULT], "" if not c[SKIP] else " (%d skipped: not a pass, not a fault)" % c[SKIP])) elif rc == 1: print("gate: FAULT -- %d leg(s) ran and graded a fault: %s" % (c[FAULT], " ".join(faulted))) else: print("gate: NOT CERTIFIED -- %d leg(s) could not grade: %s" % (c[UNGRADABLE], " ".join(ungraded))) return rc def main(argv=None): ap = argparse.ArgumentParser( description="Run the repo's gates as one command: the newest day's " "check-ins, index currency, the append audit, and the " "selftest sweep.") ap.add_argument("--repo", default=".") ap.add_argument("--queue", default=QUEUE_DEFAULT, help="repo-relative queue path (default %s)" % QUEUE_DEFAULT) ap.add_argument("--index", default=INDEX_DEFAULT, help="repo-relative index document (default %s)" % INDEX_DEFAULT) ap.add_argument("--checkin-day", default=None, metavar="DATE", help="add a --since DATE floor to the check-in leg, so only " "check-ins dated DATE or later are graded (default: the " "whole archive, against tools/checkin_baseline.tsv)") ap.add_argument("--fast", action="store_true", help="skip the selftest-sweep leg; the row is printed as SKIP") ap.add_argument("--timeout", type=int, default=300, metavar="S", help="per-leg timeout in seconds (default 300)") ap.add_argument("--json", metavar="PATH", help="write the rows here") ap.add_argument("--quiet", action="store_true") ap.add_argument("--selftest", action="store_true", help="run the fixture corpus (the run prints its cell count)") a = ap.parse_args(argv) if a.selftest: return selftest() repo, why = resolve_repo(a.repo) if why: print(why, file=sys.stderr) return 2 rows = [] for leg in LEGS: if a.fast and leg["slow"]: rows.append({"key": leg["key"], "tool": leg["tool"], "verdict": SKIP, "rc": None, "detail": "--fast: leg not run", "what": leg["what"], "ms": 0}) continue rows.append(run_leg(leg, repo, a.queue, a.index, a.timeout, a.checkin_day)) rc = report(rows, repo, a.quiet) if a.json: c = counts(rows) try: with open(a.json, "w") as fh: json.dump({"repo": repo, "queue": a.queue, "index": a.index, "fast": a.fast, "rows": rows, "counts": c, "rc": rc}, fh, indent=1) except OSError as exc: print("cannot write --json " + a.json + ": " + str(exc), file=sys.stderr) return 2 return rc # ------------------------------------------------------------------- selftest def _stub(path, rc, out="", sleep=0.0, marker=None, argv_marker=None): """A stand-in leg: it prints what the fixture needs and exits with its rc.""" body = ["#!/usr/bin/env python3", "import sys"] if sleep: body.append("import time; time.sleep(%r)" % sleep) if marker: body.append("open(%r, 'a').write('hit\\n')" % marker) if argv_marker: body.append("open(%r, 'w').write(' '.join(sys.argv[1:]))" % argv_marker) if out: body.append("sys.stdout.write(%r)" % out) body.append("sys.exit(%d)" % rc) with open(path, "w") as fh: fh.write("\n".join(body) + "\n") os.chmod(path, 0o755) def _git_init(repo, commit=False): """Make `repo` a REAL work tree; with commit=True, one commit at HEAD. The fixture used to create an empty `.git/` directory, which is not a repository -- it only looked like one to the `isdir(.git)` test this gate replaced. The fixture is now as real as what it grades, so cell 25 can grade the empty directory as the refusal it always was. """ subprocess.run(["git", "init", "-q", repo], check=True, capture_output=True, text=True) if commit: subprocess.run(["git", "-C", repo, "add", "-A"], check=True, capture_output=True, text=True) subprocess.run(["git", "-C", repo, "-c", "user.name=verify-selftest", "-c", "user.email=selftest@invalid", "-c", "commit.gpgsign=false", "commit", "-q", "-m", "fixture"], check=True, capture_output=True, text=True) def _fixture(base, rcs, outs=None, sleep=0.0, markers=None, argv_markers=None, checkins=("2026-01-01-fixture.md",), commit=False): """A throwaway repo with stubbed legs. rcs: {tool: rc}. `checkins` seeds `checkins/`: one dated file by default (so the check-in leg has a day to grade), a tuple of names for cells that need a specific set, and None for a repo with no `checkins/` directory at all. `commit` makes one commit at HEAD, which a cell needs before it can `git worktree add`. """ repo = os.path.join(base, "repo") os.makedirs(os.path.join(repo, "tools")) with open(os.path.join(repo, QUEUE_DEFAULT), "w") as fh: fh.write("## Q1 -- fixture\n") os.makedirs(os.path.join(repo, "docs")) with open(os.path.join(repo, INDEX_DEFAULT), "w") as fh: fh.write("fixture\n") if checkins is not None: os.makedirs(os.path.join(repo, CHECKINS_DEFAULT)) for name in checkins: with open(os.path.join(repo, CHECKINS_DEFAULT, name), "w") as fh: fh.write("Did: fixture\nBlocked: nothing\nNext: nothing\n" "Ask: nothing\n") for tool, rc in rcs.items(): if rc is None: # tool absent on purpose continue _stub(os.path.join(repo, "tools", tool), rc, out=(outs or {}).get(tool, ""), sleep=sleep, marker=(markers or {}).get(tool), argv_marker=(argv_markers or {}).get(tool)) _git_init(repo, commit=commit) return repo def _verify(*argv): """Run this file as the CLI, so the fixture grades the real entry point.""" proc = subprocess.run([sys.executable, os.path.abspath(__file__)] + list(argv), capture_output=True, text=True) return proc.returncode, (proc.stdout or "") + (proc.stderr or "") GREEN_ALL = {"checkin_check.py": 0, "reply_check.py": 0, "queue_index.py": 0, "queue_append_audit.py": 0, "selftest_sweep.py": 0, "write_probe.py": 0} def selftest(): cells = 0 passed = 0 failures = [] def check(label, got, want): nonlocal cells, passed cells += 1 if got == want: passed += 1 else: failures.append("%s\n got: %r\n want: %r" % (label, got, want)) with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td: # 1. six green stubs -> rc 0 repo = _fixture(os.path.join(td, "c1"), GREEN_ALL) rc, out = _verify("--repo", repo) check("1 all six legs green -> rc 0", rc, 0) check("1b summary counts the six legs", "verify: 6 legs, 6 pass, 0 fault, " "0 ungradable, 0 skipped" in out, True) # 2. a stale index is a FAULT, and its own line reaches the row repo = _fixture(os.path.join(td, "c2"), dict(GREEN_ALL, **{"queue_index.py": 1}), outs={"queue_index.py": "STALE: rendered document differs at line 7\n"}) rc, out = _verify("--repo", repo) check("2 stale index -> rc 1", rc, 1) check("2b the STALE line is carried into the row", "index-current" in out and "differs at line 7" in out, True) # 3. a refusal is UNGRADABLE, not a fault repo = _fixture(os.path.join(td, "c3"), dict(GREEN_ALL, **{"queue_index.py": 2}), outs={"queue_index.py": "cannot read queue: nope\n"}) rc, out = _verify("--repo", repo) check("3 refusal -> rc 2", rc, 2) check("3b refusal is named in the summary", "ungradable: index-current" in out, True) # 4. precedence: ungradable outranks fault, and the fault is NOT erased repo = _fixture(os.path.join(td, "c4"), dict(GREEN_ALL, **{"queue_index.py": 1, "selftest_sweep.py": 2}), outs={"queue_index.py": "STALE: line 3\n", "selftest_sweep.py": "cannot read: no tools\n"}) rc, out = _verify("--repo", repo) check("4 ungradable outranks fault -> rc 2", rc, 2) check("4b both classes are still printed in one run", "1 fault, 1 ungradable" in out and "fault: index-current" in out, True) # 5. a missing leg tool is ungradable, and says which path was missing repo = _fixture(os.path.join(td, "c5"), dict(GREEN_ALL, **{"queue_append_audit.py": None})) rc, out = _verify("--repo", repo) check("5 missing tool -> rc 2", rc, 2) check("5b the missing path is named", "no such tool:" in out, True) # 6. --fast skips the sweep and does not run it marker = os.path.join(td, "c6-sweep-ran") repo = _fixture(os.path.join(td, "c6"), GREEN_ALL, markers={"selftest_sweep.py": marker}) rc, out = _verify("--repo", repo, "--fast") check("6 --fast on a green tree -> rc 0", rc, 0) check("6b the skipped leg is printed as SKIP", "selftest-sweep" in out and "SKIP" in out, True) check("6c the skipped stub never ran", os.path.exists(marker), False) # 7. --json is a real machine-readable record repo = _fixture(os.path.join(td, "c7"), GREEN_ALL) jpath = os.path.join(td, "c7.json") rc, out = _verify("--repo", repo, "--json", jpath) obj = json.load(open(jpath)) if os.path.exists(jpath) else {} check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])], ["checkin", "reply-headings", "index-current", "append-audit", "selftest-sweep", "write-axis"]) check("7b --json carries the same rc the process returned", obj.get("rc"), rc) # 8. a repo that is not there is refused before any leg runs rc, out = _verify("--repo", os.path.join(td, "no-such-repo")) check("8 missing repo -> rc 2", rc, 2) check("8b the refusal names the path", "cannot read repo:" in out, True) # 9. rc 3 is retired: it is ungradable, never a fault and never a pass repo = _fixture(os.path.join(td, "c9"), dict(GREEN_ALL, **{"queue_index.py": 3}), outs={"queue_index.py": "unknown state\n"}) rc, out = _verify("--repo", repo) check("9 rc 3 -> rc 2 (retired code)", rc, 2) check("9b the unexpected code is named", "unexpected rc 3" in out, True) # 10. a hung leg is a timeout, not a pass and not a fault repo = _fixture(os.path.join(td, "c10"), dict(GREEN_ALL, **{"selftest_sweep.py": 0}), sleep=5.0) rc, out = _verify("--repo", repo, "--timeout", "1") check("10 timeout -> rc 2", rc, 2) check("10b the timeout is named", "TIMEOUT after 1s" in out, True) # 11. order is fixed, cheapest first: the table is not re-sorted by outcome repo = _fixture(os.path.join(td, "c11"), dict(GREEN_ALL, **{"queue_index.py": 1}), outs={"queue_index.py": "STALE: line 1\n"}) rc, out = _verify("--repo", repo) order = [out.find(k) for k in ("checkin", "reply-headings", "index-current", "append-audit", "selftest-sweep", "write-axis")] check("11 legs print in fixed order", order == sorted(order) and -1 not in order, True) # 12. a skip is counted as a skip, never folded into pass repo = _fixture(os.path.join(td, "c12"), GREEN_ALL) jpath = os.path.join(td, "c12.json") rc, out = _verify("--repo", repo, "--fast", "--json", jpath) obj = json.load(open(jpath)) if os.path.exists(jpath) else {} check("12 skipped leg is its own count in --json", obj.get("counts", {}).get("SKIP"), 1) check("12b and the skipped leg is not counted as a pass", obj.get("counts", {}).get(PASS), 5) # 13. the detail line must not contradict the verdict: a row graded # FAULT beside "0 fatal" is the SILENT failure this gate exists to catch. check("13 a zero-fault summary is not a reason line", pick_detail("sweep: 15 tools, 15 OK, 0 fatal\n" "read-refusal: 20 probes, 19 ok, 1 fault (NO_PROBE)\n" "malformed-input: 18 probes, 17 ok, 1 fault (NO_PROBE)\n"), "read-refusal: 20 probes, 19 ok, 1 fault (NO_PROBE)") check("13b a clean run still gets its sweep line", pick_detail("sweep: 14 tools, 14 OK, 0 fatal\n" "read-refusal: 19 probes, 19 ok, 0 fault\n"), "sweep: 14 tools, 14 OK, 0 fatal") # 14. THE HOLE THIS LEG CLOSES: a contract-breaking check-in is rc 1, # and the gate is no longer green on a tree that carries one. repo = _fixture(os.path.join(td, "c14"), dict(GREEN_ALL, **{"checkin_check.py": 1}), outs={"checkin_check.py": "FAIL: 2 file(s) violate the check-in contract\n"}) rc, out = _verify("--repo", repo, "--fast") check("14 a broken check-in -> rc 1 (was rc 0 with three legs)", rc, 1) check("14b the fault row is named checkin and carries its reason", "checkin" in out and "violate the check-in contract" in out and "fault: checkin" in out, True) # 15. no checkins/ directory is ungradable, not a pass and not a fault repo = _fixture(os.path.join(td, "c15"), GREEN_ALL, checkins=None) rc, out = _verify("--repo", repo, "--fast") check("15 no checkins dir -> rc 2", rc, 2) check("15b the ungradable row says which directory is missing", "ungradable: checkin" in out and "no check-in directory:" in out, True) # 16. the leg grades the WHOLE archive against its own ledger -- the # fix for a newest-day floor that made a backdated fault invisible. av = os.path.join(td, "c16-argv") repo = _fixture(os.path.join(td, "c16"), GREEN_ALL, argv_markers={"checkin_check.py": av}, checkins=("2026-01-01-fixture.md", "2026-03-04-fixture.md")) rc, out = _verify("--repo", repo, "--fast") seen = open(av).read() if os.path.exists(av) else "" check("16 the check-in leg is run with --fail", "--fail" in seen, True) check("16b and with the ledger resolved from --repo, so no day is excluded", "--baseline" in seen and os.path.join(repo, "tools", "checkin_baseline.tsv") in seen and "--since" not in seen, True) # 17. --checkin-day still narrows the window explicitly, in the open av = os.path.join(td, "c17-argv") repo = _fixture(os.path.join(td, "c17"), GREEN_ALL, argv_markers={"checkin_check.py": av}, checkins=("2026-01-01-fixture.md", "2026-03-04-fixture.md")) rc, out = _verify("--repo", repo, "--fast", "--checkin-day", "2026-01-01") seen = open(av).read() if os.path.exists(av) else "" check("17 --checkin-day adds a --since floor", "--since 2026-01-01" in seen, True) # 19. a ledger that cannot be read is UNGRADABLE (rc 2), never a pass: # the tool's own reason has to reach the row, because "no ledger" and # "clean archive" print nothing alike and must not grade alike. repo = _fixture(os.path.join(td, "c19"), dict(GREEN_ALL, **{"checkin_check.py": 2}), outs={"checkin_check.py": "cannot grade: baseline %s/tools/checkin_baseline.tsv: " "[Errno 2] No such file or directory\n" % os.path.join(td, "c19", "repo")}) rc, out = _verify("--repo", repo, "--fast") check("19 a missing ledger is rc 2 and names the ledger", rc, 2) check("19b the ungradable row carries the tool's own reason", "ungradable: checkin" in out and "checkin_baseline.tsv" in out, True) # 18. the detail line must survive a real tool's noise: `checkin_check` # prints a README SKIP note carrying "not a " on every run, and a WEAK # hint must not beat the FAIL line that actually names the verdict. noisy = ("== 2 check-in files in /tmp/x/checkins ==\n" "roster: 9 handles; today 2026-10-19\n" "since 2026-10-19: 2 graded, 72 older file(s) not graded\n" " SKIP README.md (the directory's own README: not a check-in)\n" "\n" "2026-10-19-volt-02.md BAD_NAME,DUPLICATE 2 files for volt\n" "checkin_check: 2 files, 2 with a fault (BAD_NAME 1)\n" "FAIL: 2 file(s) violate the check-in contract\n") check("18 a README skip note does not mask the FAIL line", pick_detail(noisy), "FAIL: 2 file(s) violate the check-in contract") # 18b. and on a clean run the same noise must not become the detail: # the row reads PASS beside the summary, not beside a README note. clean = noisy.replace("checkin_check: 2 files, 2 with a fault (BAD_NAME 1)\n" "FAIL: 2 file(s) violate the check-in contract\n", "checkin_check: 2 files, 0 with a fault, 1 advisory\n") check("18b a clean check-in row carries its summary, not the skip note", pick_detail(clean), "checkin_check: 2 files, 0 with a fault, 1 advisory") # 20. THE FIFTH LEG: a reply whose heading names the wrong party is a # FAULT on the gate and the row is named -- the class that was invisible # for as long as `replies/` had no reader. repo = _fixture(os.path.join(td, "c20"), dict(GREEN_ALL, **{"reply_check.py": 1}), outs={"reply_check.py": "reply_check: 192 file(s), 1 new fault(s), " "0 known-faulting in the baseline\n"}) rc, out = _verify("--repo", repo, "--fast") check("20 a new mis-addressed reply -> rc 1", rc, 1) check("20b the fault row is named reply-headings and carries its reason", "fault: reply-headings" in out and "1 new fault(s)" in out, True) # 21. the ledger is read from the tree being graded, not from the host av = os.path.join(td, "c21-argv") repo = _fixture(os.path.join(td, "c21"), GREEN_ALL, argv_markers={"reply_check.py": av}) rc, out = _verify("--repo", repo, "--fast") seen = open(av).read() if os.path.exists(av) else "" check("21 the reply leg is run with the repo's own ledger", "--baseline" in seen and os.path.join(repo, "tools", "reply_baseline.tsv") in seen, True) # 22. on a PASS row the detail is the tool's own concluding summary, not # the first line that happens to contain a hint substring. reply_noise = ("reply_check -- 192 file(s) under /tmp/r (roster: x)\n" " NO_RECIPIENT_IN_HEADING 7\n" " 2026-10-12-to-merlin-19.md | heading: # the precondition refuses\n" "reply_check: 192 file(s), 0 new fault(s), 1 known-faulting in the baseline\n") check("22 a clean row's detail is the tool's summary", pick_detail(reply_noise, summary=True), "reply_check: 192 file(s), 0 new fault(s), 1 known-faulting in the baseline") check("22b a fault run still prefers the reason", pick_detail("cannot grade: baseline x: No such file\n" "tool: 1 new fault(s)\n"), "cannot grade: baseline x: No such file") # 23. THE SANDBOX THE GATE REFUSED. A linked worktree's `.git` is a FILE # (`gitdir: ...`), not a directory, so `os.path.isdir(repo/.git)` -- the # test this gate used until 2026-10-19 -- answered "not a git repository" # (rc 2) on a checkout every leg would have graded. Measured RED against # the old test on this exact fixture, GREEN here. repo = _fixture(os.path.join(td, "c23"), GREEN_ALL, commit=True) wt = os.path.join(td, "c23-wt") subprocess.run(["git", "-C", repo, "worktree", "add", "--detach", wt, "HEAD"], check=True, capture_output=True, text=True) with open(os.path.join(wt, ".git")) as fh: gitfile = fh.read() check("23 the fixture worktree really carries a .git file, not a dir", gitfile.startswith("gitdir: ") and not os.path.isdir(wt + "/.git"), True) rc, out = _verify("--repo", wt, "--fast") check("23b a linked worktree is graded, not refused -> rc 0", rc, 0) check("23c the row names the worktree, not the tree it links to", "verify -- repo: %s" % os.path.abspath(wt) in out, True) # 24. --repo stays the ROOT: a subdirectory is refused with the root # named, because widening it silently would grade a tree nobody asked for rc, out = _verify("--repo", os.path.join(repo, "tools"), "--fast") check("24 a subdirectory is refused, not widened -> rc 2", rc, 2) check("24b the refusal names the root it would have graded", "--repo takes the root" in out and os.path.realpath(repo) in out, True) # 25. the OLD test's false positive: an empty `.git` directory is not a # repository, and git says so where `isdir` said otherwise. fake = os.path.join(td, "c25") os.makedirs(os.path.join(fake, ".git")) rc, out = _verify("--repo", fake, "--fast") check("25 an empty '.git' directory is not a repo -> rc 2", rc, 2) check("25b and the refusal says git refused it", "is not a git work tree" in out and "cannot read repo:" in out, True) # 26. a bare repository has no work tree to grade bare = os.path.join(td, "c26.git") subprocess.run(["git", "init", "-q", "--bare", bare], check=True, capture_output=True, text=True) rc, out = _verify("--repo", bare, "--fast") check("26 a bare repository -> rc 2", rc, 2) # 27. the count in this file's own docstring is a CLAIM, so it is graded # against the count that actually ran. On 2026-10-19 the docstring and # the run said 50 while the argparse help string still said 42 and # nothing failed -- the same count-word defect this repo keeps finding in # documents, closed in the document that prints the number. check("27 the docstring's cell count is the count that ran", DOC_CELLS, cells + 1) print("selftest: %d cells, %d passed, %d failed" % (cells, passed, len(failures))) for f in failures: print(" FAIL %s" % f) return 0 if passed == cells else 1 if __name__ == "__main__": sys.exit(main())