From c6180b6d7d050750e6220324bd961dea1517edbc Mon Sep 17 00:00:00 2001 From: Volt Date: Sun, 18 Oct 2026 19:34:48 +0000 Subject: [PATCH] tools (volt): verify.py -- the gate's home: three legs, one rc, every red path exercised index-current (queue_index --check), append-audit (queue_append_audit --guard --to HEAD), selftest-sweep (skippable with --fast, printed SKIP and counted as skipped). rc 0 graded clean / 1 graded a fault / 2 could not grade, with 2 outranking 1 so an ungradable leg cannot be read as a certified run; the fault count is still printed beside it. Tools resolve from --repo, not from this file's directory, so a worktree is graded with its own instruments. selftest 24/24 (stub legs through the real CLI: rc mapping incl. retired rc 3, missing tool, timeout, --fast, --json, missing repo, fixed order). Added to both sweep probe axes (read-refusal 19/19, malformed 17/17, 14 tools 14 OK) and the sweep's stale "15 such probes / nine tools" prose corrected to 17 / ten. Green at 019b75c: 3/3 pass, rc 0. Red paths on a full 785-commit clone: 0 -> 1 (working-tree append, no re-render) -> 0 (--write) -> 1 (committed stale: both legs faulted) -> 2 (tool absent; index fault still printed). Queue tick appended and index re-rendered in this same commit. --- WORK-QUEUE.md | 16 ++ checkins/2026-10-18-volt.md | 48 ++++ docs/queue-index.md | 18 +- tools/selftest_sweep.py | 11 +- tools/verify.py | 424 ++++++++++++++++++++++++++++++++++++ 5 files changed, 505 insertions(+), 12 deletions(-) create mode 100644 tools/verify.py diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index 8e1c7d4..2285070 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -10015,3 +10015,19 @@ Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-index-stale/` (`REPORT.m - **The guard, adopted in-tick.** Before this append I ran `python3 tools/queue_append_audit.py --guard --to HEAD` (**rc 0**, *"every graded commit left the index current"*) and `queue_index --check` (**index current**) — the same two commands this tick is recording as the precondition. The rule is no longer a sentence I have to remember; it is a command whose rc gates the append, and this block is the first one written under it. - **The latent bug fixed in `8a450a3` is the more valuable half.** `queue_index.py --write` would have written a **0-byte** `docs/queue-index.md` and exited 1 on any queue with no nested records (`IndexError` on a heading-only queue) — a silent corruption of the derived document that would then have graded as a "fault" rather than as damage. The fix is 23/7 numstat, and the render of the real queue is byte-identical before and after. - **Nothing operational.** No item opened or closed; open set unchanged (Q37 `:5278` pipe, career-lane Q53 `:7303`); no issue, PR, release or deploy surface; no another-party file edited. Vincent's route remains dead on expired auth; Praetorian's jobs remain `blocked_config`. No secret printed. + + +**Volt tick 2026-10-18 19:33Z — the gate gets its home: `tools/verify.py`, one command, three legs, and every leg's red path exercised on a clone.** (Pure suffix append; clock read immediately before this edit; the file's true last line at `019b75c` was the `Nothing operational` bullet closing the 17:44Z block, so nothing above this anchor was touched. The index is re-rendered in this same commit and `--check` reads rc 0 at the commit this block lands in.) + +- **The advice I could not stop repeating is now a command.** Since 15:15Z this lane told Alexander to wire `queue_index --check` into "the tick's `check.sh`" — a file that does not exist (`git ls-files` shows `tools/` and nothing else). The 17:44Z block corrected the advice and left it as *"the gate needs a HOME … and that is a decision, not a wiring step."* Half of that stands and half does not: whether every lane **must** run it is still Alexander's call, but **building** the home never needed a decision, only a task. `tools/verify.py` is that task, shipped: **18,723 B / sha256 `7bea22608b3e19b5…` / 424 lines**. +- **What it runs, in fixed order, cheapest first, never in parallel.** `index-current` = this repo's own `queue_index.py --check docs/queue-index.md` (the derived document byte-compared against a fresh render); `append-audit` = this repo's own `queue_append_audit.py --guard --to HEAD` (the newest queue-touching commit graded in its own tree); `selftest-sweep` = `selftest_sweep.py` (every tool's `--selftest` plus both read-refusal axes), skippable with `--fast` and then printed as SKIP and counted as **skipped** — never as a pass. A leg is never skipped because an earlier leg failed: one run names both defects. +- **The exit contract is the house three, with `verify.py`'s own precedence stated.** 0 every graded leg passed; 1 a leg ran and graded a fault; 2 a leg could not grade (missing tool, timeout, unexpected rc, unreadable repo) or nothing was runnable. **rc 2 outranks rc 1 deliberately** — rc 1 claims "I graded a fault", and a run with an ungradable leg has not made that claim about the whole gate. Precedence never erases a fault: the summary prints the fault count and names the faulted leg beside the ungradable one (row 5 of the matrix below is exactly this case). +- **Green at `019b75c`, measured:** `python3 tools/verify.py` → **3 legs, 3 pass, 0 fault, 0 ungradable, rc 0**, 25.3 s wall (`index-current` 389 ms, `append-audit` 822 ms, `selftest-sweep` 23,833 ms), with `index current: 49 sections, 45 ids, 4 continuation sections, 9 nested ids, 10017 lines / 1250906 bytes` and `verdict: every graded commit left the index current in its own tree` as the two detail lines. +- **Every red path is exercised, not asserted.** Against a **full clone of `019b75c` (785 commits)** at `/root/.hermes/profiles/volt/cache/scratch/verify-red-full-193221/repo`, with the untracked `verify.py` from the source checkout driving that clone's own tools: (1) baseline **rc 0**; (2) one working-tree queue append, index not re-rendered → **rc 1**, `index-current FAULT … STALE: rendered document differs at line 59`; (3) the clone's own `queue_index --write docs/queue-index.md` → **rc 0** again; (4) the same append **committed** without the re-render (`c5d685b`) → **rc 1 with two legs faulted**, `index-current` and `append-audit STALE c5d685b … index:yes`, the historical defect reproduced in one command; (5) `tools/queue_append_audit.py` removed from the tree under test → **rc 2** with `no such tool: …` **and the still-live `index-current` fault printed beside it**. Transcript: `red-matrix-full.txt`, script `red-matrix.sh`. +- **A shallow clone grades as UNGRADABLE, which is the correct grade and worth recording.** The same matrix on a `--depth 1` clone first gave `append-audit UNGRADABLE rc=2 — queue_append_audit: no commit in HEAD touches WORK-QUEUE.md, so there is no index currency to grade`, i.e. **a cannot-grade result, not a false green and not a stale verdict** on a repo whose parent commits are absent. That is the distinction the 15:16Z correction was written for, observed here rather than argued. +- **The new tool is on both sweep axes, and the sweep's prose was corrected while adding it.** `verify.py` joined `READ_PROBES` (`--repo {nope}`) and `MALFORMED_PROBES` (`{notrepo}`, rc 2, `cannot read repo`), so it is not left as a `NO_PROBE` advisory. Its docstring prose said **15** malformed probes while the tool printed **16** before this tick — a one-off already stale, now corrected to **17** with `nine tools must refuse at rc 2` corrected to **ten**. Measured pre-commit with `--tools-dir tools`: **14 tools, 14 OK, 0 fatal**, `verify.py OK 24 cells`, read-refusal **19 probes, 19 ok**, malformed **17 probes, 17 ok**, rc 0. +- **`verify.py --selftest`: 24 cells, 24 passed, 0 failed** — stub legs in a throwaway repo, driven through the real CLI, covering rc 0/1/2/3 mapping, a missing tool, a hung leg (`TIMEOUT after 1s`), `--fast` (row SKIP, stub provably not executed, skip count in `--json`), `--json` shape and rc agreement, a missing repo refused before any leg runs, and the fixed print order. +- **What it does not do.** It does not enforce anything, repair a stale commit, or adjudicate closure (`queue_claim_check.py` owns that verdict). It grades the tree you point it at; `--repo` is how you grade a revision's own tools, and the `--queue`/`--index` arguments stay repo-relative because `queue_append_audit` reads them with `git show :`. +- **Nothing operational.** No queue item opened or closed; open set unchanged (Q37 `:5278` pipe, career-lane Q53 `:7303` held for Alexander); no issue/PR/release/deploy surface; no another-party file edited (the two files changed are this lane's own `tools/`); no counterparty messaged; no secret printed. Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-verify-home/`. + +— Volt diff --git a/checkins/2026-10-18-volt.md b/checkins/2026-10-18-volt.md index d7c244c..47cde18 100644 --- a/checkins/2026-10-18-volt.md +++ b/checkins/2026-10-18-volt.md @@ -484,3 +484,51 @@ Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-index-stale/` — Volt **CORRECTION (15:2xZ, same run, append-only).** The ledger block above states *"In scope since the index was introduced at `1423c4d` (10-17): 6 commits touch the queue, 3 also re-rendered it, 3 left the pin stale at their own commit"* — a mutable count with no revision named, which by this repo's own rule reads stale forever. It is measured **at `69f3b56`**. At the landing commit `de3dcdc` the same count is **7 touch / 4 re-rendered / the same 3 still stale**, the seventh being the repair commit itself (a queue+index commit, so it is in the re-rendered class, not the stale one). The sentence stays as written; this line is the frame it should have carried. + + +## 19:33Z — the gate has a home: `tools/verify.py` + +**What landed.** `tools/verify.py` (18,723 B / sha256 `7bea2260…` / 424 lines), +tracked in this commit with the sweep-probe additions in `tools/selftest_sweep.py`. +One command, three legs in fixed order, one exit code: `index-current` +(`queue_index --check docs/queue-index.md`), `append-audit` +(`queue_append_audit --guard --to HEAD`), `selftest-sweep` (skippable with +`--fast`, printed as SKIP and counted as skipped). rc 0 all graded legs pass, +rc 1 a leg ran and graded a fault, rc 2 a leg could not grade — **2 outranks 1** +because a run with an ungradable leg has not certified anything, and the summary +still prints the fault count beside it. + +**Why it exists.** Three ticks of my own advice told Alexander to wire +`queue_index --check` into "the tick's `check.sh`". `git ls-files` shows no +`check.sh` and no `scripts/`; the advice named a target that does not exist. +Whether every lane must run the gate is still Alexander's call; building the +home was a task, and this is it. + +**Measured.** Green at `019b75c`, full run: **3 legs, 3 pass, rc 0**, 25.3 s +(389 / 822 / 23,833 ms). Red paths on a **full clone (785 commits)** of +`019b75c`, `verify.py` driving that clone's own tools: baseline rc 0 → one +working-tree queue append without re-render **rc 1** (`STALE … line 59`) → +clone's own `--write` **rc 0** → the same append committed without re-render +(`c5d685b`) **rc 1 with both legs faulted** → `queue_append_audit.py` removed +**rc 2** with the still-live index fault printed beside it. A `--depth 1` clone +grades `append-audit` **UNGRADABLE rc 2** ("no commit in HEAD touches +WORK-QUEUE.md") — cannot-grade, not a false green. `--selftest` **24/24**. +Sweep with `--tools-dir tools`: **14 tools, 14 OK**, read-refusal **19/19**, +malformed **17/17**, rc 0 (prose said 15 malformed probes while the tool printed +16; corrected to 17 with "nine tools" → "ten"). + +**Blocked.** Nothing. + +**Next.** The home exists; adoption is the open question. A lane runs +`python3 tools/verify.py` in its tick (this one did), or Alexander wires it into +whatever runner the troupe settles on — the script does not decide that. + +**Ask.** Nothing for Alexander that is new: the standing adoption call (does +every tick run `verify.py`?) is the same policy question the 17:44Z block named +as needing a home, and the home now exists to be adopted. + +Artifact: `/root/.hermes/wizard-outputs/volt/2026-10-18-verify-home/` +(`REPORT.md`, `red-matrix.sh`, `red-matrix-full.txt`, `verify-green.txt`, +`verify-green.json`, `selftest.txt`, `sweep.txt`). + +— Volt diff --git a/docs/queue-index.md b/docs/queue-index.md index 7d29e4b..e8945cb 100644 --- a/docs/queue-index.md +++ b/docs/queue-index.md @@ -56,7 +56,7 @@ | 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot | | 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree | | 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 | -| 49 | Q45 | 7594-10017 | 358645 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | -- | +| 49 | Q45 | 7594-10033 | 364555 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | -- | ## Ids with no `## ` heading @@ -71,7 +71,7 @@ Opened as records inside another section. `line` is the record's own opening lin | Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD | -| Q53 | 7292 | 7246 | 110 | Q49 | BULLET+BOLD | BOLD-LEAD | +| Q53 | 7292 | 7246 | 111 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q63 | 9918 | 9909 | 16 | Q45 | BULLET+BOLD | BOLD-LEAD | **Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`. @@ -217,7 +217,7 @@ The block below is the parse this document renders, byte for byte. `--check` com "form": "BOLD-LEAD", "id": "Q53", "line": 7292, - "refs": 110, + "refs": 111, "section": "Q49", "shape": "BULLET+BOLD" }, @@ -236,9 +236,9 @@ The block below is the parse this document renders, byte for byte. `--check` com "end_line": 8, "line": 1 }, - "queue_bytes": 1250906, - "queue_lines": 10017, - "queue_sha256": "7262eb3398a79a8d02c7ef80daa5a8a8a51f2a876db5050a342cf162ccb98b3d", + "queue_bytes": 1256816, + "queue_lines": 10033, + "queue_sha256": "cd49526d58e89a073033917f552fd9eb3f93bcf62fcd004c48c1ce3a35b9d1af", "sections": [ { "bytes": 3139, @@ -817,11 +817,11 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "The runtime-resolved root's live write path: name it, reproduce it, then close it" }, { - "bytes": 358645, - "end_line": 10017, + "bytes": 364555, + "end_line": 10033, "id": "Q45", "line": 7594, - "lines": 2424, + "lines": 2440, "role": "opening", "role_detail": "", "status": null, diff --git a/tools/selftest_sweep.py b/tools/selftest_sweep.py index 47ee0c2..a282e7b 100644 --- a/tools/selftest_sweep.py +++ b/tools/selftest_sweep.py @@ -56,9 +56,9 @@ THE MALFORMED-INPUT CLASS, ADDED 2026-10-18 should be, a directory that is not a git repository, a queue file with no records. Those are the inputs where a tool is most tempted to answer "0" and return 0, which reads exactly like a clean pass (`doc_census_check`'s - `--tools` bug, `76043b7`, was this class). The axis now runs 15 such probes + `--tools` bug, `76043b7`, was this class). The axis now runs 17 such probes and grades each against ITS OWN stated answer, because "must exit 2" is not - a universal truth: nine tools must refuse at rc 2, while `check_lint` grades + a universal truth: ten tools must refuse at rc 2, while `check_lint` grades a junk log MISSING (rc 0 in report mode, rc 1 under --fail-on-fault), `checkin_check` grades a heading-less check-in as an advisory (rc 0, stated in its docstring) and `queue_claim_check` reports 0 opening records (rc 0, @@ -157,7 +157,8 @@ READ_PROBES = ( ("queue_claim_check.py", ["--queue", "{nope}/q.md"]), ("queue_index.py", ["--queue", "{nope}/q.md", "--check", "{nope}/i.md"]), ("queue_append_audit.py", ["--repo", "{nope}"]), - ("selftest_sweep.py", ["--repo", "{nope}"]) + ("selftest_sweep.py", ["--repo", "{nope}"]), + ("verify.py", ["--repo", "{nope}"]) ) NOT_UTF8_PROBES = ( ("append_only_owners.py", ["{bin}"]), @@ -212,6 +213,10 @@ MALFORMED_PROBES = ( 2, "not a git repository"), ("selftest_sweep.py", "not-a-repo", ["--repo", "{notrepo}"], 2, "no tools discovered"), + # verify.py's own two legs are the repo it is asked to grade; a directory + # that is not a git repository is the malformed input for both of them. + ("verify.py", "not-a-repo", ["--repo", "{notrepo}"], + 2, "cannot read repo"), ) diff --git a/tools/verify.py b/tools/verify.py new file mode 100644 index 0000000..94b044e --- /dev/null +++ b/tools/verify.py @@ -0,0 +1,424 @@ +#!/usr/bin/env python3 +"""verify.py -- the HOME the index gate did not have: one command, three legs, +one exit code, and a row per leg so a partial run cannot look like a clean one. + +WHY THIS EXISTS + On 2026-10-18 this lane told Alexander, three ticks running, to "wire + `queue_index --check` into the tick's check.sh". There is no `check.sh` in + this repo and no `scripts/` directory at all -- `git ls-files` shows `tools/` + and nothing else. The advice named a target that does not exist, which is why + it was never adopted. The gate did not need a decision about policy; it needed + a home. This is that home, and it is a task, not a policy call: nothing here + obliges any lane to run it, and nothing is enforced until a lane does. + + The second reason is narrower and this lane's own: three separate counts of + "which commits left the index stale" came out wrong in one week, each from a + hand-assembled probe loop. A gate whose legs are remembered is a gate that + gets run selectively. Three legs, fixed order, one rc -- that is the whole + design. + +WHAT IT RUNS -- one row per leg, cheapest first, never in parallel + index-current tools/queue_index.py --check docs/queue-index.md + the derived document, byte-compared against a fresh render + append-audit tools/queue_append_audit.py --guard --to HEAD + the newest queue-touching commit, graded in ITS OWN tree + selftest-sweep tools/selftest_sweep.py + every tool's --selftest plus its two read-refusal axes + (skip with --fast; the row is still printed and counted) + + The legs are ordered by cost, and a leg is never skipped because an earlier + one failed: a stale index and a broken selftest are different defects and the + run should name both in one pass. + +THE EXIT CONTRACT -- the house three codes + 0 every graded leg passed + 1 at least one leg ran and graded a fault + 2 at least one leg could not grade -- a missing tool, a timeout, an + unexpected exit code, an unreadable repo -- or no leg was runnable at all + + rc 2 OUTRANKS rc 1 deliberately. rc 1 means "I ran and I graded a fault"; + a run with an ungradable leg has not made that claim about the whole gate, + and folding the two (as the 15:16Z correction caught `queue_append_audit` + doing with ungradable commits) is how a "3 stale" reading hides a "2 could + not be read". The summary always prints both counts, so precedence never + erases a fault -- it only refuses to certify the run that found it. + + A skipped leg is not a pass and not a fault: it is counted as `skipped` and + printed as SKIP. `--fast` on a green tree exits 0 with one row SKIP, and the + summary says so rather than pretending the sweep ran. + +TOOLS ARE RESOLVED FROM --repo, NOT FROM THIS FILE'S DIRECTORY + `verify.py --repo /path/to/worktree` runs THAT tree's `queue_index.py`, + `queue_append_audit.py` and `selftest_sweep.py`. That is the instrument + lesson the 15:16Z correction paid for: a gate run in a working tree at HEAD + says nothing about a cited revision, so grading a revision means grading its + own tools from its own checkout. + + --queue and --index stay REPO-RELATIVE (defaults WORK-QUEUE.md and + docs/queue-index.md) because `queue_append_audit` reads them with + `git show :`, which has no absolute-path form. + +`--selftest` fixture corpus, 24 cells +""" + +import argparse +import json +import os +import subprocess +import sys +import tempfile +import time + +QUEUE_DEFAULT = "WORK-QUEUE.md" +INDEX_DEFAULT = "docs/queue-index.md" + +PASS, FAULT, UNGRADABLE, SKIP = "PASS", "FAULT", "UNGRADABLE", "SKIP" + +# Legs, in the fixed order they are printed and run. `tool` is resolved inside +# the repo being graded. `slow` marks the leg --fast drops. +LEGS = ( + {"key": "index-current", "tool": "queue_index.py", "slow": False, + "what": "render the index and byte-compare it with the document"}, + {"key": "append-audit", "tool": "queue_append_audit.py", "slow": False, + "what": "grade the newest queue-touching commit in its own tree"}, + {"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True, + "what": "every tool's --selftest and both read-refusal axes"}, +) + +# A detail line is a courtesy, not the verdict -- but a verdict with no line +# beside it is what the read-refusal axis calls SILENT, so it is graded too. +HINTS = ("STALE:", "FAIL", "FAILED", "fatal", "cannot", "TIMEOUT", "NO_", + "error", "Error", "refus", "not a ") + + +def pick_detail(text, limit=150): + """First line that names a reason, else the last line that says anything.""" + lines = [ln.strip() for ln in text.splitlines() if ln.strip()] + for ln in lines: + if any(h in ln for h in HINTS): + return ln[:limit] + return lines[-1][:limit] if lines else "" + + +def leg_argv(key, repo, queue, index): + if key == "index-current": + return [sys.executable, os.path.join(repo, "tools", "queue_index.py"), + "--repo", repo, "--queue", queue, "--check", index] + if key == "append-audit": + return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"), + "--repo", repo, "--queue", queue, "--index", index, + "--guard", "--to", "HEAD"] + if key == "selftest-sweep": + return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"), + "--repo", repo, "--quiet"] + raise KeyError(key) + + +def run_leg(leg, repo, queue, index, timeout): + """One leg -> one row. rc 0/1/2 are graded; anything else is ungradable.""" + tool = os.path.join(repo, "tools", leg["tool"]) + started = time.time() + row = {"key": leg["key"], "tool": leg["tool"], "verdict": None, + "rc": None, "detail": "", "what": leg["what"]} + if not os.path.exists(tool): + row.update(verdict=UNGRADABLE, detail="no such tool: %s" % tool) + row["ms"] = 0 + return row + try: + proc = subprocess.run(leg_argv(leg["key"], repo, queue, index), + cwd=repo, capture_output=True, text=True, + timeout=timeout) + except subprocess.TimeoutExpired: + row.update(verdict=UNGRADABLE, detail="TIMEOUT after %ss" % timeout) + row["ms"] = int((time.time() - started) * 1000) + return row + except OSError as exc: + row.update(verdict=UNGRADABLE, detail="could not run: %s" % exc) + row["ms"] = int((time.time() - started) * 1000) + return row + + out = (proc.stdout or "") + (proc.stderr or "") + row["rc"] = proc.returncode + row["detail"] = pick_detail(out) + if proc.returncode == 0: + row["verdict"] = PASS + elif proc.returncode == 1: + row["verdict"] = FAULT + elif proc.returncode == 2: + row["verdict"] = UNGRADABLE + else: + # rc 3 is retired in this repo (one meaning, two codes); any other code + # is a tool that left the contract, and the run cannot grade through it. + row["verdict"] = UNGRADABLE + row["detail"] = "unexpected rc %d: %s" % (proc.returncode, row["detail"]) + row["ms"] = int((time.time() - started) * 1000) + return row + + +def counts(rows): + c = {PASS: 0, FAULT: 0, UNGRADABLE: 0, SKIP: 0} + for r in rows: + c[r["verdict"]] += 1 + return c + + +def decide(rows): + """The precedence, in one place: ungradable outranks fault outranks clean.""" + c = counts(rows) + if c[UNGRADABLE]: + return 2 + if c[FAULT]: + return 1 + return 0 + + +def report(rows, repo, quiet=False): + c = counts(rows) + if not quiet: + print("verify -- repo: %s" % repo) + for r in rows: + rc = "-" if r["rc"] is None else str(r["rc"]) + print(" %-16s %-10s rc=%-2s %-6s %s" + % (r["key"], r["verdict"], rc, "%dms" % r["ms"], r["detail"])) + print() + print("verify: %d legs, %d pass, %d fault, %d ungradable, %d skipped" + % (len(rows), c[PASS], c[FAULT], c[UNGRADABLE], c[SKIP])) + faulted = [r["key"] for r in rows if r["verdict"] == FAULT] + ungraded = [r["key"] for r in rows if r["verdict"] == UNGRADABLE] + if faulted: + print("fault: %s" % " ".join(faulted)) + if ungraded: + print("ungradable: %s" % " ".join(ungraded)) + rc = decide(rows) + if rc == 0: + print("gate: OK -- %d of %d graded leg(s) passed%s" + % (c[PASS], c[PASS] + c[FAULT], + "" if not c[SKIP] else + " (%d skipped: not a pass, not a fault)" % c[SKIP])) + elif rc == 1: + print("gate: FAULT -- %d leg(s) ran and graded a fault: %s" + % (c[FAULT], " ".join(faulted))) + else: + print("gate: NOT CERTIFIED -- %d leg(s) could not grade: %s" + % (c[UNGRADABLE], " ".join(ungraded))) + return rc + + +def main(argv=None): + ap = argparse.ArgumentParser( + description="Run the repo's gates as one command: index currency, the " + "append audit, and the selftest sweep.") + ap.add_argument("--repo", default=".") + ap.add_argument("--queue", default=QUEUE_DEFAULT, + help="repo-relative queue path (default %s)" % QUEUE_DEFAULT) + ap.add_argument("--index", default=INDEX_DEFAULT, + help="repo-relative index document (default %s)" % INDEX_DEFAULT) + ap.add_argument("--fast", action="store_true", + help="skip the selftest-sweep leg; the row is printed as SKIP") + ap.add_argument("--timeout", type=int, default=300, metavar="S", + help="per-leg timeout in seconds (default 300)") + ap.add_argument("--json", metavar="PATH", help="write the rows here") + ap.add_argument("--quiet", action="store_true") + ap.add_argument("--selftest", action="store_true", help="fixture corpus, 24 cells") + a = ap.parse_args(argv) + + if a.selftest: + return selftest() + + repo = os.path.abspath(a.repo) + if not os.path.isdir(repo): + print("cannot read repo: %s is not a directory" % repo, file=sys.stderr) + return 2 + if not os.path.isdir(os.path.join(repo, ".git")): + print("cannot read repo: %s is not a git repository" % repo, file=sys.stderr) + return 2 + + rows = [] + for leg in LEGS: + if a.fast and leg["slow"]: + rows.append({"key": leg["key"], "tool": leg["tool"], "verdict": SKIP, + "rc": None, "detail": "--fast: leg not run", + "what": leg["what"], "ms": 0}) + continue + rows.append(run_leg(leg, repo, a.queue, a.index, a.timeout)) + + rc = report(rows, repo, a.quiet) + if a.json: + c = counts(rows) + with open(a.json, "w") as fh: + json.dump({"repo": repo, "queue": a.queue, "index": a.index, + "fast": a.fast, "rows": rows, "counts": c, "rc": rc}, + fh, indent=1) + return rc + + +# ------------------------------------------------------------------- selftest + +def _stub(path, rc, out="", sleep=0.0, marker=None): + """A stand-in leg: it prints what the fixture needs and exits with its rc.""" + body = ["#!/usr/bin/env python3", "import sys"] + if sleep: + body.append("import time; time.sleep(%r)" % sleep) + if marker: + body.append("open(%r, 'a').write('hit\\n')" % marker) + if out: + body.append("sys.stdout.write(%r)" % out) + body.append("sys.exit(%d)" % rc) + with open(path, "w") as fh: + fh.write("\n".join(body) + "\n") + os.chmod(path, 0o755) + + +def _fixture(base, rcs, outs=None, sleep=0.0, markers=None): + """A throwaway repo with three stub legs. rcs: {tool: rc}.""" + repo = os.path.join(base, "repo") + os.makedirs(os.path.join(repo, "tools")) + os.makedirs(os.path.join(repo, ".git")) + with open(os.path.join(repo, QUEUE_DEFAULT), "w") as fh: + fh.write("## Q1 -- fixture\n") + os.makedirs(os.path.join(repo, "docs")) + with open(os.path.join(repo, INDEX_DEFAULT), "w") as fh: + fh.write("fixture\n") + for tool, rc in rcs.items(): + if rc is None: # tool absent on purpose + continue + _stub(os.path.join(repo, "tools", tool), rc, + out=(outs or {}).get(tool, ""), sleep=sleep, + marker=(markers or {}).get(tool)) + return repo + + +def _verify(*argv): + """Run this file as the CLI, so the fixture grades the real entry point.""" + proc = subprocess.run([sys.executable, os.path.abspath(__file__)] + list(argv), + capture_output=True, text=True) + return proc.returncode, (proc.stdout or "") + (proc.stderr or "") + + +THREE = {"queue_index.py": 0, "queue_append_audit.py": 0, "selftest_sweep.py": 0} + + +def selftest(): + cells = 0 + passed = 0 + failures = [] + + def check(label, got, want): + nonlocal cells, passed + cells += 1 + if got == want: + passed += 1 + else: + failures.append("%s\n got: %r\n want: %r" % (label, got, want)) + + with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td: + # 1. three green stubs -> rc 0 + repo = _fixture(os.path.join(td, "c1"), THREE) + rc, out = _verify("--repo", repo) + check("1 all three legs green -> rc 0", rc, 0) + check("1b summary counts the three legs", "verify: 3 legs, 3 pass, 0 fault, " + "0 ungradable, 0 skipped" in out, True) + + # 2. a stale index is a FAULT, and its own line reaches the row + repo = _fixture(os.path.join(td, "c2"), + dict(THREE, **{"queue_index.py": 1}), + outs={"queue_index.py": + "STALE: rendered document differs at line 7\n"}) + rc, out = _verify("--repo", repo) + check("2 stale index -> rc 1", rc, 1) + check("2b the STALE line is carried into the row", + "index-current" in out and "differs at line 7" in out, True) + + # 3. a refusal is UNGRADABLE, not a fault + repo = _fixture(os.path.join(td, "c3"), + dict(THREE, **{"queue_index.py": 2}), + outs={"queue_index.py": "cannot read queue: nope\n"}) + rc, out = _verify("--repo", repo) + check("3 refusal -> rc 2", rc, 2) + check("3b refusal is named in the summary", "ungradable: index-current" in out, True) + + # 4. precedence: ungradable outranks fault, and the fault is NOT erased + repo = _fixture(os.path.join(td, "c4"), + dict(THREE, **{"queue_index.py": 1, + "selftest_sweep.py": 2}), + outs={"queue_index.py": "STALE: line 3\n", + "selftest_sweep.py": "cannot read: no tools\n"}) + rc, out = _verify("--repo", repo) + check("4 ungradable outranks fault -> rc 2", rc, 2) + check("4b both classes are still printed in one run", + "1 fault, 1 ungradable" in out and "fault: index-current" in out, + True) + + # 5. a missing leg tool is ungradable, and says which path was missing + repo = _fixture(os.path.join(td, "c5"), + dict(THREE, **{"queue_append_audit.py": None})) + rc, out = _verify("--repo", repo) + check("5 missing tool -> rc 2", rc, 2) + check("5b the missing path is named", "no such tool:" in out, True) + + # 6. --fast skips the sweep and does not run it + marker = os.path.join(td, "c6-sweep-ran") + repo = _fixture(os.path.join(td, "c6"), THREE, markers={"selftest_sweep.py": marker}) + rc, out = _verify("--repo", repo, "--fast") + check("6 --fast on a green tree -> rc 0", rc, 0) + check("6b the skipped leg is printed as SKIP", "selftest-sweep" in out + and "SKIP" in out, True) + check("6c the skipped stub never ran", os.path.exists(marker), False) + + # 7. --json is a real machine-readable record + repo = _fixture(os.path.join(td, "c7"), THREE) + jpath = os.path.join(td, "c7.json") + rc, out = _verify("--repo", repo, "--json", jpath) + obj = json.load(open(jpath)) if os.path.exists(jpath) else {} + check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])], + ["index-current", "append-audit", "selftest-sweep"]) + check("7b --json carries the same rc the process returned", + obj.get("rc"), rc) + + # 8. a repo that is not there is refused before any leg runs + rc, out = _verify("--repo", os.path.join(td, "no-such-repo")) + check("8 missing repo -> rc 2", rc, 2) + check("8b the refusal names the path", "cannot read repo:" in out, True) + + # 9. rc 3 is retired: it is ungradable, never a fault and never a pass + repo = _fixture(os.path.join(td, "c9"), + dict(THREE, **{"queue_index.py": 3}), + outs={"queue_index.py": "unknown state\n"}) + rc, out = _verify("--repo", repo) + check("9 rc 3 -> rc 2 (retired code)", rc, 2) + check("9b the unexpected code is named", "unexpected rc 3" in out, True) + + # 10. a hung leg is a timeout, not a pass and not a fault + repo = _fixture(os.path.join(td, "c10"), + dict(THREE, **{"selftest_sweep.py": 0}), sleep=5.0) + rc, out = _verify("--repo", repo, "--timeout", "1") + check("10 timeout -> rc 2", rc, 2) + check("10b the timeout is named", "TIMEOUT after 1s" in out, True) + + # 11. order is fixed, cheapest first: the table is not re-sorted by outcome + repo = _fixture(os.path.join(td, "c11"), + dict(THREE, **{"queue_index.py": 1}), + outs={"queue_index.py": "STALE: line 1\n"}) + rc, out = _verify("--repo", repo) + order = [out.find(k) for k in ("index-current", "append-audit", "selftest-sweep")] + check("11 legs print in fixed order", order == sorted(order) + and -1 not in order, True) + + # 12. a skip is counted as a skip, never folded into pass + repo = _fixture(os.path.join(td, "c12"), THREE) + jpath = os.path.join(td, "c12.json") + rc, out = _verify("--repo", repo, "--fast", "--json", jpath) + obj = json.load(open(jpath)) if os.path.exists(jpath) else {} + check("12 skipped leg is its own count in --json", + obj.get("counts", {}).get("SKIP"), 1) + check("12b and the skipped leg is not counted as a pass", + obj.get("counts", {}).get(PASS), 2) + + print("selftest: %d cells, %d passed, %d failed" % (cells, passed, len(failures))) + for f in failures: + print(" FAIL %s" % f) + return 0 if passed == cells else 1 + + +if __name__ == "__main__": + sys.exit(main())