spark
8ec77256b6
Lane gate measured at 77d5eba: q65_gate.sh rc 0 in 11.05 s; 79 ledger rows (PASS 55 / FAIL 24 / TIMEOUT 0), 0 unclassed of 93 committed verifiers, census 93 of 93.
Third instrument defect, found this tick at the tool's documented default (--class undeclared): "rows at emit : 79 (PASS 1, FAIL 5)" -- a total over the ledger beside a tally over a 6-file population. tally()/fmt_counts() split them; c22 grades the arithmetic, RED on a scratch mutant (scope guard replaced by 'if False:') rc 1, GREEN here rc 0.
833 lines
40 KiB
Python
833 lines
40 KiB
Python
#!/usr/bin/env python3
|
|
"""q65_runner.py -- the gate Q65 asked for: execute every committed verifier that
|
|
nothing else invokes, grade its exit code, and keep the grade in a ledger (Q65,
|
|
volt lane, 2026-10-24).
|
|
|
|
WHY THIS EXISTS
|
|
Q63 measured the class and Q65 holds the residual: at `6c26bc5`, 72 of the 87
|
|
committed `wizard-work/**/verify_*.py` were invoked by nothing. `unittest
|
|
discover` matches `test_*`, no `CHECK` globs `verify_*.py`, and no `tools/` leg
|
|
runs them. A verifier in that state is not a broken check, it is an unread
|
|
claim: its verdict is whatever its author last saw by hand, and if the whole
|
|
population were deleted the archive would not notice. Q65's done-when is one
|
|
recorded outcome per file -- invoked-and-graded, retired, or declared one-off.
|
|
The measure was a grep for an invocation-shaped line. This is the other half:
|
|
a gate that actually RUNS them, so the invocation is an execution rather than
|
|
a mention, and the outcome is an rc rather than a reading.
|
|
|
|
The alternative was 72 in-file declarations, one per file, each naming a grep
|
|
that shows the file is unwired. That records the defect 72 times; it does not
|
|
remove it. One runner grades the population, and a file added tomorrow is
|
|
graded by the next run without anyone remembering to add it.
|
|
|
|
WHAT IT MEASURES -- one row per population file, one verdict per row
|
|
PASS rc 0: the file ran and its checks held.
|
|
FAIL rc 1: the file ran and its checks did NOT hold. A finding, not a
|
|
bookkeeping error -- the file's claim is false at this tree.
|
|
REFUSED rc 2: the file declined to grade (its input is gone, its subject
|
|
is absent). Not a pass. The house contract will not let "could
|
|
not grade" read as "graded clean".
|
|
OTHER_RC rc 3 or above: a code the contract does not define.
|
|
TIMEOUT did not finish inside --timeout seconds.
|
|
ERROR the interpreter could not run the file at all (syntax error, an
|
|
exec failure): there is no rc to grade.
|
|
MISSING the path is in the ledger but not in the tree at --rev.
|
|
|
|
A row is the whole point. `--check` re-derives the population from `--rev`,
|
|
reads the ledger, and grades the bookkeeping: every population file has
|
|
exactly one row, every row is still in the tree, and every row's verdict is
|
|
consistent with the rc recorded beside it. rc 0 clean, rc 1 a fault, rc 2
|
|
could not grade -- the house three codes.
|
|
|
|
`--check` does NOT re-execute (that is `--emit`): a gate that takes minutes
|
|
is a gate that gets skipped. `--check --strict` makes any FAIL / REFUSED /
|
|
OTHER_RC / TIMEOUT / ERROR row fatal, for an operator who wants the runner to
|
|
be a red gate on the verifiers themselves rather than on the ledger.
|
|
|
|
THREE THINGS THIS LEDGER USED TO GET WRONG, AND THE COLUMN EACH ONE TOOK
|
|
1. ONE GLOBAL WINDOW. `--timeout` was a single figure for the whole sweep, so
|
|
a verifier whose own work takes minutes (recursive `tools/` runs, live HTTP)
|
|
came back TIMEOUT at whatever the sweep happened to pass -- 45s for one row,
|
|
600s for another -- and a TIMEOUT row is a row with no rc, which is not a
|
|
grade. Different files cost different amounts of time; that is a property of
|
|
the file, not of the caller. `--windows <tsv>` names one window per path
|
|
(path, seconds, why) and `run_one` uses it; `--timeout` stays the default
|
|
for paths the table does not name. A TIMEOUT row's note names the window it
|
|
actually got, so the table cannot silently disagree with the ledger.
|
|
2. A ROW WAS NOT BOUND TO CONTENT. Rows carried path, verdict, rc, seconds and
|
|
note, and `--check` compared path SETS. A verifier edited to `raise
|
|
SystemExit(1)` kept its `PASS 0` row and the gate returned rc 0: the ledger
|
|
graded a blob it no longer described. Column 7 is the git blob id the rc was
|
|
measured against (`git rev-parse <rev>:<path>`); `--check` recomputes it at
|
|
its own `--rev` and a mismatch is a DRIFT fault. Editing a verifier is now a
|
|
request for `--emit`, which is the only thing that can produce an rc.
|
|
3. A PASS COULD BE A PASS ON SOMEBODY ELSE'S TREE. An exit code is not a claim
|
|
about the commit: a verifier that reads `/root/troupe-standup` returns rc 0
|
|
from any checkout, because it graded this host's copy rather than the tree
|
|
that invoked it. Column 8 is a static host-pin probe -- the committed blob is
|
|
parsed as an AST and every string constant that is not a docstring is tested
|
|
for a host-absolute root (`/root/troupe-standup`, `/root/.hermes`). Rows
|
|
carrying one print as `PASS*` and are excluded from the `graded-clean`
|
|
figure, so a host-pinned pass is still a row, still a verdict, and no longer
|
|
counted as coverage of the tree.
|
|
|
|
WHY THE POPULATION IS A GLOB AND NOT A LIST OF NAMES
|
|
`q63_verify_scan.py` classes a file as wired when a committed line invokes it
|
|
by name. A globbed runner names no file, so the scan cannot see it -- and the
|
|
scan is right to be suspicious, because "a gate runs it" is exactly the claim
|
|
that is easy to make and hard to check. The evidence here is stronger than a
|
|
grep: the ledger rows are produced by execution, one per path discovered at
|
|
`--rev`, and `--check` fails if the two sets differ by a single path. A file
|
|
that the runner silently skipped would be a MISSING row, not an invisible
|
|
one. `q63_verify_scan.py` therefore accepts class `gate-run` for a path this
|
|
ledger carries a graded row for, and re-emits 0 `undeclared` rows.
|
|
|
|
WHAT IT DOES NOT DO
|
|
It does not sandbox. A verifier runs in its own directory, with bytecode
|
|
writing off, and the run prints any tracked file it dirtied (`git status`
|
|
before and after) -- but a verifier that wants to write will write. It does
|
|
not run two files at once: ordering is fixed and a shared temp name in one
|
|
file cannot make another look flaky. It does not re-run a flaky row; a row is
|
|
what one run said, and `--emit` again is how a second opinion is taken.
|
|
|
|
USAGE
|
|
python3 wizard-work/volt/q65_runner.py --emit # measure -> ledger
|
|
python3 wizard-work/volt/q65_runner.py --check # grade ledger, fast
|
|
python3 wizard-work/volt/q65_runner.py --why # per-row detail
|
|
python3 wizard-work/volt/q65_runner.py --class all --emit --timeout 120
|
|
"""
|
|
import argparse
|
|
import ast
|
|
import json
|
|
import os
|
|
import re
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
import time
|
|
|
|
HERE = os.path.dirname(os.path.abspath(__file__))
|
|
DEFAULT_REPO = os.path.abspath(os.path.join(HERE, "..", ".."))
|
|
DEFAULT_LEDGER = os.path.join(HERE, "q65_runner_baseline.tsv")
|
|
DEFAULT_Q63 = os.path.join(HERE, "q63_unwired_baseline.tsv")
|
|
DEFAULT_WINDOWS = os.path.join(HERE, "q65_row_windows.tsv")
|
|
VERIFIER_RE = re.compile(r"^verify_[^/]*\.py$")
|
|
# a host-absolute root in a string constant is what Q62/Q65-95 mean by
|
|
# "host-pinned": the verifier grades this machine's copy from any checkout.
|
|
HOST_ROOTS = ("/root/troupe-standup", "/root/.hermes")
|
|
|
|
PASS, FAIL, REFUSED, OTHER_RC, TIMEOUT, ERROR, MISSING = (
|
|
"PASS", "FAIL", "REFUSED", "OTHER_RC", "TIMEOUT", "ERROR", "MISSING")
|
|
VERDICTS = (PASS, FAIL, REFUSED, OTHER_RC, TIMEOUT, ERROR, MISSING)
|
|
# verdict <-> rc, the only pairs a ledger row may state
|
|
RC_OF = {PASS: "0", FAIL: "1", REFUSED: "2"}
|
|
VERDICT_OF_RC0 = PASS
|
|
FATAL_STRICT = (FAIL, REFUSED, OTHER_RC, TIMEOUT, ERROR, MISSING)
|
|
|
|
|
|
def lane_of(path):
|
|
"""`wizard-work/<lane>/...` -> lane; anything else -> '?'."""
|
|
parts = path.split("/")
|
|
return parts[1] if len(parts) > 2 and parts[0] == "wizard-work" else "?"
|
|
|
|
|
|
def git(repo, *args):
|
|
return subprocess.run(["git"] + list(args), cwd=repo, capture_output=True,
|
|
text=True)
|
|
|
|
|
|
def rev_files(repo, rev):
|
|
"""Every tracked path at `rev` whose basename is verify_*.py."""
|
|
r = git(repo, "ls-tree", "-r", "--name-only", rev)
|
|
if r.returncode != 0:
|
|
raise SystemExit("q65_runner: cannot read tree at %s: %s"
|
|
% (rev, (r.stderr or "").strip()[:200]))
|
|
return sorted(p for p in r.stdout.splitlines() if VERIFIER_RE.match(
|
|
os.path.basename(p)))
|
|
|
|
|
|
def read_classes(path):
|
|
"""path -> class, from a <class>\\t<path> ledger with # comments."""
|
|
out = {}
|
|
with open(path, encoding="utf-8") as fh:
|
|
for line in fh:
|
|
line = line.rstrip("\n")
|
|
if not line or line.startswith("#"):
|
|
continue
|
|
parts = line.split("\t")
|
|
if len(parts) < 2:
|
|
raise SystemExit("q65_runner: malformed row in %s: %r" % (path, line))
|
|
out[parts[1]] = parts[0]
|
|
return out
|
|
|
|
|
|
def read_windows(path):
|
|
"""path -> (seconds, why), from a <path>\\t<seconds>\\t<why> table.
|
|
|
|
Absent or malformed is NOT fatal: a window is a measurement budget, and a
|
|
missing budget falls back to --timeout. What it must never be is silent, so
|
|
a malformed row raises rather than being skipped (a skipped row would look
|
|
like a file with no window, which is indistinguishable from one that ran).
|
|
"""
|
|
out = {}
|
|
if not path or not os.path.exists(path):
|
|
return out
|
|
with open(path, encoding="utf-8") as fh:
|
|
for line in fh:
|
|
line = line.rstrip("\n")
|
|
if not line or line.startswith("#"):
|
|
continue
|
|
parts = line.split("\t")
|
|
if len(parts) < 2 or not parts[1].strip().replace(".", "", 1).isdigit():
|
|
raise SystemExit("q65_runner: malformed window row in %s: %r"
|
|
% (path, line))
|
|
out[parts[0]] = (float(parts[1]), parts[2] if len(parts) > 2 else "")
|
|
return out
|
|
|
|
|
|
def blob_of(repo, rev, path):
|
|
"""git blob id for `path` at `rev`, or '' when it is not in that tree."""
|
|
r = git(repo, "rev-parse", "--verify", "--quiet", "%s:%s" % (rev, path))
|
|
return (r.stdout or "").strip() if r.returncode == 0 else ""
|
|
|
|
|
|
def host_pin(repo, rev, path):
|
|
"""'' or 'L<n>:<literal>' -- a host-absolute root in a NON-docstring string.
|
|
|
|
Why an AST and not a grep: every verifier's module docstring names the
|
|
command that runs it, and most of those commands are written as
|
|
`python3 /root/troupe-standup/...`. A grep flags that prose and drowns the
|
|
signal. `ast` can tell a docstring from a value, so the probe answers the
|
|
question Q62 asks -- does this file READ an absolute host root -- and not
|
|
the weaker one -- does this file MENTION one.
|
|
"""
|
|
r = git(repo, "show", "%s:%s" % (rev, path))
|
|
if r.returncode != 0:
|
|
return ""
|
|
try:
|
|
tree = ast.parse(r.stdout)
|
|
except SyntaxError:
|
|
return ""
|
|
docstrings = set()
|
|
for node in ast.walk(tree):
|
|
if isinstance(node, (ast.Module, ast.FunctionDef, ast.AsyncFunctionDef,
|
|
ast.ClassDef)):
|
|
body = getattr(node, "body", [])
|
|
if (body and isinstance(body[0], ast.Expr)
|
|
and isinstance(body[0].value, ast.Constant)
|
|
and isinstance(body[0].value.value, str)):
|
|
docstrings.add(id(body[0].value))
|
|
for node in ast.walk(tree):
|
|
if not isinstance(node, ast.Constant) or id(node) in docstrings:
|
|
continue
|
|
val = node.value
|
|
if not isinstance(val, str):
|
|
continue
|
|
for root in HOST_ROOTS:
|
|
if root in val:
|
|
return "L%d:%s" % (getattr(node, "lineno", 0), root)
|
|
return ""
|
|
|
|
|
|
def population(repo, rev, classes_path, want):
|
|
"""The files --check holds to a ledger row: census classes, or `all`.
|
|
|
|
`--class` takes one class or a comma list, because the population a gate
|
|
needs is not always one class: `gate-run,undeclared` is "every committed
|
|
verifier the census says the runner graded, plus every one it says nothing
|
|
does" -- the exact set that must carry a row, without dragging in the
|
|
`wired-invoke` files (graded by their own gate) or the `declared-one-off`
|
|
ones (which state in-file that no gate invokes them).
|
|
"""
|
|
tracked = set(rev_files(repo, rev))
|
|
if want == "all":
|
|
return {p: "ungated" for p in tracked}
|
|
cls = read_classes(classes_path)
|
|
want = [want] if isinstance(want, str) else list(want)
|
|
want = [w.strip() for w in ",".join(want).split(",") if w.strip()]
|
|
return {p: c for p, c in cls.items() if c in want and p in tracked}
|
|
|
|
|
|
def unclassed(repo, rev, classes_path, ledger_paths):
|
|
"""Committed verifiers that NEITHER ledger names -- the coverage rule.
|
|
|
|
The population is whatever the class ledger lists, so a committed verifier
|
|
absent from that ledger is invisible, and `--check` used to print
|
|
`0 fault(s)` over it: at `b01ec0e` that was 8 of 88 committed verifiers
|
|
(6 genuinely unwired, 2 invoked from a committed gate). The ledger is the
|
|
side of the comparison that can omit itself; the tree at `--rev` cannot.
|
|
Measured here so the omission is a fault rather than a silence.
|
|
"""
|
|
cls = read_classes(classes_path)
|
|
return sorted(p for p in rev_files(repo, rev)
|
|
if p not in cls and p not in ledger_paths)
|
|
|
|
|
|
def run_one(repo, path, timeout, cwd_mode, windows=None):
|
|
"""Execute one verifier. Returns a row dict; never raises."""
|
|
windows = windows or {}
|
|
window, why = windows.get(path, (timeout, ""))
|
|
full = os.path.join(repo, path)
|
|
if not os.path.exists(full):
|
|
return {"path": path, "verdict": MISSING, "rc": None, "seconds": 0.0,
|
|
"note": "not in worktree"}
|
|
cwd = repo if cwd_mode == "repo" else os.path.dirname(full)
|
|
target = os.path.relpath(full, cwd)
|
|
env = dict(os.environ, PYTHONDONTWRITEBYTECODE="1", PYTHONUNBUFFERED="1")
|
|
env.pop("PYTHONOPTIMIZE", None)
|
|
t0 = time.time()
|
|
try:
|
|
r = subprocess.run([sys.executable, "-B", target],
|
|
cwd=cwd, env=env, capture_output=True, timeout=window,
|
|
text=True, errors="replace")
|
|
except subprocess.TimeoutExpired:
|
|
return {"path": path, "verdict": TIMEOUT, "rc": None,
|
|
"seconds": round(time.time() - t0, 2),
|
|
"note": "no finish in %ss%s" % (window, (" (row window: %s)" % why)
|
|
if why else "")}
|
|
secs = round(time.time() - t0, 2)
|
|
rc = r.returncode
|
|
if rc == 0:
|
|
verdict = PASS
|
|
elif rc == 1:
|
|
verdict = FAIL
|
|
elif rc == 2:
|
|
verdict = REFUSED
|
|
else:
|
|
verdict = OTHER_RC if rc > 2 else ERROR
|
|
note = ""
|
|
tail = (r.stderr or "").strip().splitlines()
|
|
if verdict != PASS and tail:
|
|
note = tail[-1][:160]
|
|
return {"path": path, "verdict": verdict, "rc": rc, "seconds": secs,
|
|
"note": note}
|
|
|
|
|
|
def emit(rows, path, repo, rev, timeout):
|
|
# The rev is PINNED to a sha, not written as the word `HEAD`: a ledger whose
|
|
# law is "the rc at a named rev" cannot name a moving ref, because HEAD one
|
|
# commit later is a different population and the same rows read as current.
|
|
sha = (git(repo, "rev-parse", "--short", rev).stdout or rev).strip() or rev
|
|
lines = [
|
|
"# Q65 runner ledger -- one row per committed verify_*.py the runner",
|
|
"# executes, with the exit code it actually returned at the stated rev.",
|
|
"# row: <class>\\t<path>\\t<verdict>\\t<rc>\\t<seconds>\\t<note>\\t<blob>\\t<host>",
|
|
"# verdicts: PASS rc0 / FAIL rc1 / REFUSED rc2 / OTHER_RC >2 /",
|
|
"# TIMEOUT, ERROR, MISSING (no rc to grade).",
|
|
"# class is the Q63 ledger class the file came from at emit time.",
|
|
"# A FAIL row is a finding held open, not a tolerated pass: --check",
|
|
"# grades the bookkeeping, --check --strict makes any non-PASS fatal.",
|
|
"# timeout is PER ROW: a TIMEOUT row names its own window in the note, so",
|
|
"# a --merge cannot leave a stale single figure standing for rows run",
|
|
"# under a different one (the defect this line replaces). --windows",
|
|
"# names one window per path; --timeout is the fallback for unlisted.",
|
|
"# blob is the git blob id this rc was measured against: --check recomputes",
|
|
"# it at its own --rev and a mismatch is DRIFT, so an edited verifier",
|
|
"# cannot keep a PASS it no longer earns.",
|
|
"# host is a static host-pin probe (L<line>:<root>), empty when clean: a",
|
|
"# non-docstring string constant naming an absolute host root. Such a row",
|
|
"# is printed PASS* and excluded from the graded-clean figure.",
|
|
"# Regenerate: q65_runner.py --emit",
|
|
"# rev: %s rows: %d emitted: q65_runner.py" % (sha, len(rows)),
|
|
]
|
|
for r in sorted(rows, key=lambda x: x["path"]):
|
|
blob = r.get("blob") or blob_of(repo, rev, r["path"])
|
|
host = r.get("host")
|
|
if not host:
|
|
# An empty column is "not filled in" as well as "clean": the probe
|
|
# is cheap and deterministic, so it is re-run rather than trusted.
|
|
host = host_pin(repo, rev, r["path"])
|
|
lines.append("\t".join([r.get("class", "ungated"), r["path"], r["verdict"],
|
|
"" if r["rc"] is None else str(r["rc"]),
|
|
"%.2f" % (float(r["seconds"]) if r.get("seconds") else 0.0),
|
|
(r.get("note") or "").replace("\t", " "), blob, host]))
|
|
tmp = path + ".tmp"
|
|
with open(tmp, "w", encoding="utf-8") as fh:
|
|
fh.write("\n".join(lines) + "\n")
|
|
os.replace(tmp, path)
|
|
return path
|
|
|
|
|
|
def read_ledger(path):
|
|
"""(rows, error, meta). rows keep every column; error is a string for rc 2.
|
|
|
|
meta carries the ledger's own `# rev:` line, so `--check` can say which tree
|
|
the rows claim and grade the blobs against it rather than against nothing.
|
|
"""
|
|
if not os.path.exists(path):
|
|
return [], "no ledger at %s (run --emit first)" % path, {}
|
|
rows = []
|
|
meta = {}
|
|
with open(path, encoding="utf-8") as fh:
|
|
for n, line in enumerate(fh, 1):
|
|
line = line.rstrip("\n")
|
|
if not line or line.startswith("#"):
|
|
m = re.match(r"#\s*rev:\s*(\S+)", line)
|
|
if m:
|
|
meta["rev"] = m.group(1)
|
|
continue
|
|
parts = line.split("\t")
|
|
if len(parts) < 4:
|
|
return [], "malformed row %d in %s: %r" % (n, path, line), {}
|
|
if parts[2] not in VERDICTS:
|
|
return [], "unknown verdict %r at row %d in %s" % (parts[2], n, path), {}
|
|
rows.append({"class": parts[0], "path": parts[1], "verdict": parts[2],
|
|
"rc": parts[3] if parts[3] else None,
|
|
"seconds": parts[4] if len(parts) > 4 else "",
|
|
"note": parts[5] if len(parts) > 5 else "",
|
|
"blob": parts[6] if len(parts) > 6 else "",
|
|
"host": parts[7] if len(parts) > 7 else ""})
|
|
return rows, None, meta
|
|
|
|
|
|
def rc_consistent(row):
|
|
"""The verdict must match the rc recorded beside it. Returns a complaint."""
|
|
v, rc = row["verdict"], row["rc"]
|
|
if v in RC_OF:
|
|
if rc != RC_OF[v]:
|
|
return "verdict %s with rc %s (expected %s)" % (v, rc, RC_OF[v])
|
|
return None
|
|
if v == OTHER_RC:
|
|
if rc is None or not rc.isdigit() or int(rc) <= 2:
|
|
return "OTHER_RC with rc %r" % rc
|
|
return None
|
|
if rc is not None:
|
|
return "%s carries rc %s (a code to grade)" % (v, rc)
|
|
return None
|
|
|
|
|
|
def row_is_clean(row, pop):
|
|
"""Is this row a PASS on a host-independent verifier, for a graded path?
|
|
|
|
WHY THIS IS A FUNCTION, AND WHY IT COMPARES WITH `==`. `read_ledger` builds
|
|
each verdict by splitting a line, so the string it returns is a NEW object
|
|
and not the module constant: `r["verdict"] is PASS` was False for every row
|
|
read from disk. Measured at `d45a5ed` on 2026-10-25, before this line
|
|
existed: `--check` printed `graded-clean: 0 of 79` over a ledger holding 7
|
|
clean PASS rows, printed no `host-pinned` block over 48 host-pinned PASS
|
|
rows, and `--why` printed all 79 rows as findings. Identity is the right
|
|
question for a constant a function just returned and the wrong one for a
|
|
value parsed out of a file; the selftest cell c20 holds that distinction.
|
|
"""
|
|
return (row["path"] in pop and row["verdict"] == PASS and not row["host"]
|
|
and rc_consistent(row) is None)
|
|
|
|
|
|
def tally(rows, pop=None):
|
|
"""Verdict counts over `rows`, or only over the rows in `pop`.
|
|
|
|
WHY THIS IS A FUNCTION. `--check` printed one figure's total beside
|
|
another figure's tally: at 2026-10-25 02:18Z, with the default
|
|
`--class undeclared`, it printed `rows at emit : 79 (PASS 1, FAIL 5)` --
|
|
79 ledger rows under a tally of 6 population rows, and 1 + 5 is not 79.
|
|
Every number was right; the sentence was not. Splitting the count from the
|
|
print lets selftest cell c22 grade the arithmetic instead of the prose.
|
|
"""
|
|
out = {}
|
|
for r in rows:
|
|
if pop is not None and r["path"] not in pop:
|
|
continue
|
|
out[r["verdict"]] = out.get(r["verdict"], 0) + 1
|
|
return out
|
|
|
|
|
|
def fmt_counts(counts):
|
|
return ", ".join("%s %d" % (k, counts[k]) for k in VERDICTS if counts.get(k))
|
|
|
|
|
|
def check(repo, rev, classes_path, want, ledger_path, strict):
|
|
"""Grade the ledger against the tree. rc 0 clean / 1 fault / 2 could not grade."""
|
|
try:
|
|
pop = population(repo, rev, classes_path, want)
|
|
except SystemExit as exc:
|
|
print("q65_runner: %s" % exc)
|
|
return 2
|
|
rows, err, meta = read_ledger(ledger_path)
|
|
if err:
|
|
print("q65_runner: %s" % err)
|
|
return 2
|
|
faults = []
|
|
seen = {}
|
|
for r in rows:
|
|
seen[r["path"]] = seen.get(r["path"], 0) + 1
|
|
why = rc_consistent(r)
|
|
if why:
|
|
faults.append("INCONSISTENT %s: %s" % (r["path"], why))
|
|
for p, n in sorted(seen.items()):
|
|
if n > 1:
|
|
faults.append("DUPLICATE %s: %d rows" % (p, n))
|
|
# Content binding: a row's rc is evidence about one blob and no other. This
|
|
# is the defect reply 93 named -- a verifier edited to `raise SystemExit(1)`
|
|
# kept its PASS row and `--check` returned 0 fault(s).
|
|
drift = 0
|
|
for r in rows:
|
|
if r["path"] not in pop:
|
|
continue
|
|
want_blob = r["blob"]
|
|
now_blob = blob_of(repo, rev, r["path"])
|
|
if not want_blob:
|
|
drift += 1
|
|
faults.append("UNBOUND %s: no blob column -- the row does not say "
|
|
"which content its rc describes (re-emit)" % r["path"])
|
|
elif now_blob and now_blob != want_blob:
|
|
drift += 1
|
|
faults.append("DRIFT %s: rc measured against blobs %s, tree at %s has "
|
|
"%s" % (r["path"], want_blob[:10], rev, now_blob[:10]))
|
|
# Host-pin: a PASS on a verifier that reads an absolute host root graded THIS
|
|
# host, not the tree that invoked it (Q62's class, reply 95's row).
|
|
pinned = 0
|
|
for r in rows:
|
|
if r["verdict"] == PASS and r["host"] and r["path"] in pop:
|
|
pinned += 1
|
|
audit = [] if want == "all" else unclassed(repo, rev, classes_path, set(seen))
|
|
for p in audit:
|
|
faults.append("UNCLASSED %s: committed verify_*.py at %s, in no class row "
|
|
"in %s and no ledger row"
|
|
% (p, rev, os.path.basename(classes_path)))
|
|
ungraded = sorted(set(pop) - set(seen))
|
|
stale = sorted(set(seen) - set(pop))
|
|
for p in ungraded:
|
|
faults.append("UNGRADED %s: in the population at %s, no ledger row" % (p, rev))
|
|
for p in stale:
|
|
if not os.path.exists(os.path.join(repo, p)):
|
|
faults.append("STALE %s: ledger row, file not in the tree" % p)
|
|
elif strict:
|
|
faults.append("OUTSIDE %s: ledger row, not in the --class population" % p)
|
|
counts = tally(rows, pop)
|
|
print("q65_runner -- rev %s, %d population file(s), %d ledger row(s)%s"
|
|
% (rev, len(pop), len(rows),
|
|
(" [ledger rev %s]" % meta["rev"]) if meta.get("rev") else ""))
|
|
graded = sum(1 for r in rows if r["path"] in pop and r["rc"] is not None
|
|
and rc_consistent(r) is None)
|
|
clean = sum(1 for r in rows if row_is_clean(r, pop))
|
|
print(" rows at emit : %d (%s)" % (len(rows), fmt_counts(tally(rows))))
|
|
if any(r["path"] not in pop for r in rows):
|
|
print(" in population : %d of %d (%s)"
|
|
% (len(pop), len(rows), fmt_counts(counts)))
|
|
print(" graded (has rc): %d of %d" % (graded, len(pop)))
|
|
print(" graded-clean : %d of %d (PASS on a host-independent verifier)"
|
|
% (clean, len(pop)))
|
|
if pinned:
|
|
print(" host-pinned : %d PASS row(s) excluded as PASS* -- they read an "
|
|
"absolute host root" % pinned)
|
|
for r in rows:
|
|
if r["verdict"] == PASS and r["host"] and r["path"] in pop:
|
|
print(" PASS* %s %s" % (r["path"], r["host"]))
|
|
if drift:
|
|
print(" content-bound : %d row(s) not bound to the tree at %s (DRIFT/UNBOUND)"
|
|
% (drift, rev))
|
|
print(" tree audit : %d unclassed of %d committed verifier(s)%s"
|
|
% (len(audit), len(pop) if want == "all" else len(rev_files(repo, rev)),
|
|
"" if not audit else " <-- named by neither ledger"))
|
|
if strict:
|
|
for r in rows:
|
|
if r["verdict"] in FATAL_STRICT:
|
|
faults.append("%s %s rc %s %s" % (r["verdict"], r["path"], r["rc"],
|
|
r["note"][:80]))
|
|
for f in faults:
|
|
print(" " + f)
|
|
print("q65: %d fault(s)" % len(faults))
|
|
return 1 if faults else 0
|
|
|
|
|
|
def selftest():
|
|
"""Small cells for the two rules that can silently pass: the verdict/rc
|
|
pairing and the population coverage."""
|
|
cells = []
|
|
|
|
def cell(n, ok):
|
|
cells.append((n, bool(ok)))
|
|
|
|
good = {"path": "a/b.py", "verdict": PASS, "rc": "0", "note": ""}
|
|
cell("c1 PASS rc 0 is consistent", rc_consistent(good) is None)
|
|
cell("c2 PASS rc 1 is a fault",
|
|
"rc 1" in (rc_consistent({"path": "a/b.py", "verdict": PASS, "rc": "1",
|
|
"note": ""}) or ""))
|
|
cell("c3 TIMEOUT carrying an rc is a fault",
|
|
rc_consistent({"path": "x", "verdict": TIMEOUT, "rc": "1", "note": ""}) is not None)
|
|
cell("c4 OTHER_RC rc 3 is consistent",
|
|
rc_consistent({"path": "x", "verdict": OTHER_RC, "rc": "3", "note": ""}) is None)
|
|
cell("c5 OTHER_RC rc 7 is consistent",
|
|
rc_consistent({"path": "x", "verdict": OTHER_RC, "rc": "7", "note": ""}) is None)
|
|
cell("c6 OTHER_RC rc 2 is a fault (that is REFUSED)",
|
|
rc_consistent({"path": "x", "verdict": OTHER_RC, "rc": "2", "note": ""}) is not None)
|
|
tmp = tempfile.mkdtemp(prefix="q65-selftest-")
|
|
led = os.path.join(tmp, "l.tsv")
|
|
with open(led, "w", encoding="utf-8") as fh:
|
|
fh.write("# c\n")
|
|
fh.write("ungated\twizard-work/x/verify_a.py\tPASS\t0\t0.01\t\n")
|
|
fh.write("ungated\twizard-work/x/verify_b.py\tFAIL\t1\t0.02\tboom\n")
|
|
rows, err, meta = read_ledger(led)
|
|
cell("c7 ledger round-trips two rows", err is None and len(rows) == 2)
|
|
cell("c8 a missing ledger is an error, not an empty pass",
|
|
read_ledger(os.path.join(tmp, "nope.tsv"))[1] is not None)
|
|
bad = os.path.join(tmp, "bad.tsv")
|
|
with open(bad, "w", encoding="utf-8") as fh:
|
|
fh.write("ungated\twizard-work/x/verify_a.py\tPASSED\t0\t0.01\t\n")
|
|
cell("c9 an unknown verdict refuses the ledger",
|
|
read_ledger(bad)[1] is not None)
|
|
# c10/c11: the coverage rule. A committed verifier named by neither ledger is
|
|
# a fault, even though the ledger that would grade it is the one it is
|
|
# missing from -- that is how 8 of 88 stood invisible at `b01ec0e`.
|
|
def mk_tree(prefix):
|
|
t = tempfile.mkdtemp(prefix=prefix)
|
|
e = dict(os.environ, GIT_AUTHOR_NAME="t", GIT_AUTHOR_EMAIL="t@t",
|
|
GIT_COMMITTER_NAME="t", GIT_COMMITTER_EMAIL="t@t")
|
|
subprocess.run(["git", "init", "-q", t], check=True, env=e)
|
|
return t, e
|
|
|
|
tree, env = mk_tree("q65-tree-")
|
|
lane = os.path.join(tree, "wizard-work", "x")
|
|
os.makedirs(lane)
|
|
with open(os.path.join(lane, "verify_orphan.py"), "w", encoding="utf-8") as fh:
|
|
fh.write(chr(10).join(("import sys", "sys.exit(0)")) + chr(10))
|
|
subprocess.run(["git", "-C", tree, "add", "-A"], check=True, env=env)
|
|
subprocess.run(["git", "-C", tree, "commit", "-qm", "c"], check=True, env=env)
|
|
census = os.path.join(tree, "census.tsv")
|
|
open(census, "w", encoding="utf-8").close()
|
|
cell("c10 a committed verifier in neither ledger is UNCLASSED",
|
|
unclassed(tree, "HEAD", census, set()) == ["wizard-work/x/verify_orphan.py"])
|
|
with open(census, "w", encoding="utf-8") as fh:
|
|
fh.write(chr(9).join(("undeclared", "wizard-work/x/verify_orphan.py",
|
|
"no run")) + chr(10))
|
|
cell("c11 a class row covers the same file",
|
|
unclassed(tree, "HEAD", census, set()) == [])
|
|
|
|
# c12-c17: the three rules added after reply 93/95, each with a cell that
|
|
# fails if the rule is removed.
|
|
win = os.path.join(tmp, "win.tsv")
|
|
with open(win, "w", encoding="utf-8") as fh:
|
|
fh.write("# path\tseconds\twhy\n")
|
|
fh.write("wizard-work/x/verify_slow.py\t240\trecursive tools/ run\n")
|
|
w = read_windows(win)
|
|
cell("c12 a window row parses to (seconds, why)",
|
|
w.get("wizard-work/x/verify_slow.py", (0, ""))[0] == 240.0)
|
|
cell("c13 an unlisted path falls back to --timeout",
|
|
"wizard-work/x/verify_other.py" not in w
|
|
and run_one(tree, "wizard-work/x/verify_orphan.py", 7, "dir",
|
|
windows=w)["verdict"] is PASS)
|
|
with open(win, "w", encoding="utf-8") as fh:
|
|
fh.write("wizard-work/x/verify_slow.py\tnot-a-number\t\n")
|
|
try:
|
|
read_windows(win)
|
|
bad_window = False
|
|
except SystemExit:
|
|
bad_window = True
|
|
cell("c14 a malformed window row refuses instead of silently skipping", bad_window)
|
|
|
|
# blob binding: the rc is evidence about one blob
|
|
blob = blob_of(tree, "HEAD", "wizard-work/x/verify_orphan.py")
|
|
cell("c15 a committed path has a blob id", len(blob) == 40)
|
|
cell("c16 an absent path has no blob id",
|
|
blob_of(tree, "HEAD", "wizard-work/x/verify_gone.py") == "")
|
|
with open(os.path.join(lane, "verify_orphan.py"), "w", encoding="utf-8") as fh:
|
|
fh.write(chr(10).join(("import sys", "sys.exit(1)")) + chr(10))
|
|
subprocess.run(["git", "-C", tree, "commit", "-qam", "c2"], check=True, env=env)
|
|
cell("c17 the same path at a new commit is a different blob (DRIFT is detectable)",
|
|
blob_of(tree, "HEAD", "wizard-work/x/verify_orphan.py") not in ("", blob))
|
|
|
|
# host-pin probe: a value, not prose
|
|
pinned = os.path.join(lane, "verify_pinned.py")
|
|
with open(pinned, "w", encoding="utf-8") as fh:
|
|
fh.write(chr(10).join((
|
|
'"""usage: python3 /root/troupe-standup/wizard-work/x/verify_pinned.py"""',
|
|
'import os',
|
|
'REPO = "/root/troupe-standup"',
|
|
'print(REPO)')) + chr(10))
|
|
prose = os.path.join(lane, "verify_prose.py")
|
|
with open(prose, "w", encoding="utf-8") as fh:
|
|
fh.write(chr(10).join((
|
|
'"""Run as: python3 /root/troupe-standup/wizard-work/x/verify_prose.py."""',
|
|
'print("ok")')) + chr(10))
|
|
subprocess.run(["git", "-C", tree, "add", "-A"], check=True, env=env)
|
|
subprocess.run(["git", "-C", tree, "commit", "-qm", "c3"], check=True, env=env)
|
|
cell("c18 a host root in a string VALUE is a host pin",
|
|
host_pin(tree, "HEAD", "wizard-work/x/verify_pinned.py").startswith("L3:"))
|
|
cell("c19 a host root in a docstring is prose, not a pin",
|
|
host_pin(tree, "HEAD", "wizard-work/x/verify_prose.py") == "")
|
|
|
|
# identity vs equality on a PARSED row -- the defect measured at d45a5ed:
|
|
# `r["verdict"] is PASS` was False for every row read from disk, so the
|
|
# graded-clean figure read 0 over a ledger that held clean PASS rows.
|
|
parsed = os.path.join(tmp, "parsed.tsv")
|
|
with open(parsed, "w", encoding="utf-8") as fh:
|
|
fh.write("# c\n")
|
|
fh.write("ungated\twizard-work/x/verify_clean.py\tPASS\t0\t0.01\t\tdeadbeef\t\n")
|
|
fh.write("ungated\twizard-work/x/verify_pin.py\tPASS\t0\t0.01\t\tcafe\t"
|
|
"L3:/root/troupe-standup\n")
|
|
fh.write("ungated\twizard-work/x/verify_other.py\tOTHER_RC\t7\t0.01\t\tbeef\t\n")
|
|
praw, perr, _ = read_ledger(parsed)
|
|
cell("c20 a PARSED PASS row on a host-independent verifier is graded clean",
|
|
perr is None and len(praw) == 3
|
|
and row_is_clean(praw[0], {praw[0]["path"]}) is True
|
|
and row_is_clean(praw[1], {praw[1]["path"]}) is False)
|
|
cell("c21 a PARSED OTHER_RC row with rc 7 is consistent (not a fault)",
|
|
perr is None and rc_consistent(praw[2]) is None)
|
|
# the summary sentence: one figure's total beside another figure's tally.
|
|
# Measured 2026-10-25 02:18Z at HEAD `77d5eba`, default `--class undeclared`:
|
|
# `rows at emit : 79 (PASS 1, FAIL 5)`. Each count was right; the line
|
|
# claimed 1 + 5 rows out of 79. c22 fails if tally() stops scoping to pop.
|
|
cell("c22 a class-narrowed tally sums to its own population, not the ledger",
|
|
sum(tally(praw).values()) == 3
|
|
and sum(tally(praw, {praw[0]["path"]}).values()) == 1)
|
|
|
|
passed = sum(1 for _, ok in cells if ok)
|
|
for n, ok in cells:
|
|
print("%-4s %s" % ("ok" if ok else "FAIL", n))
|
|
print("selftest: %d cells, %d passed, %d failed" % (len(cells), passed,
|
|
len(cells) - passed))
|
|
return 0 if passed == len(cells) else 1
|
|
|
|
|
|
def main():
|
|
ap = argparse.ArgumentParser(
|
|
description="run every committed verify_*.py nothing else invokes, grade its rc")
|
|
ap.add_argument("--repo", default=DEFAULT_REPO)
|
|
ap.add_argument("--rev", default="HEAD")
|
|
ap.add_argument("--q63", default=DEFAULT_Q63, help="the class ledger to read")
|
|
ap.add_argument("--class", dest="cls", default="undeclared",
|
|
help="undeclared (Q65 residual) | declared-one-off | all")
|
|
ap.add_argument("--lane", help="with --emit: only wizard-work/<lane>/ rows")
|
|
ap.add_argument("--only", help="with --emit: only paths containing this substring")
|
|
ap.add_argument("--merge", action="store_true",
|
|
help="with --emit: keep rows for paths this run did not touch")
|
|
ap.add_argument("--ledger", default=DEFAULT_LEDGER)
|
|
ap.add_argument("--windows", default=DEFAULT_WINDOWS,
|
|
help="per-row timeout table: <path>\\t<seconds>\\t<why>")
|
|
ap.add_argument("--timeout", type=float, default=60.0,
|
|
help="fallback window for paths --windows does not name")
|
|
ap.add_argument("--cwd", choices=("dir", "repo"), default="dir")
|
|
ap.add_argument("--emit", action="store_true", help="execute and write the ledger")
|
|
ap.add_argument("--rebind", action="store_true",
|
|
help="rewrite the static blob/host columns of the ledger's "
|
|
"existing rows without executing anything")
|
|
ap.add_argument("--check", action="store_true", help="grade the ledger, no execution")
|
|
ap.add_argument("--strict", action="store_true",
|
|
help="with --check: any non-PASS row is a fault")
|
|
ap.add_argument("--why", action="store_true", help="per-row detail")
|
|
ap.add_argument("--limit", type=int, default=0, help="stop after N rows (probe runs)")
|
|
ap.add_argument("--json")
|
|
ap.add_argument("--selftest", action="store_true")
|
|
args = ap.parse_args()
|
|
|
|
if args.selftest:
|
|
return selftest()
|
|
|
|
if not (args.emit or args.check or args.why or args.rebind):
|
|
ap.print_help()
|
|
return 2
|
|
|
|
if args.rebind:
|
|
# No execution: blob and host are static properties of the commit, so
|
|
# they can be re-derived without re-measuring an rc. This exists because
|
|
# an rc is evidence about content, and the ledger must be able to say
|
|
# WHICH content after the archive has moved without pretending it
|
|
# re-ran anything.
|
|
rows, err, _ = read_ledger(args.ledger)
|
|
if err:
|
|
print("q65_runner: %s" % err)
|
|
return 2
|
|
for r in rows:
|
|
r["blob"] = blob_of(args.repo, args.rev, r["path"])
|
|
r["host"] = host_pin(args.repo, args.rev, r["path"])
|
|
out = emit(rows, args.ledger, args.repo, args.rev, args.timeout)
|
|
pinned = sum(1 for r in rows if r["host"])
|
|
print("q65_runner --rebind: %d row(s), %d host-pinned, -> %s"
|
|
% (len(rows), pinned, out))
|
|
return 0
|
|
|
|
try:
|
|
pop = population(args.repo, args.rev, args.q63, args.cls)
|
|
except SystemExit as exc:
|
|
print("q65_runner: %s" % exc)
|
|
return 2
|
|
if not pop:
|
|
print("q65_runner: population is empty for --class %s at %s"
|
|
% (args.cls, args.rev))
|
|
return 2
|
|
|
|
try:
|
|
win = read_windows(args.windows)
|
|
except SystemExit as exc:
|
|
print("q65_runner: %s" % exc)
|
|
return 2
|
|
|
|
if args.emit:
|
|
before = git(args.repo, "status", "--porcelain").stdout
|
|
rows = []
|
|
todo = sorted(pop)
|
|
if args.lane:
|
|
todo = [p for p in todo if lane_of(p) == args.lane]
|
|
if args.only:
|
|
todo = [p for p in todo if args.only in p]
|
|
if args.limit:
|
|
todo = todo[:args.limit]
|
|
if args.merge and os.path.exists(args.ledger):
|
|
have, err, _ = read_ledger(args.ledger)
|
|
if err:
|
|
print("q65_runner: %s" % err)
|
|
return 2
|
|
rows = [r for r in have if r["path"] not in set(todo)]
|
|
t0 = time.time()
|
|
for p in todo:
|
|
r = run_one(args.repo, p, args.timeout, args.cwd, windows=win)
|
|
r["class"] = pop[p]
|
|
rows.append(r)
|
|
# rewrite after every row: an interrupted sweep keeps the rows it
|
|
# finished, and --merge resumes from them
|
|
emit(rows, args.ledger, args.repo, args.rev, args.timeout)
|
|
print(" %-8s %6.2fs %s" % (r["verdict"], r["seconds"], p), flush=True)
|
|
out = emit(rows, args.ledger, args.repo, args.rev, args.timeout)
|
|
after = git(args.repo, "status", "--porcelain").stdout
|
|
led_rel = os.path.relpath(args.ledger, args.repo)
|
|
dirtied = sorted(d for d in set(after.splitlines()) - set(before.splitlines())
|
|
if led_rel not in d)
|
|
counts = {}
|
|
for r in rows:
|
|
counts[r["verdict"]] = counts.get(r["verdict"], 0) + 1
|
|
print("q65_runner: %d row(s) in %.1fs -> %s" % (len(rows), time.time() - t0, out))
|
|
print(" " + ", ".join("%s %d" % (k, counts[k]) for k in VERDICTS if counts.get(k)))
|
|
if dirtied:
|
|
print(" WARNING: the run changed the worktree:")
|
|
for d in dirtied:
|
|
print(" " + d)
|
|
if args.json:
|
|
with open(args.json, "w", encoding="utf-8") as fh:
|
|
json.dump({"rev": args.rev, "rows": rows, "counts": counts,
|
|
"dirtied": dirtied}, fh, indent=1)
|
|
return 0 if not dirtied else 0
|
|
|
|
if args.check:
|
|
return check(args.repo, args.rev, args.q63, args.cls, args.ledger, args.strict)
|
|
|
|
# --why on its own: print the ledger, grouped by lane
|
|
rows, err, meta = read_ledger(args.ledger)
|
|
if err:
|
|
print("q65_runner: %s" % err)
|
|
return 2
|
|
print("# ledger rev %s, %d row(s)" % (meta.get("rev", "?"), len(rows)))
|
|
by_lane = {}
|
|
for r in rows:
|
|
lane = r["path"].split("/")[1] if r["path"].startswith("wizard-work/") else "?"
|
|
by_lane.setdefault(lane, []).append(r)
|
|
for lane in sorted(by_lane):
|
|
rs = by_lane[lane]
|
|
mark = ",".join(sorted({x["verdict"] for x in rs}))
|
|
print("%-9s %3d row(s) %s" % (lane, len(rs), mark))
|
|
for r in sorted(rs, key=lambda x: x["path"]):
|
|
if r["verdict"] != PASS:
|
|
print(" %-8s rc %-4s %s %s" % (r["verdict"], r["rc"] or "-",
|
|
r["path"], r["note"][:90]))
|
|
elif r["host"]:
|
|
print(" %-8s %-8s %s reads %s" % ("PASS*", "rc 0", r["path"],
|
|
r["host"]))
|
|
pinned = sum(1 for r in rows if r["verdict"] == PASS and r["host"])
|
|
unbound = sum(1 for r in rows if not r["blob"])
|
|
print("# PASS* (host-pinned) %d, rows without a blob %d of %d"
|
|
% (pinned, unbound, len(rows)))
|
|
return 0
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|