stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
fdc4a2178d24bccca55cbc26cf10b8b5f245af8f
troupe-standup/tools/verify.py
T
volt a114eadf3d tools (volt): the gate gets its fourth leg -- the check-in contract, graded on the newest day 
Measured hole: on a clone of 3890c12 carrying checkins/2026-10-19-volt-02.md,
checkin_check.py --since 2026-10-19 --fail returned rc 1 (BAD_NAME 1, DUPLICATE
group 1) while verify.py --fast returned rc 0, 'gate: OK -- 2 of 2 graded leg(s)
passed'. Three legs grade the queue, its append discipline and the tools'
selftests; none graded the one artifact every lane writes every tick.

The leg grades the newest day present in checkins/ (read from the archive, not a
clock and not a constant) because whole-archive grading is rc 1 with 35 faults
permanently -- 2026-09-24 through 2026-10-07, before the rule they break was
written. checkin_check --fail at 3890c12: 73 files, 35 with a fault, rc 1.
--checkin-day DATE sets the --since floor explicitly. No checkins/ dir or an
empty archive is UNGRADABLE (rc 2), never a pass and never a fault.

pick_detail() gains STRONG>WEAK>SKIP tiering: the first real run put
'FAULT rc=1  SKIP README.md (the directory's own README: not a check-in)' on the
row, because the weak hint 'not a ' matched the README note before the tool's
own FAIL line. Silent-class defect, fixed here with cells 18/18b.

Evidence: verify --selftest 26 -> 35 cells, 35 passed, rc 0 (cells 14-18b new:
fault->rc1, no-dir->rc2, newest-day --since, --checkin-day, the two detail-line
cases). ruff -F,E9 tools/ All checks passed. Live gate: 4 legs, 4 pass, rc 0.
Same clone after the change: 4 legs, 2 pass, 1 fault, rc 1, detail
'FAIL: 2 file(s) violate the check-in contract'. doc_census_check --tools
tools/verify.py: 6 claims, 0 STALE, 0 INTERNAL SPLIT, rc 0.
Artifact: /root/.hermes/wizard-outputs/volt/2026-10-19-checkin-leg/
2026-10-19 12:14:47 +00:00

606 lines
29 KiB
Python
Raw Blame History

#!/usr/bin/env python3
"""verify.py -- the HOME the index gate did not have: one command, four legs,
one exit code, and a row per leg so a partial run cannot look like a clean one.
WHY THIS EXISTS
On 2026-10-18 this lane told Alexander, three ticks running, to "wire
`queue_index --check` into the tick's check.sh". There is no `check.sh` in
this repo and no `scripts/` directory at all -- `git ls-files` shows `tools/`
and nothing else. The advice named a target that does not exist, which is why
it was never adopted. The gate did not need a decision about policy; it needed
a home. This is that home, and it is a task, not a policy call: nothing here
obliges any lane to run it, and nothing is enforced until a lane does.
The second reason is narrower and this lane's own: three separate counts of
"which commits left the index stale" came out wrong in one week, each from a
hand-assembled probe loop. A gate whose legs are remembered is a gate that
gets run selectively. Fixed order, one rc -- that is the whole design.
The third reason is measured, not argued. On 2026-10-19T09:42Z this lane put a
contract-breaking check-in into a clone of HEAD -- `checkins/2026-10-19-volt-02.md`,
a second same-day file carrying a suffix -- and ran both instruments on it:
`checkin_check.py --since 2026-10-19 --fail` returned **rc 1** (BAD_NAME 1,
DUPLICATE group 1) while `verify.py --fast` returned **rc 0** on the same tree,
"gate: OK -- 2 of 2 graded leg(s) passed". Three legs that grade the queue, its
append discipline and the tools' selftests said nothing about the one artifact
every lane writes every tick. A fourth leg closes that hole.
WHAT IT RUNS -- one row per leg, cheapest first, never in parallel
checkin tools/checkin_check.py --since <newest day> --fail
the newest day's check-ins against F1-F5 (names, duplicates,
unknown handles, dates, emptiness); `--checkin-day DATE` sets
the floor explicitly, from DATE forward
index-current tools/queue_index.py --check docs/queue-index.md
the derived document, byte-compared against a fresh render
append-audit tools/queue_append_audit.py --guard --to HEAD
the newest queue-touching commit, graded in ITS OWN tree
selftest-sweep tools/selftest_sweep.py
every tool's --selftest plus its two read-refusal axes
(skip with --fast; the row is still printed and counted)
The legs are ordered by cost, and a leg is never skipped because an earlier
one failed: a stale index and a broken selftest are different defects and the
run should name both in one pass.
WHY THE CHECK-IN LEG GRADES ONE DAY, NOT THE ARCHIVE
Grading the whole archive is not a stricter gate, it is a permanently red one:
`checkin_check.py --repo . --fail` at HEAD returns **rc 1 with 35 faults** (25
BAD_NAME, 10 DUPLICATE groups), every one of them dated 2026-09-24 through
2026-10-07, before the rule it breaks was written. A gate wired to that number
can never go green, so it stops being read -- which is the same failure as a
gate that is never run. The newest day in `checkins/` is the only day on which a NEW fault
can appear, so it is the day this leg grades: history is left as the record it
is (the tool renames nothing, by design), and a suffix, duplicate, bad date or
empty file on the current day is rc 1 the moment it lands. `--checkin-day DATE`
lowers the floor explicitly, in the open, when an operator wants a wider window
(it is a `--since`, so it can only widen: an older DATE pulls more history in,
it never narrows the window past a newer DATE).
THE EXIT CONTRACT -- the house three codes
0 every graded leg passed
1 at least one leg ran and graded a fault
2 at least one leg could not grade -- a missing tool, a timeout, an
unexpected exit code, an unreadable repo -- or no leg was runnable at all
rc 2 OUTRANKS rc 1 deliberately. rc 1 means "I ran and I graded a fault";
a run with an ungradable leg has not made that claim about the whole gate,
and folding the two (as the 15:16Z correction caught `queue_append_audit`
doing with ungradable commits) is how a "3 stale" reading hides a "2 could
not be read". The summary always prints both counts, so precedence never
erases a fault -- it only refuses to certify the run that found it.
A skipped leg is not a pass and not a fault: it is counted as `skipped` and
printed as SKIP. `--fast` on a green tree exits 0 with one row SKIP, and the
summary says so rather than pretending the sweep ran.
TOOLS ARE RESOLVED FROM --repo, NOT FROM THIS FILE'S DIRECTORY
`verify.py --repo /path/to/worktree` runs THAT tree's `queue_index.py`,
`queue_append_audit.py` and `selftest_sweep.py`. That is the instrument
lesson the 15:16Z correction paid for: a gate run in a working tree at HEAD
says nothing about a cited revision, so grading a revision means grading its
own tools from its own checkout.
--queue and --index stay REPO-RELATIVE (defaults WORK-QUEUE.md and
docs/queue-index.md) because `queue_append_audit` reads them with
`git show <rev>:<path>`, which has no absolute-path form.
`--selftest` fixture corpus, 35 cells
"""
import argparse
import json
import os
import re
import subprocess
import sys
import tempfile
import time
QUEUE_DEFAULT = "WORK-QUEUE.md"
INDEX_DEFAULT = "docs/queue-index.md"
CHECKINS_DEFAULT = "checkins"
PASS, FAULT, UNGRADABLE, SKIP = "PASS", "FAULT", "UNGRADABLE", "SKIP"
# Legs, in the fixed order they are printed and run. `tool` is resolved inside
# the repo being graded. `slow` marks the leg --fast drops.
LEGS = (
{"key": "checkin", "tool": "checkin_check.py", "slow": False,
"what": "grade the newest day's check-ins against the naming and "
"duplicate rules"},
{"key": "index-current", "tool": "queue_index.py", "slow": False,
"what": "render the index and byte-compare it with the document"},
{"key": "append-audit", "tool": "queue_append_audit.py", "slow": False,
"what": "grade the newest queue-touching commit in its own tree"},
{"key": "selftest-sweep", "tool": "selftest_sweep.py", "slow": True,
"what": "every tool's --selftest and both read-refusal axes"},
)
# A detail line is a courtesy, not the verdict -- but a verdict with no line
# beside it is what the read-refusal axis calls SILENT, so it is graded too.
#
# STRONG outranks WEAK, and both outrank a per-file SKIP note. That tiering is
# not cosmetic: `checkin_check.py` prints " SKIP README.md (the directory's own
# README: not a check-in)" as its second line on every real run, "not a " is a
# WEAK hint, and before the tiering that note was carried onto a FAULT row --
# the row said FAULT beside a line about the README while the tool's own
# "FAIL: 2 file(s) violate the check-in contract" sat further down, unpicked.
STRONG_HINTS = ("STALE:", "FAIL:", "FAILED", "fatal", "TIMEOUT", "NO_",
"cannot", "error", "Error", "refus")
WEAK_HINTS = ("FAIL", "not a ")
def pick_detail(text, limit=150):
"""First line that names a reason, else the last line that says anything.
A summary line that reports ZERO faults is not a reason: `selftest_sweep`
prints `sweep: 15 tools, 15 OK, 0 fatal` even on the run where its input
axes went red, and that line was being carried onto a FAULT row -- the row
said FAULT beside the words "0 fatal". Zero-count summaries are therefore
only reachable on the second pass (a clean run still gets its sweep line),
and any line that names a real reason outranks them.
"""
lines = [ln.strip() for ln in text.splitlines() if ln.strip()]
kept = [ln for ln in lines if not ln.startswith("SKIP ")] or lines
hinted = [ln for ln in kept if any(h in ln for h in STRONG_HINTS + WEAK_HINTS)]
reasons = [ln for ln in hinted
if " 0 fatal" not in ln and " 0 fault" not in ln]
strong = [ln for ln in reasons if any(h in ln for h in STRONG_HINTS)]
if strong:
return strong[0][:limit]
if reasons:
return reasons[0][:limit]
if hinted:
return hinted[0][:limit]
return kept[-1][:limit] if kept else ""
def newest_checkin_day(repo):
"""The newest `<YYYY-MM-DD>` prefix in checkins/ -> (day, why-not).
The leg grades one day and this is how that day is chosen: the archive's own
latest date, so nothing has to be configured and no clock is consulted. A
repo with no `checkins/` directory or no dated file in it cannot be graded
and says which of the two it is.
"""
d = os.path.join(repo, CHECKINS_DEFAULT)
if not os.path.isdir(d):
return None, "no check-in directory: %s" % d
days = sorted(m.group(1) for m in
(re.match(r"^(\d{4}-\d{2}-\d{2})-", n) for n in os.listdir(d))
if m)
if not days:
return None, "no dated check-in files in %s" % d
return days[-1], None
def leg_argv(key, repo, queue, index, checkin_day=None):
if key == "checkin":
return [sys.executable, os.path.join(repo, "tools", "checkin_check.py"),
"--repo", repo, "--fail", "--since", checkin_day]
if key == "index-current":
return [sys.executable, os.path.join(repo, "tools", "queue_index.py"),
"--repo", repo, "--queue", queue, "--check", index]
if key == "append-audit":
return [sys.executable, os.path.join(repo, "tools", "queue_append_audit.py"),
"--repo", repo, "--queue", queue, "--index", index,
"--guard", "--to", "HEAD"]
if key == "selftest-sweep":
return [sys.executable, os.path.join(repo, "tools", "selftest_sweep.py"),
"--repo", repo, "--quiet"]
raise KeyError(key)
def run_leg(leg, repo, queue, index, timeout, checkin_day=None):
"""One leg -> one row. rc 0/1/2 are graded; anything else is ungradable."""
tool = os.path.join(repo, "tools", leg["tool"])
started = time.time()
row = {"key": leg["key"], "tool": leg["tool"], "verdict": None,
"rc": None, "detail": "", "what": leg["what"]}
if not os.path.exists(tool):
row.update(verdict=UNGRADABLE, detail="no such tool: %s" % tool)
row["ms"] = 0
return row
day = checkin_day
if leg["key"] == "checkin" and not day:
day, why = newest_checkin_day(repo)
if day is None:
row.update(verdict=UNGRADABLE, detail=why)
row["ms"] = 0
return row
try:
proc = subprocess.run(leg_argv(leg["key"], repo, queue, index, day),
cwd=repo, capture_output=True, text=True,
timeout=timeout)
except subprocess.TimeoutExpired:
row.update(verdict=UNGRADABLE, detail="TIMEOUT after %ss" % timeout)
row["ms"] = int((time.time() - started) * 1000)
return row
except OSError as exc:
row.update(verdict=UNGRADABLE, detail="could not run: %s" % exc)
row["ms"] = int((time.time() - started) * 1000)
return row
out = (proc.stdout or "") + (proc.stderr or "")
row["rc"] = proc.returncode
row["detail"] = pick_detail(out)
if proc.returncode == 0:
row["verdict"] = PASS
elif proc.returncode == 1:
row["verdict"] = FAULT
elif proc.returncode == 2:
row["verdict"] = UNGRADABLE
else:
# rc 3 is retired in this repo (one meaning, two codes); any other code
# is a tool that left the contract, and the run cannot grade through it.
row["verdict"] = UNGRADABLE
row["detail"] = "unexpected rc %d: %s" % (proc.returncode, row["detail"])
row["ms"] = int((time.time() - started) * 1000)
return row
def counts(rows):
c = {PASS: 0, FAULT: 0, UNGRADABLE: 0, SKIP: 0}
for r in rows:
c[r["verdict"]] += 1
return c
def decide(rows):
"""The precedence, in one place: ungradable outranks fault outranks clean."""
c = counts(rows)
if c[UNGRADABLE]:
return 2
if c[FAULT]:
return 1
return 0
def report(rows, repo, quiet=False):
c = counts(rows)
if not quiet:
print("verify -- repo: %s" % repo)
for r in rows:
rc = "-" if r["rc"] is None else str(r["rc"])
print(" %-16s %-10s rc=%-2s %-6s %s"
% (r["key"], r["verdict"], rc, "%dms" % r["ms"], r["detail"]))
print()
print("verify: %d legs, %d pass, %d fault, %d ungradable, %d skipped"
% (len(rows), c[PASS], c[FAULT], c[UNGRADABLE], c[SKIP]))
faulted = [r["key"] for r in rows if r["verdict"] == FAULT]
ungraded = [r["key"] for r in rows if r["verdict"] == UNGRADABLE]
if faulted:
print("fault: %s" % " ".join(faulted))
if ungraded:
print("ungradable: %s" % " ".join(ungraded))
rc = decide(rows)
if rc == 0:
print("gate: OK -- %d of %d graded leg(s) passed%s"
% (c[PASS], c[PASS] + c[FAULT],
"" if not c[SKIP] else
" (%d skipped: not a pass, not a fault)" % c[SKIP]))
elif rc == 1:
print("gate: FAULT -- %d leg(s) ran and graded a fault: %s"
% (c[FAULT], " ".join(faulted)))
else:
print("gate: NOT CERTIFIED -- %d leg(s) could not grade: %s"
% (c[UNGRADABLE], " ".join(ungraded)))
return rc
def main(argv=None):
ap = argparse.ArgumentParser(
description="Run the repo's gates as one command: the newest day's "
"check-ins, index currency, the append audit, and the "
"selftest sweep.")
ap.add_argument("--repo", default=".")
ap.add_argument("--queue", default=QUEUE_DEFAULT,
help="repo-relative queue path (default %s)" % QUEUE_DEFAULT)
ap.add_argument("--index", default=INDEX_DEFAULT,
help="repo-relative index document (default %s)" % INDEX_DEFAULT)
ap.add_argument("--checkin-day", default=None, metavar="DATE",
help="grade check-ins from DATE forward instead of from the "
"newest day present in checkins/ (a --since floor: it "
"can widen the window, never narrow it)")
ap.add_argument("--fast", action="store_true",
help="skip the selftest-sweep leg; the row is printed as SKIP")
ap.add_argument("--timeout", type=int, default=300, metavar="S",
help="per-leg timeout in seconds (default 300)")
ap.add_argument("--json", metavar="PATH", help="write the rows here")
ap.add_argument("--quiet", action="store_true")
ap.add_argument("--selftest", action="store_true", help="fixture corpus, 35 cells")
a = ap.parse_args(argv)
if a.selftest:
return selftest()
repo = os.path.abspath(a.repo)
if not os.path.isdir(repo):
print("cannot read repo: %s is not a directory" % repo, file=sys.stderr)
return 2
if not os.path.isdir(os.path.join(repo, ".git")):
print("cannot read repo: %s is not a git repository" % repo, file=sys.stderr)
return 2
rows = []
for leg in LEGS:
if a.fast and leg["slow"]:
rows.append({"key": leg["key"], "tool": leg["tool"], "verdict": SKIP,
"rc": None, "detail": "--fast: leg not run",
"what": leg["what"], "ms": 0})
continue
rows.append(run_leg(leg, repo, a.queue, a.index, a.timeout,
a.checkin_day))
rc = report(rows, repo, a.quiet)
if a.json:
c = counts(rows)
with open(a.json, "w") as fh:
json.dump({"repo": repo, "queue": a.queue, "index": a.index,
"fast": a.fast, "rows": rows, "counts": c, "rc": rc},
fh, indent=1)
return rc
# ------------------------------------------------------------------- selftest
def _stub(path, rc, out="", sleep=0.0, marker=None, argv_marker=None):
"""A stand-in leg: it prints what the fixture needs and exits with its rc."""
body = ["#!/usr/bin/env python3", "import sys"]
if sleep:
body.append("import time; time.sleep(%r)" % sleep)
if marker:
body.append("open(%r, 'a').write('hit\\n')" % marker)
if argv_marker:
body.append("open(%r, 'w').write(' '.join(sys.argv[1:]))" % argv_marker)
if out:
body.append("sys.stdout.write(%r)" % out)
body.append("sys.exit(%d)" % rc)
with open(path, "w") as fh:
fh.write("\n".join(body) + "\n")
os.chmod(path, 0o755)
def _fixture(base, rcs, outs=None, sleep=0.0, markers=None, argv_markers=None,
checkins=("2026-01-01-fixture.md",)):
"""A throwaway repo with stubbed legs. rcs: {tool: rc}.
`checkins` seeds `checkins/`: one dated file by default (so the check-in leg
has a day to grade), a tuple of names for cells that need a specific set,
and None for a repo with no `checkins/` directory at all.
"""
repo = os.path.join(base, "repo")
os.makedirs(os.path.join(repo, "tools"))
os.makedirs(os.path.join(repo, ".git"))
with open(os.path.join(repo, QUEUE_DEFAULT), "w") as fh:
fh.write("## Q1 -- fixture\n")
os.makedirs(os.path.join(repo, "docs"))
with open(os.path.join(repo, INDEX_DEFAULT), "w") as fh:
fh.write("fixture\n")
if checkins is not None:
os.makedirs(os.path.join(repo, CHECKINS_DEFAULT))
for name in checkins:
with open(os.path.join(repo, CHECKINS_DEFAULT, name), "w") as fh:
fh.write("Did: fixture\nBlocked: nothing\nNext: nothing\n"
"Ask: nothing\n")
for tool, rc in rcs.items():
if rc is None: # tool absent on purpose
continue
_stub(os.path.join(repo, "tools", tool), rc,
out=(outs or {}).get(tool, ""), sleep=sleep,
marker=(markers or {}).get(tool),
argv_marker=(argv_markers or {}).get(tool))
return repo
def _verify(*argv):
"""Run this file as the CLI, so the fixture grades the real entry point."""
proc = subprocess.run([sys.executable, os.path.abspath(__file__)] + list(argv),
capture_output=True, text=True)
return proc.returncode, (proc.stdout or "") + (proc.stderr or "")
GREEN4 = {"checkin_check.py": 0, "queue_index.py": 0,
"queue_append_audit.py": 0, "selftest_sweep.py": 0}
def selftest():
cells = 0
passed = 0
failures = []
def check(label, got, want):
nonlocal cells, passed
cells += 1
if got == want:
passed += 1
else:
failures.append("%s\n got: %r\n want: %r" % (label, got, want))
with tempfile.TemporaryDirectory(prefix="verify-selftest-") as td:
# 1. three green stubs -> rc 0
repo = _fixture(os.path.join(td, "c1"), GREEN4)
rc, out = _verify("--repo", repo)
check("1 all four legs green -> rc 0", rc, 0)
check("1b summary counts the four legs", "verify: 4 legs, 4 pass, 0 fault, "
"0 ungradable, 0 skipped" in out, True)
# 2. a stale index is a FAULT, and its own line reaches the row
repo = _fixture(os.path.join(td, "c2"),
dict(GREEN4, **{"queue_index.py": 1}),
outs={"queue_index.py":
"STALE: rendered document differs at line 7\n"})
rc, out = _verify("--repo", repo)
check("2 stale index -> rc 1", rc, 1)
check("2b the STALE line is carried into the row",
"index-current" in out and "differs at line 7" in out, True)
# 3. a refusal is UNGRADABLE, not a fault
repo = _fixture(os.path.join(td, "c3"),
dict(GREEN4, **{"queue_index.py": 2}),
outs={"queue_index.py": "cannot read queue: nope\n"})
rc, out = _verify("--repo", repo)
check("3 refusal -> rc 2", rc, 2)
check("3b refusal is named in the summary", "ungradable: index-current" in out, True)
# 4. precedence: ungradable outranks fault, and the fault is NOT erased
repo = _fixture(os.path.join(td, "c4"),
dict(GREEN4, **{"queue_index.py": 1,
"selftest_sweep.py": 2}),
outs={"queue_index.py": "STALE: line 3\n",
"selftest_sweep.py": "cannot read: no tools\n"})
rc, out = _verify("--repo", repo)
check("4 ungradable outranks fault -> rc 2", rc, 2)
check("4b both classes are still printed in one run",
"1 fault, 1 ungradable" in out and "fault: index-current" in out,
True)
# 5. a missing leg tool is ungradable, and says which path was missing
repo = _fixture(os.path.join(td, "c5"),
dict(GREEN4, **{"queue_append_audit.py": None}))
rc, out = _verify("--repo", repo)
check("5 missing tool -> rc 2", rc, 2)
check("5b the missing path is named", "no such tool:" in out, True)
# 6. --fast skips the sweep and does not run it
marker = os.path.join(td, "c6-sweep-ran")
repo = _fixture(os.path.join(td, "c6"), GREEN4, markers={"selftest_sweep.py": marker})
rc, out = _verify("--repo", repo, "--fast")
check("6 --fast on a green tree -> rc 0", rc, 0)
check("6b the skipped leg is printed as SKIP", "selftest-sweep" in out
and "SKIP" in out, True)
check("6c the skipped stub never ran", os.path.exists(marker), False)
# 7. --json is a real machine-readable record
repo = _fixture(os.path.join(td, "c7"), GREEN4)
jpath = os.path.join(td, "c7.json")
rc, out = _verify("--repo", repo, "--json", jpath)
obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
check("7 --json writes the rows", [r["key"] for r in obj.get("rows", [])],
["checkin", "index-current", "append-audit", "selftest-sweep"])
check("7b --json carries the same rc the process returned",
obj.get("rc"), rc)
# 8. a repo that is not there is refused before any leg runs
rc, out = _verify("--repo", os.path.join(td, "no-such-repo"))
check("8 missing repo -> rc 2", rc, 2)
check("8b the refusal names the path", "cannot read repo:" in out, True)
# 9. rc 3 is retired: it is ungradable, never a fault and never a pass
repo = _fixture(os.path.join(td, "c9"),
dict(GREEN4, **{"queue_index.py": 3}),
outs={"queue_index.py": "unknown state\n"})
rc, out = _verify("--repo", repo)
check("9 rc 3 -> rc 2 (retired code)", rc, 2)
check("9b the unexpected code is named", "unexpected rc 3" in out, True)
# 10. a hung leg is a timeout, not a pass and not a fault
repo = _fixture(os.path.join(td, "c10"),
dict(GREEN4, **{"selftest_sweep.py": 0}), sleep=5.0)
rc, out = _verify("--repo", repo, "--timeout", "1")
check("10 timeout -> rc 2", rc, 2)
check("10b the timeout is named", "TIMEOUT after 1s" in out, True)
# 11. order is fixed, cheapest first: the table is not re-sorted by outcome
repo = _fixture(os.path.join(td, "c11"),
dict(GREEN4, **{"queue_index.py": 1}),
outs={"queue_index.py": "STALE: line 1\n"})
rc, out = _verify("--repo", repo)
order = [out.find(k) for k in ("checkin", "index-current", "append-audit",
"selftest-sweep")]
check("11 legs print in fixed order", order == sorted(order)
and -1 not in order, True)
# 12. a skip is counted as a skip, never folded into pass
repo = _fixture(os.path.join(td, "c12"), GREEN4)
jpath = os.path.join(td, "c12.json")
rc, out = _verify("--repo", repo, "--fast", "--json", jpath)
obj = json.load(open(jpath)) if os.path.exists(jpath) else {}
check("12 skipped leg is its own count in --json",
obj.get("counts", {}).get("SKIP"), 1)
check("12b and the skipped leg is not counted as a pass",
obj.get("counts", {}).get(PASS), 3)
# 13. the detail line must not contradict the verdict: a row graded
# FAULT beside "0 fatal" is the SILENT failure this gate exists to catch.
check("13 a zero-fault summary is not a reason line",
pick_detail("sweep: 15 tools, 15 OK, 0 fatal\n"
"read-refusal: 20 probes, 19 ok, 1 fault (NO_PROBE)\n"
"malformed-input: 18 probes, 17 ok, 1 fault (NO_PROBE)\n"),
"read-refusal: 20 probes, 19 ok, 1 fault (NO_PROBE)")
check("13b a clean run still gets its sweep line",
pick_detail("sweep: 14 tools, 14 OK, 0 fatal\n"
"read-refusal: 19 probes, 19 ok, 0 fault\n"),
"sweep: 14 tools, 14 OK, 0 fatal")
# 14. THE HOLE THIS LEG CLOSES: a contract-breaking check-in is rc 1,
# and the gate is no longer green on a tree that carries one.
repo = _fixture(os.path.join(td, "c14"),
dict(GREEN4, **{"checkin_check.py": 1}),
outs={"checkin_check.py":
"FAIL: 2 file(s) violate the check-in contract\n"})
rc, out = _verify("--repo", repo, "--fast")
check("14 a broken check-in -> rc 1 (was rc 0 with three legs)", rc, 1)
check("14b the fault row is named checkin and carries its reason",
"checkin" in out and "violate the check-in contract" in out
and "fault: checkin" in out, True)
# 15. no checkins/ directory is ungradable, not a pass and not a fault
repo = _fixture(os.path.join(td, "c15"), GREEN4, checkins=None)
rc, out = _verify("--repo", repo, "--fast")
check("15 no checkins dir -> rc 2", rc, 2)
check("15b the ungradable row says which directory is missing",
"ungradable: checkin" in out and "no check-in directory:" in out, True)
# 16. the leg grades the NEWEST day in the archive, not the whole archive
av = os.path.join(td, "c16-argv")
repo = _fixture(os.path.join(td, "c16"), GREEN4,
argv_markers={"checkin_check.py": av},
checkins=("2026-01-01-fixture.md", "2026-03-04-fixture.md"))
rc, out = _verify("--repo", repo, "--fast")
seen = open(av).read() if os.path.exists(av) else ""
check("16 the check-in leg is run with --fail", "--fail" in seen, True)
check("16b and --since is the newest day present, not a clock or a constant",
"--since 2026-03-04" in seen, True)
# 17. --checkin-day retargets the window in the open
av = os.path.join(td, "c17-argv")
repo = _fixture(os.path.join(td, "c17"), GREEN4,
argv_markers={"checkin_check.py": av},
checkins=("2026-01-01-fixture.md", "2026-03-04-fixture.md"))
rc, out = _verify("--repo", repo, "--fast", "--checkin-day", "2026-01-01")
seen = open(av).read() if os.path.exists(av) else ""
check("17 --checkin-day overrides the default day",
"--since 2026-01-01" in seen, True)
# 18. the detail line must survive a real tool's noise: `checkin_check`
# prints a README SKIP note carrying "not a " on every run, and a WEAK
# hint must not beat the FAIL line that actually names the verdict.
noisy = ("== 2 check-in files in /tmp/x/checkins ==\n"
"roster: 9 handles; today 2026-10-19\n"
"since 2026-10-19: 2 graded, 72 older file(s) not graded\n"
" SKIP README.md (the directory's own README: not a check-in)\n"
"\n"
"2026-10-19-volt-02.md BAD_NAME,DUPLICATE 2 files for volt\n"
"checkin_check: 2 files, 2 with a fault (BAD_NAME 1)\n"
"FAIL: 2 file(s) violate the check-in contract\n")
check("18 a README skip note does not mask the FAIL line",
pick_detail(noisy), "FAIL: 2 file(s) violate the check-in contract")
# 18b. and on a clean run the same noise must not become the detail:
# the row reads PASS beside the summary, not beside a README note.
clean = noisy.replace("checkin_check: 2 files, 2 with a fault (BAD_NAME 1)\n"
"FAIL: 2 file(s) violate the check-in contract\n",
"checkin_check: 2 files, 0 with a fault, 1 advisory\n")
check("18b a clean check-in row carries its summary, not the skip note",
pick_detail(clean), "checkin_check: 2 files, 0 with a fault, 1 advisory")
print("selftest: %d cells, %d passed, %d failed" % (cells, passed, len(failures)))
for f in failures:
print(" FAIL %s" % f)
return 0 if passed == cells else 1
if __name__ == "__main__":
sys.exit(main())
Reference in new issue View Git Blame Copy Permalink