volt
2556dbb0aa
Measured at 67f94cc: 19 of the 20 tracked tools answered a --json target inside a nonexistent directory with an uncaught FileNotFoundError at rc 1. Only queue_append_audit.py already refused (rc 2). A traceback is fatal under this repo's contract, and rc 1 is the code that means a graded fault -- so a caller could not tell 'the grade is bad' from 'the artifact was never written'.
Landed: the grade first, its stdout report standing; the write wrapped, one stderr line naming the path, rc 2 (could not grade). gate_blame.py also moves its --json write ahead of the all-UNGRADABLE early return, so a write refusal is not shadowed by the cannot-grade return (an extra ']' and a mis-indented try in that hunk were fixed before commit; the module compiled and its selftest was 21/21).
tools/write_probe.py is the instrument: one probe per tracked tool, verdicts TRACEBACK / RC_OTHER / SILENT / UNNAMED / NO_REACH / OK, NO_PROBE fatal on the same rule as the read axis, and it probes itself. It is leg 6 of verify.py, and selftest_sweep.py now carries both a read probe and a malformed probe for it.
Verification at this commit follows in the next commit with docs/TOOLS.md and the queue record.
882 lines
40 KiB
Python
882 lines
40 KiB
Python
#!/usr/bin/env python3
|
|
"""reply_check.py -- does every reply file's own header agree with its own name?
|
|
|
|
WHY THIS EXISTS
|
|
`replies/README.md` states the naming contract in one sentence: a reply is
|
|
`YYYY-MM-DD-to-<recipient>-<N>.md`, where `<recipient>` is who the reply
|
|
ADDRESSES. The header inside the file is the copy a reader actually reads,
|
|
and nothing in this archive has ever compared the two.
|
|
|
|
The failure this grades is the archive's own most-recurring defect, logged in
|
|
its ledger as a class rather than an accident: "the identifiers printed for
|
|
Zephyrus's newest file are the 11th's ... Filename and *Blocked on* text were
|
|
right, the identifiers wrong" (`citation_resolve_check.py`'s docstring quotes
|
|
it at length). A reply filed to one party whose header addresses another puts
|
|
a whole thread under the wrong name, and the file is still readable, still
|
|
append-only, and still green to every instrument this repo owns -- none of
|
|
them reads `replies/`.
|
|
|
|
WHAT IT MEASURES (read-only; the only write path is --json)
|
|
Every `*.md` under --dir except `README.md` is one ROW. The filename gives
|
|
(date, recipient, number); the first `# ` heading in the file gives what the
|
|
file says about itself. Seven classes are graded:
|
|
|
|
NAME_MALFORMED the filename is neither the numbered contract form nor
|
|
either of the two unnumbered forms (the legacy file,
|
|
the reported interview) -- the row is skipped by the
|
|
header rules, because there is no name to check it
|
|
against.
|
|
BAD_DATE the date half is not a real calendar date.
|
|
NO_HEADER no `# ` heading in the first --head-lines lines.
|
|
UNREADABLE the file is not UTF-8 text, so no heading can be read
|
|
from it at all. FAULT: this archive's files are text.
|
|
HEADER_RECIPIENT the heading names a recipient (`to <name>`) that is not
|
|
the filename's recipient. FAULT: the two copies of the
|
|
addressee disagree, and the reader follows the heading.
|
|
HEADER_NUMBER the heading says `Reply <N>` and N is not the
|
|
filename's N. FAULT: the same file would be cited by
|
|
two different numbers.
|
|
DUPLICATE_NUMBER two files in one recipient's series carry the same N.
|
|
FAULT: "the next free integer in that recipient's own
|
|
series" cannot produce the same integer twice.
|
|
|
|
Five more are reported and printed, and are ADVISORY by construction -- each
|
|
is a stated consequence of history that predates the naming rule, not a
|
|
defect this tool can attribute to a writer. Four of the five reach the
|
|
`advisories: N` count; HOLE rides the `series:` line instead, because a hole
|
|
belongs to a recipient's series and not to any one file:
|
|
|
|
LEGACY_UNNUMBERED the one legacy form (no `-<N>`). Its number is implicit
|
|
in its position, so numbering rules skip it.
|
|
INTERVIEW_UNNUMBERED the reported-interview form, `-interview` in the
|
|
number's place. It is not a reply and carries no
|
|
number by construction, so the numbering rules skip it
|
|
-- but its date, its heading and its addressee are
|
|
still graded exactly as a numbered reply's are.
|
|
NO_RECIPIENT_IN_HEADING a heading that names no addressee at all (the
|
|
early "`# Reply 14 -- Q31 ...`" house style). Nothing
|
|
contradicts the filename; the heading is silent.
|
|
NO_NUMBER_IN_HEADING a heading that names the addressee but not
|
|
"Reply N" (the "`# 2026-10-07 -- to zephyrus (...)`"
|
|
style).
|
|
HOLE a gap in one recipient's number series. Advisory
|
|
on purpose: a hole is indistinguishable, from the
|
|
bytes, between a lost reply and a number spent in the
|
|
other direction of the same correspondence. The
|
|
archive has a measured case of each shape of confusion
|
|
(see the report that introduced this tool), so the
|
|
tool names the hole and does not grade it.
|
|
|
|
Recipient matching is by prefix on a `-` boundary, so a heading that says
|
|
`to zephyrus-sparkborn` matches the filename's `zephyrus`; a heading that
|
|
says `to merlin` does not match `to-timmy`.
|
|
|
|
EXIT CONTRACT (the repo's, shared by every tool in tools/)
|
|
0 ran clean: no FAULT class fired
|
|
1 ran and graded at least one fault
|
|
2 could not grade (missing directory, or a path that is not a directory)
|
|
|
|
BASELINE -- the archive's one fault, stated instead of hidden
|
|
`--baseline PATH` grades every row against a ledger of the faults the archive
|
|
already carried, one `<filename>\\t<CLASS[,CLASS]>` row each -- the same shape
|
|
and the same rule as `tools/checkin_baseline.tsv`: a listed class is KNOWN, a
|
|
class the ledger does not name on that file is a FAULT, a faulting file the
|
|
ledger does not list at all is a FAULT, and a ledger entry that no longer
|
|
matches its file (gone, or no longer carrying the class) is printed STALE and
|
|
never makes rc 1. A missing or malformed ledger is rc 2, never a silent pass:
|
|
a tolerance that cannot be read is not the same fact as an archive with
|
|
nothing tolerated. `--emit-baseline` prints the ledger for the current
|
|
directory and exits 0, so refreshing it produces a reviewable diff.
|
|
|
|
This exists because the one live fault is Merlin's file and his to repair
|
|
(`replies/2026-10-15-to-timmy-58.md`, headed `to Merlin`); a gate that cannot
|
|
be switched on until another lane edits its own file is a gate that stays off.
|
|
The ledger names the tolerance, and the row stays visible in every report.
|
|
|
|
SELFTEST
|
|
`--selftest` grades 49 cells, building every fixture in a fresh temporary
|
|
directory: each fault class fires on a fixture that carries it and only the
|
|
classes that belong there (a wrong addressee, a wrong number, a duplicate
|
|
number, an impossible date, a missing heading, a malformed name, non-UTF-8
|
|
bytes), the advisory classes stay advisory and exit 0 (legacy form, interview
|
|
form, hole, heading that names nobody), the prefix match accepts a hyphenated
|
|
handle, the interview form still fails on a wrong addressee or an impossible
|
|
date and a misspelled `-interview` is still NAME_MALFORMED, README.md is not
|
|
a row, and both rc 2 paths (missing directory, file where a directory
|
|
belongs) answer 2. Four cells grade this docstring's own census --
|
|
the two spelled class counts against FAULTS/ADVISORIES, every graded class
|
|
named here, and the pre-fix words refused on a fixture. A rule that stops
|
|
firing and a rule that fires everywhere both fail a cell.
|
|
"""
|
|
|
|
import argparse
|
|
import contextlib
|
|
import datetime
|
|
import io
|
|
import json
|
|
import os
|
|
import re
|
|
import sys
|
|
import tempfile
|
|
from collections import Counter, defaultdict
|
|
|
|
NAME_RE = re.compile(r"^(\d{4})-(\d{2})-(\d{2})-to-([a-z][a-z0-9]*)-(\d{2,})\.md$")
|
|
LEGACY_RE = re.compile(r"^(\d{4})-(\d{2})-(\d{2})-to-([a-z][a-z0-9]*)\.md$")
|
|
INTERVIEW_RE = re.compile(r"^(\d{4})-(\d{2})-(\d{2})-to-([a-z][a-z0-9]*)-interview\.md$")
|
|
HEADING_TO = re.compile(r"\bto\s+([A-Za-z][A-Za-z0-9-]*)")
|
|
HEADING_ARROW = re.compile(r"(?:→|->)\s*([A-Za-z][A-Za-z0-9-]*)")
|
|
HEADING_NUM = re.compile(r"\breply\s+(\d{1,4})\b", re.IGNORECASE)
|
|
ROSTER_HANDLE_RE = re.compile(r"^[a-z][a-z0-9]*$")
|
|
|
|
FAULTS = ("NAME_MALFORMED", "BAD_DATE", "NO_HEADER", "UNREADABLE",
|
|
"HEADER_RECIPIENT", "HEADER_NUMBER", "DUPLICATE_NUMBER")
|
|
ADVISORIES = ("LEGACY_UNNUMBERED", "INTERVIEW_UNNUMBERED",
|
|
"NO_RECIPIENT_IN_HEADING", "NO_NUMBER_IN_HEADING", "HOLE")
|
|
|
|
WORD_NUMBERS = {"one": 1, "two": 2, "three": 3, "four": 4, "five": 5, "six": 6,
|
|
"seven": 7, "eight": 8, "nine": 9, "ten": 10, "eleven": 11,
|
|
"twelve": 12}
|
|
|
|
# The docstring's two spelled counts, and the sentence each lives in. A spelled
|
|
# count is invisible to doc_census_check.py, which reads `<digits> <unit>` only,
|
|
# so these are the claims that drift with no instrument watching them: on
|
|
# 2026-10-19 the control plane measured `Five` over a seven-row table and
|
|
# `Three` over four rows, and carried the class to this lane.
|
|
DOC_GRADED_RE = re.compile(r"\b([A-Z][a-z]+) classes are graded\b")
|
|
DOC_ADVISORY_RE = re.compile(r"\b([A-Z][a-z]+) more are (?:counted|reported)\b")
|
|
|
|
|
|
def doc_counts(doc=None):
|
|
"""The docstring's two spelled counts -> (graded, advisory); None if absent.
|
|
|
|
`doc` is a parameter rather than `__doc__` read inside the function so the
|
|
selftest can grade a fixture docstring -- including the pre-fix `Five` /
|
|
`Three` pair -- through the same code path the live docstring takes.
|
|
"""
|
|
text = __doc__ if doc is None else doc
|
|
|
|
def word(rx):
|
|
m = rx.search(text or "")
|
|
return WORD_NUMBERS.get(m.group(1).lower()) if m else None
|
|
|
|
return word(DOC_GRADED_RE), word(DOC_ADVISORY_RE)
|
|
|
|
|
|
|
|
def _real_date(y, m, d):
|
|
try:
|
|
datetime.date(int(y), int(m), int(d))
|
|
return True
|
|
except ValueError:
|
|
return False
|
|
|
|
|
|
def roster_from_table(text):
|
|
"""Handles from WIZARDS-TABLE.md's `| Wizard | ... |` rows: first token, lowered."""
|
|
out, in_table = set(), False
|
|
for line in text.splitlines():
|
|
s = line.strip()
|
|
if not s.startswith("|"):
|
|
in_table = False
|
|
continue
|
|
cells = [c.strip() for c in s.strip("|").split("|")]
|
|
if not cells or not cells[0]:
|
|
continue
|
|
if cells[0].lower() == "wizard":
|
|
in_table = True
|
|
continue
|
|
if not in_table or set(cells[0]) <= set("-: "):
|
|
continue
|
|
first = cells[0].split()[0].lower() if cells[0].split() else ""
|
|
if ROSTER_HANDLE_RE.match(first):
|
|
out.add(first)
|
|
return out
|
|
|
|
|
|
def roster_from_mailmap(text):
|
|
"""Handles from the canonical side of `.mailmap`: `Name <mail> other <mail>`."""
|
|
out = set()
|
|
for line in text.splitlines():
|
|
s = line.strip()
|
|
if not s or s.startswith("#") or "<" not in s:
|
|
continue
|
|
first = s.split("<")[0].split()
|
|
if first and ROSTER_HANDLE_RE.match(first[0].lower()):
|
|
out.add(first[0].lower())
|
|
return out
|
|
|
|
|
|
def load_roster(repo, override):
|
|
"""(handles, source). The roster exists only to tell a person's name from an
|
|
ordinary word after `to` -- `to this host` must not read as an addressee."""
|
|
if override:
|
|
hs = {h.strip().lower() for h in override.split(",") if h.strip()}
|
|
if hs:
|
|
return hs, "--roster"
|
|
out, sources = set(), []
|
|
for fname, fn in (("WIZARDS-TABLE.md", roster_from_table),
|
|
(".mailmap", roster_from_mailmap)):
|
|
path = os.path.join(repo, fname)
|
|
if not os.path.exists(path):
|
|
continue
|
|
try:
|
|
with open(path, encoding="utf-8") as fh:
|
|
got = fn(fh.read())
|
|
except (OSError, UnicodeDecodeError):
|
|
continue
|
|
if got:
|
|
out |= got
|
|
sources.append(fname)
|
|
return out, "+".join(sources) or "(none)"
|
|
|
|
|
|
def _recipient_matches(named, wanted):
|
|
"""Prefix match on a '-' boundary, both directions (a heading may be
|
|
narrower than the filename or fuller than it)."""
|
|
named = named.lower()
|
|
wanted = wanted.lower()
|
|
return (named == wanted or named.startswith(wanted + "-")
|
|
or wanted.startswith(named + "-"))
|
|
|
|
|
|
def _known(cand, roster):
|
|
return any(_recipient_matches(cand, handle) for handle in roster)
|
|
|
|
|
|
def heading_addressee(heading, roster):
|
|
"""The person a heading says the file is addressed to, or None.
|
|
|
|
Two forms are read: `to <name>`, and the arrow form `<author> → <name>`
|
|
(whose addressee is the arrow's target). A candidate counts only if it is a
|
|
roster handle -- otherwise `to this host` would read as an addressee named
|
|
"this", which is how a naive matcher invents a contradiction.
|
|
"""
|
|
for m in HEADING_ARROW.finditer(heading):
|
|
cand = m.group(1).lower()
|
|
if _known(cand, roster):
|
|
return cand
|
|
for m in HEADING_TO.finditer(heading):
|
|
cand = m.group(1).lower()
|
|
if _known(cand, roster):
|
|
return cand
|
|
return None
|
|
|
|
|
|
def read_heading(path, head_lines):
|
|
"""The file's first `# ` heading inside the first `head_lines` lines, or
|
|
None. Blank lines are skipped; a non-blank line before any heading ends the
|
|
search (a file that opens with prose has no heading where one belongs).
|
|
False means the file could not be decoded as UTF-8 at all."""
|
|
try:
|
|
with open(path, encoding="utf-8") as fh:
|
|
for i, line in enumerate(fh):
|
|
if i >= head_lines:
|
|
return None
|
|
s = line.strip()
|
|
if not s:
|
|
continue
|
|
return s if s.startswith("# ") else None
|
|
except UnicodeDecodeError:
|
|
return False
|
|
except OSError:
|
|
return None
|
|
return None
|
|
|
|
|
|
def scan(dirpath, head_lines=10, roster=()):
|
|
"""(rows, faults, advisories, series, holes) for one replies directory.
|
|
|
|
rows: one dict per file, in sorted filename order, each with `file`,
|
|
`date`, `recipient`, `number` (None on both unnumbered forms), `heading`,
|
|
and `classes` (a list). Nothing is written.
|
|
"""
|
|
rows = []
|
|
for name in sorted(os.listdir(dirpath)):
|
|
if not name.endswith(".md") or name == "README.md":
|
|
continue
|
|
path = os.path.join(dirpath, name)
|
|
if not os.path.isfile(path):
|
|
continue
|
|
row = {"file": name, "date": None, "recipient": None, "number": None,
|
|
"heading": None, "classes": []}
|
|
m = NAME_RE.match(name)
|
|
legacy = interview = False
|
|
if m:
|
|
y, mo, d, row["recipient"], num = m.groups()
|
|
row["number"] = int(num)
|
|
else:
|
|
m = LEGACY_RE.match(name)
|
|
if m:
|
|
y, mo, d, row["recipient"] = m.groups()
|
|
legacy = True
|
|
else:
|
|
m = INTERVIEW_RE.match(name)
|
|
if not m:
|
|
row["classes"].append("NAME_MALFORMED")
|
|
rows.append(row)
|
|
continue
|
|
y, mo, d, row["recipient"] = m.groups()
|
|
interview = True
|
|
row["date"] = "%s-%s-%s" % (y, mo, d)
|
|
if not _real_date(y, mo, d):
|
|
row["classes"].append("BAD_DATE")
|
|
if legacy:
|
|
row["classes"].append("LEGACY_UNNUMBERED")
|
|
if interview:
|
|
row["classes"].append("INTERVIEW_UNNUMBERED")
|
|
heading = read_heading(path, head_lines)
|
|
unreadable = heading is False
|
|
row["heading"] = heading if isinstance(heading, str) else None
|
|
if unreadable:
|
|
row["classes"].append("UNREADABLE")
|
|
elif heading is None:
|
|
row["classes"].append("NO_HEADER")
|
|
else:
|
|
named = heading_addressee(heading, roster)
|
|
if named is None:
|
|
row["classes"].append("NO_RECIPIENT_IN_HEADING")
|
|
elif not _recipient_matches(named, row["recipient"]):
|
|
row["classes"].append("HEADER_RECIPIENT")
|
|
if row["number"] is not None:
|
|
hnum = HEADING_NUM.search(heading)
|
|
if hnum is not None:
|
|
if int(hnum.group(1)) != row["number"]:
|
|
row["classes"].append("HEADER_NUMBER")
|
|
elif not re.search(r"(?<!\d)%d(?!\d)" % row["number"], heading):
|
|
# No "Reply N", and the filename's N is not in the heading
|
|
# in any form: the heading is silent on the number.
|
|
row["classes"].append("NO_NUMBER_IN_HEADING")
|
|
rows.append(row)
|
|
|
|
series = {}
|
|
for row in rows:
|
|
if row["number"] is None or "NAME_MALFORMED" in row["classes"]:
|
|
continue
|
|
series.setdefault(row["recipient"], []).append(row["number"])
|
|
holes = {}
|
|
for who, nums in series.items():
|
|
dup = sorted({n for n in nums if nums.count(n) > 1})
|
|
for n in dup:
|
|
for row in rows:
|
|
if row["number"] == n and row["recipient"] == who:
|
|
row["classes"].append("DUPLICATE_NUMBER")
|
|
missing = [n for n in range(min(nums), max(nums) + 1) if n not in nums]
|
|
if missing:
|
|
holes[who] = missing
|
|
|
|
faults = [r for r in rows if any(c in FAULTS for c in r["classes"])]
|
|
advisories = [r for r in rows if any(c in ADVISORIES for c in r["classes"])]
|
|
return rows, faults, advisories, series, holes
|
|
|
|
|
|
# ---------------------------------------------------------------- baseline
|
|
|
|
BASELINE_CLASSES = set(FAULTS)
|
|
|
|
|
|
def emit_baseline(rows):
|
|
"""The ledger as text: one `<filename>\\t<CLASS,CLASS>` row per faulting file.
|
|
|
|
Advisory classes are left out on purpose: they are never rc 1, so a ledger
|
|
that named them would read as if they had been tolerated out of a grade
|
|
they never had. Sorted, so an unchanged archive regenerates byte-identical.
|
|
"""
|
|
head = ("# reply baseline -- the faults this archive already carried when the\n"
|
|
"# heading rule was written. One row per file: <filename>\\t<CLASS[,CLASS]>.\n"
|
|
"# Regenerate: reply_check.py --repo . --emit-baseline > tools/reply_baseline.tsv\n"
|
|
"# A row is a stated tolerance, not an exemption: a file that gains a class,\n"
|
|
"# and a file that is not listed, both make rc 1 under --baseline.\n")
|
|
lines = []
|
|
for row in sorted(rows, key=lambda r: r["file"]):
|
|
cls = sorted(c for c in row["classes"] if c in BASELINE_CLASSES)
|
|
if cls:
|
|
lines.append("%s\t%s" % (row["file"], ",".join(cls)))
|
|
return head + "\n".join(lines) + ("\n" if lines else "")
|
|
|
|
|
|
def load_baseline(path):
|
|
"""Read the ledger -> (ledger, None), or (None, reason) when it cannot be read.
|
|
|
|
Same contract as `checkin_baseline.tsv`: an unreadable, non-UTF-8, wrongly
|
|
shaped or unknown-class row is a refusal, because grading against an empty
|
|
ledger would call the whole archive new and skipping the check would call
|
|
it fine -- both are silent lies about the same fact.
|
|
"""
|
|
try:
|
|
with open(path, "r", encoding="utf-8") as fh:
|
|
text = fh.read()
|
|
except OSError as exc:
|
|
return None, "cannot grade: baseline %s: %s" % (path, exc)
|
|
except UnicodeDecodeError as exc:
|
|
return None, "cannot grade: baseline %s is not UTF-8: %s" % (path, exc)
|
|
out = {}
|
|
for n, line in enumerate(text.splitlines(), 1):
|
|
s = line.rstrip("\r")
|
|
if not s.strip() or s.lstrip().startswith("#"):
|
|
continue
|
|
parts = s.split("\t")
|
|
if len(parts) != 2:
|
|
return None, ("cannot grade: baseline %s:%d is not "
|
|
"<filename>\\t<CLASS[,CLASS]>: %r" % (path, n, s))
|
|
name = parts[0].strip()
|
|
cls = [c.strip() for c in parts[1].split(",") if c.strip()]
|
|
bad = [c for c in cls if c not in BASELINE_CLASSES]
|
|
if not name or not cls or bad:
|
|
return None, ("cannot grade: baseline %s:%d names an empty filename, "
|
|
"an empty class list, or an unknown class %s: %r"
|
|
% (path, n, bad, s))
|
|
out[name] = set(cls)
|
|
return out, None
|
|
|
|
|
|
def apply_baseline(rows, ledger, path=""):
|
|
"""Split every row's graded classes into known and new, and count the stale."""
|
|
current = {r["file"]: set(c for c in r["classes"] if c in BASELINE_CLASSES)
|
|
for r in rows}
|
|
known_files = 0
|
|
for row in rows:
|
|
cls = current[row["file"]]
|
|
known = ledger.get(row["file"], set())
|
|
new = sorted(cls - known)
|
|
row["baseline"] = {"known": sorted(cls & known), "new": new}
|
|
if cls and not new:
|
|
known_files += 1
|
|
stale = sorted(f for f in ledger if f not in current)
|
|
cleared = sorted(f for f in ledger
|
|
if f in current and (ledger[f] - current[f]))
|
|
return {"path": path, "entries": len(ledger), "known_files": known_files,
|
|
"new_faults": sum(1 for r in rows if r["baseline"]["new"]),
|
|
"stale": stale, "cleared": cleared}
|
|
|
|
|
|
def report(path, rows, faults, advisories, series, holes, roster_source, as_json=None,
|
|
ledger=None):
|
|
# DRY -- one pass over `rows` builds BOTH the per-class tally (Counter) and
|
|
# the per-class row groups (defaultdict(list)). The previous code made a full
|
|
# scan for the tally and a SEPARATE full scan for EACH of the 12 classes in
|
|
# FAULTS+ADVISORIES (13 scans of `rows`). Counter + defaultdict collapses
|
|
# that to one pass; rows are grouped in filename order (the order scan()
|
|
# produced them), so the rendered output and the JSON `counts` are
|
|
# byte-identical to the per-class `for row in rows` sweep.
|
|
counts = Counter()
|
|
rows_by_class = defaultdict(list)
|
|
for row in rows:
|
|
for c in row["classes"]:
|
|
counts[c] += 1
|
|
rows_by_class[c].append(row)
|
|
print("reply_check -- %d file(s) under %s (roster: %s)"
|
|
% (len(rows), path, roster_source))
|
|
for cls in FAULTS + ADVISORIES:
|
|
n = counts.get(cls, 0)
|
|
if n:
|
|
print(" %-24s %d" % (cls, n))
|
|
for row in rows_by_class.get(cls, []):
|
|
extra = ""
|
|
if row["heading"]:
|
|
extra = " | heading: %s" % row["heading"][:96]
|
|
print(" %s%s" % (row["file"], extra))
|
|
else:
|
|
print(" %-24s 0" % cls)
|
|
print("series: %d recipient(s), %d numbered file(s), %d hole(s)%s"
|
|
% (len(series), sum(len(v) for v in series.values()), len(holes),
|
|
(": " + ", ".join("%s %s" % (k, v) for k, v in sorted(holes.items())))
|
|
if holes else ""))
|
|
print("faults: %d advisories: %d" % (len(faults), len(advisories)))
|
|
if ledger is not None:
|
|
print("baseline %s: %d entries, %d file(s) known-faulting, "
|
|
"%d new fault(s), %d stale, %d cleared"
|
|
% (ledger["path"], ledger["entries"], ledger["known_files"],
|
|
ledger["new_faults"], len(ledger["stale"]),
|
|
len(ledger["cleared"])))
|
|
for f in ledger["stale"]:
|
|
print(" STALE BASELINE %s (no longer in the directory: the record "
|
|
"was repaired)" % f)
|
|
for f in ledger["cleared"]:
|
|
print(" STALE BASELINE %s (no longer carries every class the "
|
|
"ledger names)" % f)
|
|
if as_json:
|
|
try:
|
|
with open(as_json, "w", encoding="utf-8") as fh:
|
|
json.dump({"path": path, "files": len(rows), "counts": counts,
|
|
"holes": holes, "rows": rows, "baseline": ledger},
|
|
fh, indent=2, sort_keys=True)
|
|
fh.write("\n")
|
|
except OSError as exc:
|
|
print("cannot write --json " + as_json + ": " + str(exc),
|
|
file=sys.stderr)
|
|
return 2
|
|
print("json: %s" % as_json)
|
|
if ledger is not None:
|
|
bad = [(r["file"], ",".join(r["baseline"]["new"])) for r in rows
|
|
if r["baseline"]["new"]]
|
|
print("reply_check: %d file(s), %d new fault(s)%s, %d known-faulting in "
|
|
"the baseline"
|
|
% (len(rows), len(bad),
|
|
(": " + ", ".join("%s (%s)" % b for b in bad)) if bad else "",
|
|
ledger["known_files"]))
|
|
if ledger["new_faults"]:
|
|
print("FAIL: %d file(s) carry a class the baseline does not tolerate"
|
|
% ledger["new_faults"])
|
|
return 1 if ledger["new_faults"] else 0
|
|
if faults:
|
|
print("FAIL: %d file(s) violate the reply heading contract" % len(faults))
|
|
return 1
|
|
return 0
|
|
|
|
|
|
def _write(d, name, text):
|
|
with open(os.path.join(d, name), "w", encoding="utf-8") as fh:
|
|
fh.write(text)
|
|
|
|
|
|
def selftest():
|
|
"""Fixture cells. Each cell builds a directory, runs scan()/main(), and
|
|
asserts the class list AND the exit code -- both directions, so a rule that
|
|
stops firing and a rule that fires everywhere both fail."""
|
|
cells = []
|
|
|
|
ROSTER = {"volt", "merlin", "timmy", "zephyrus"}
|
|
ROSTER_ARG = "volt,merlin,timmy,zephyrus"
|
|
|
|
def cell(label, ok, detail=None):
|
|
cells.append((label, bool(ok), detail))
|
|
|
|
def classes(d, name):
|
|
rows, _, _, _, _ = scan(d, roster=ROSTER)
|
|
for row in rows:
|
|
if row["file"] == name:
|
|
return row["classes"]
|
|
return None
|
|
|
|
def rc(argv):
|
|
# The fixtures' own reports are noise here; the cell is the verdict.
|
|
with contextlib.redirect_stdout(io.StringIO()), \
|
|
contextlib.redirect_stderr(io.StringIO()):
|
|
try:
|
|
return run(argv + ["--roster", ROSTER_ARG])
|
|
except SystemExit as e: # pragma: no cover - defensive
|
|
return e.code
|
|
|
|
root = tempfile.mkdtemp(prefix="reply-selftest-")
|
|
|
|
# 1 clean numbered file, heading agrees in name and number
|
|
d = os.path.join(root, "clean")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-01-to-zephyrus-02.md", "# Reply 2 -- to Zephyrus\n\nbody\n")
|
|
_write(d, "README.md", "not a reply\n")
|
|
cell("clean file has no class", classes(d, "2026-10-01-to-zephyrus-02.md") == [])
|
|
cell("README.md is not a row", len(scan(d)[0]) == 1)
|
|
cell("clean run exits 0", rc(["--dir", d]) == 0)
|
|
|
|
# 2 header names the other party -- the money rule
|
|
d = os.path.join(root, "wrongparty")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-02-to-timmy-58.md",
|
|
"# Reply 58 -- to Merlin (Helm lane), Run 33\n\nbody\n")
|
|
cell("wrong addressee is HEADER_RECIPIENT",
|
|
classes(d, "2026-10-02-to-timmy-58.md") == ["HEADER_RECIPIENT"])
|
|
cell("wrong addressee exits 1", rc(["--dir", d]) == 1)
|
|
|
|
# 3 heading's number disagrees with the filename's
|
|
d = os.path.join(root, "wrongnum")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-02-to-merlin-07.md", "# Reply 6 -- to Merlin\n\nbody\n")
|
|
cell("wrong number is HEADER_NUMBER",
|
|
classes(d, "2026-10-02-to-merlin-07.md") == ["HEADER_NUMBER"])
|
|
cell("wrong number exits 1", rc(["--dir", d]) == 1)
|
|
|
|
# 4 a hyphenated handle in the heading still matches its prefix
|
|
d = os.path.join(root, "handle")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-03-to-zephyrus-58.md",
|
|
"# 2026-10-03 -- to zephyrus-sparkborn (reply 58)\n\nbody\n")
|
|
cell("hyphenated handle matches", classes(d, "2026-10-03-to-zephyrus-58.md") == [])
|
|
|
|
# 5 duplicate number within one recipient's series
|
|
d = os.path.join(root, "dup")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-04-to-volt-03.md", "# Reply 3 -- to Volt\n")
|
|
_write(d, "2026-10-05-to-volt-03.md", "# Reply 3 -- to Volt\n")
|
|
got = classes(d, "2026-10-05-to-volt-03.md")
|
|
cell("duplicate number is DUPLICATE_NUMBER", got == ["DUPLICATE_NUMBER"])
|
|
cell("duplicate exits 1", rc(["--dir", d]) == 1)
|
|
|
|
# 6 a hole is named, never graded
|
|
d = os.path.join(root, "hole")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-04-to-volt-03.md", "# Reply 3 -- to Volt\n")
|
|
_write(d, "2026-10-05-to-volt-05.md", "# Reply 5 -- to Volt\n")
|
|
_, faults, _, _, holes = scan(d)
|
|
cell("hole is reported with its number", holes == {"volt": [4]})
|
|
cell("hole is not a fault", not faults and rc(["--dir", d]) == 0)
|
|
|
|
# 7 the one legacy unnumbered form is advisory, not a fault
|
|
d = os.path.join(root, "legacy")
|
|
os.makedirs(d)
|
|
_write(d, "2026-09-24-to-zephyrus.md", "# 2026-09-24 -- to zephyrus\n\nbody\n")
|
|
cell("legacy form is advisory",
|
|
classes(d, "2026-09-24-to-zephyrus.md") == ["LEGACY_UNNUMBERED"])
|
|
cell("legacy form exits 0", rc(["--dir", d]) == 0)
|
|
|
|
# 7b the reported-interview form: no number by construction, so the
|
|
# numbering rules skip it -- but everything a name can still contradict
|
|
# (its date, its addressee) is graded exactly as on a numbered reply.
|
|
d = os.path.join(root, "interview")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-19-to-zephyrus-interview.md",
|
|
"# Interview to zephyrus -- 2026-10-19\n\nbody\n")
|
|
cell("interview form is advisory, and only that",
|
|
classes(d, "2026-10-19-to-zephyrus-interview.md") == ["INTERVIEW_UNNUMBERED"])
|
|
cell("interview form exits 0", rc(["--dir", d]) == 0)
|
|
_, _, _, series_i, holes_i = scan(d)
|
|
cell("interview form contributes no number to any series",
|
|
series_i == {} and holes_i == {})
|
|
|
|
d = os.path.join(root, "interview-wrongparty")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-19-to-zephyrus-interview.md",
|
|
"# Interview to Merlin -- 2026-10-19\n")
|
|
cell("interview heading still grades its addressee",
|
|
classes(d, "2026-10-19-to-zephyrus-interview.md") ==
|
|
["INTERVIEW_UNNUMBERED", "HEADER_RECIPIENT"])
|
|
cell("interview with the wrong addressee exits 1", rc(["--dir", d]) == 1)
|
|
|
|
d = os.path.join(root, "interview-baddate")
|
|
os.makedirs(d)
|
|
_write(d, "2026-02-30-to-zephyrus-interview.md", "# Interview to zephyrus\n")
|
|
cell("interview form still grades its date",
|
|
classes(d, "2026-02-30-to-zephyrus-interview.md") ==
|
|
["BAD_DATE", "INTERVIEW_UNNUMBERED"])
|
|
|
|
d = os.path.join(root, "interview-nearmiss")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-19-to-zephyrus-inteview.md", "# Interview to zephyrus\n")
|
|
cell("a misspelled interview slug is still NAME_MALFORMED",
|
|
classes(d, "2026-10-19-to-zephyrus-inteview.md") == ["NAME_MALFORMED"])
|
|
cell("a misspelled interview slug exits 1", rc(["--dir", d]) == 1)
|
|
|
|
# 8 a real-calendar check, not a shape check
|
|
d = os.path.join(root, "baddate")
|
|
os.makedirs(d)
|
|
_write(d, "2026-02-30-to-volt-04.md", "# Reply 4 -- to Volt\n")
|
|
cell("impossible date is BAD_DATE",
|
|
classes(d, "2026-02-30-to-volt-04.md") == ["BAD_DATE"])
|
|
|
|
# 9 no heading where one belongs
|
|
d = os.path.join(root, "noheader")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-06-to-volt-06.md", "Reply 6 -- to Volt\n\nbody\n")
|
|
cell("prose before any heading is NO_HEADER",
|
|
classes(d, "2026-10-06-to-volt-06.md") == ["NO_HEADER"])
|
|
|
|
# 10 a name that is neither form
|
|
d = os.path.join(root, "badname")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-06-to-volt-6.md", "# Reply 6 -- to Volt\n")
|
|
cell("unpadded number is NAME_MALFORMED",
|
|
classes(d, "2026-10-06-to-volt-6.md") == ["NAME_MALFORMED"])
|
|
cell("NAME_MALFORMED exits 1", rc(["--dir", d]) == 1)
|
|
|
|
# 11 an early heading that names nobody: advisory, never a fault
|
|
d = os.path.join(root, "norecipient")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-11-to-merlin-14.md",
|
|
"# Reply 14 -- Q31 (Run 6) picked up from this host\n")
|
|
cell("heading naming nobody is advisory",
|
|
classes(d, "2026-10-11-to-merlin-14.md") == ["NO_RECIPIENT_IN_HEADING"])
|
|
cell("advisory exits 0", rc(["--dir", d]) == 0)
|
|
|
|
# 12 bytes that are not UTF-8: a fault with the decoding named, never a
|
|
# silent "no heading"
|
|
d = os.path.join(root, "binary")
|
|
os.makedirs(d)
|
|
with open(os.path.join(d, "2026-10-07-to-volt-07.md"), "wb") as fh:
|
|
fh.write(b"# Reply 7 \xff\xfe to Volt\n")
|
|
cell("non-UTF-8 bytes are UNREADABLE",
|
|
classes(d, "2026-10-07-to-volt-07.md") == ["UNREADABLE"])
|
|
|
|
# 13 the read contract: a path that cannot be read is rc 2, never 0/1
|
|
cell("missing directory exits 2",
|
|
rc(["--dir", os.path.join(root, "nope")]) == 2)
|
|
_write(root, "afile", "not a directory\n")
|
|
cell("a file where a directory belongs exits 2",
|
|
rc(["--dir", os.path.join(root, "afile")]) == 2)
|
|
|
|
# 14-25: the baseline ledger. The live archive carries one fault in another
|
|
# lane's file, so the gate has to be able to state that tolerance without
|
|
# hiding a new fault: every cell here reads the tolerant side AND the side
|
|
# that must still fail.
|
|
d = os.path.join(root, "baseline")
|
|
os.makedirs(d)
|
|
_write(d, "2026-10-02-to-timmy-58.md", "# Reply 58 -- to Merlin (Helm lane)\n")
|
|
_write(d, "2026-10-03-to-merlin-07.md", "# Reply 7 -- to Merlin\n")
|
|
rows_b, _, _, _, _ = scan(d, roster=ROSTER)
|
|
led_text = emit_baseline(rows_b)
|
|
cell("emit names the faulting file and its class",
|
|
"2026-10-02-to-timmy-58.md\tHEADER_RECIPIENT" in led_text)
|
|
cell("emit leaves the clean file out",
|
|
"2026-10-03-to-merlin-07.md" not in led_text)
|
|
bl = os.path.join(root, "reply.tsv")
|
|
with open(bl, "w", encoding="utf-8") as fh:
|
|
fh.write(led_text)
|
|
ledger_, why_ = load_baseline(bl)
|
|
cell("the emitted ledger round-trips",
|
|
ledger_ == {"2026-10-02-to-timmy-58.md": {"HEADER_RECIPIENT"}}
|
|
and why_ is None, (ledger_, why_))
|
|
with contextlib.redirect_stdout(io.StringIO()):
|
|
rv_known = run(["--dir", d, "--baseline", bl, "--roster", ROSTER_ARG])
|
|
cell("a known fault under its ledger exits 0", rv_known == 0)
|
|
buf = io.StringIO()
|
|
with contextlib.redirect_stdout(buf):
|
|
run(["--dir", d, "--baseline", bl, "--roster", ROSTER_ARG])
|
|
printed = buf.getvalue()
|
|
cell("the tolerated fault is still printed, never hidden",
|
|
"HEADER_RECIPIENT" in printed and "1 known-faulting" in printed,
|
|
printed[-160:])
|
|
|
|
# the same file gains a class the ledger does not name -> rc 1, and the new
|
|
# class is named rather than the whole row re-graded
|
|
_write(d, "2026-10-02-to-timmy-58.md", "# Reply 9 -- to Merlin (Helm lane)\n")
|
|
rows_new, _, _, _, _ = scan(d)
|
|
led_new = apply_baseline(rows_new, load_baseline(bl)[0], bl)
|
|
cell("a new class on a known file is the only new fault",
|
|
led_new["new_faults"] == 1
|
|
and [r["baseline"]["new"] for r in rows_new
|
|
if r["baseline"]["new"]] == [["HEADER_NUMBER"]],
|
|
led_new)
|
|
with contextlib.redirect_stdout(io.StringIO()):
|
|
rc_grown = run(["--dir", d, "--baseline", bl, "--roster", ROSTER_ARG])
|
|
cell("a known file that gains a class exits 1", rc_grown == 1)
|
|
|
|
# a faulting file the ledger does not list at all -> rc 1
|
|
_write(d, "2026-10-02-to-timmy-58.md", "# Reply 58 -- to Merlin (Helm lane)\n")
|
|
_write(d, "2026-10-04-to-volt-01.md", "# Reply 1 -- to Timmy\n")
|
|
with contextlib.redirect_stdout(io.StringIO()):
|
|
rc_unlisted = run(["--dir", d, "--baseline", bl, "--roster", ROSTER_ARG])
|
|
cell("an unlisted faulting file exits 1", rc_unlisted == 1)
|
|
|
|
# without a ledger the run still names its verdict in one line, so a reader
|
|
# (and verify.py's detail extractor) gets a reason, not just a table
|
|
buf = io.StringIO()
|
|
with contextlib.redirect_stdout(buf):
|
|
run(["--dir", d, "--roster", ROSTER_ARG])
|
|
cell("a faulting run prints a FAIL line", "FAIL: " in buf.getvalue(),
|
|
buf.getvalue()[-160:])
|
|
|
|
# a ledger entry whose file is gone: printed STALE, never rc 1
|
|
ds = os.path.join(root, "baseline-stale")
|
|
os.makedirs(ds)
|
|
_write(ds, "2026-10-03-to-merlin-07.md", "# Reply 7 -- to Merlin\n")
|
|
bls = os.path.join(root, "stale.tsv")
|
|
with open(bls, "w", encoding="utf-8") as fh:
|
|
fh.write("2026-10-09-to-merlin-99.md\tHEADER_RECIPIENT\n")
|
|
fh.write("2026-10-03-to-merlin-07.md\tHEADER_NUMBER\n")
|
|
buf = io.StringIO()
|
|
with contextlib.redirect_stdout(buf):
|
|
rc_stale = run(["--dir", ds, "--baseline", bls, "--roster", ROSTER_ARG])
|
|
stale_out = buf.getvalue()
|
|
cell("a stale ledger entry is reported and never rc 1",
|
|
rc_stale == 0 and "2026-10-09-to-merlin-99.md" in stale_out,
|
|
stale_out[-200:])
|
|
cell("a cleared class is reported as stale, not as clean",
|
|
"no longer carries every class" in stale_out, stale_out[-200:])
|
|
|
|
# a ledger that cannot be read is rc 2, never a pass
|
|
cell("a missing ledger exits 2",
|
|
rc(["--dir", ds, "--baseline", os.path.join(root, "absent.tsv")]) == 2)
|
|
bad1 = os.path.join(root, "bad-nosep.tsv")
|
|
with open(bad1, "w", encoding="utf-8") as fh:
|
|
fh.write("2026-10-09-to-merlin-99.md HEADER_RECIPIENT\n")
|
|
cell("a ledger line without the tab exits 2",
|
|
rc(["--dir", ds, "--baseline", bad1]) == 2)
|
|
bad2 = os.path.join(root, "bad-class.tsv")
|
|
with open(bad2, "w", encoding="utf-8") as fh:
|
|
fh.write("2026-10-09-to-merlin-99.md\tNOT_A_CLASS\n")
|
|
cell("a ledger naming an unknown class exits 2",
|
|
rc(["--dir", ds, "--baseline", bad2]) == 2)
|
|
|
|
# 38-41: the docstring's own census. The two spelled counts live only in
|
|
# prose, so nothing graded them until this cell; the enumeration beside each
|
|
# one is the copy a reader follows, and FAULTS/ADVISORIES are what the code
|
|
# grades. A class added to either table without its docstring row, or a
|
|
# count word left behind, fails here rather than in the next reader's quote.
|
|
graded_w, advisory_w = doc_counts()
|
|
cell("docstring graded-count word matches FAULTS",
|
|
graded_w == len(FAULTS),
|
|
"docstring says %r; FAULTS holds %d" % (graded_w, len(FAULTS)))
|
|
cell("docstring advisory-count word matches ADVISORIES",
|
|
advisory_w == len(ADVISORIES),
|
|
"docstring says %r; ADVISORIES holds %d" % (advisory_w, len(ADVISORIES)))
|
|
unnamed = [c for c in FAULTS + ADVISORIES if c not in (__doc__ or "")]
|
|
cell("every class the code grades is named in the docstring",
|
|
not unnamed, unnamed)
|
|
# the instrument itself, against the words this file carried before the fix
|
|
prefix_doc = (" file says about itself. Five classes are graded:\n"
|
|
" Three more are counted and printed, and are ADVISORY.\n")
|
|
cell("the pre-fix count words are refused",
|
|
doc_counts(prefix_doc) != (len(FAULTS), len(ADVISORIES)),
|
|
doc_counts(prefix_doc))
|
|
|
|
failed = [(label, detail) for label, ok, detail in cells if not ok]
|
|
print("selftest: %d cells, %d passed, %d failed" %
|
|
(len(cells), len(cells) - len(failed), len(failed)))
|
|
for label, detail in failed:
|
|
print(" FAILED: %s" % label)
|
|
if detail is not None:
|
|
print(" got: %r" % (detail,))
|
|
return 1 if failed else 0
|
|
|
|
|
|
def run(argv):
|
|
ap = argparse.ArgumentParser(
|
|
description="grade every reply file's heading against its own filename")
|
|
ap.add_argument("--repo", default=".",
|
|
help="the repository the roster is read from (default: .)")
|
|
ap.add_argument("--dir", default=None,
|
|
help="the replies directory (default: <repo>/replies)")
|
|
ap.add_argument("--roster", default=None,
|
|
help="comma-separated handles, overriding WIZARDS-TABLE.md and .mailmap")
|
|
ap.add_argument("--head-lines", type=int, default=10,
|
|
help="how many leading lines may hold the heading (default 10)")
|
|
ap.add_argument("--json", help="write the full row report here")
|
|
ap.add_argument("--baseline", default=None,
|
|
help="known-fault ledger (lines of <filename>\\t<CLASS[,CLASS]>): "
|
|
"only a class the ledger does not name on that file is rc 1")
|
|
ap.add_argument("--emit-baseline", action="store_true",
|
|
help="print the ledger for this directory and exit 0")
|
|
args = ap.parse_args(argv)
|
|
dirpath = args.dir if args.dir is not None else os.path.join(args.repo, "replies")
|
|
if not os.path.exists(dirpath):
|
|
print("reply_check: cannot read replies: %s: no such directory" % dirpath,
|
|
file=sys.stderr)
|
|
return 2
|
|
if not os.path.isdir(dirpath):
|
|
print("reply_check: cannot read replies: %s: not a directory" % dirpath,
|
|
file=sys.stderr)
|
|
return 2
|
|
roster, source = load_roster(args.repo, args.roster)
|
|
rows, faults, advisories, series, holes = scan(dirpath, args.head_lines, roster)
|
|
if args.emit_baseline:
|
|
sys.stdout.write(emit_baseline(rows))
|
|
return 0
|
|
ledger = None
|
|
if args.baseline:
|
|
ledger, why = load_baseline(args.baseline)
|
|
if ledger is None:
|
|
print("reply_check: %s" % why, file=sys.stderr)
|
|
return 2
|
|
ledger = apply_baseline(rows, ledger, args.baseline)
|
|
return report(dirpath, rows, faults, advisories, series, holes, source,
|
|
args.json, ledger)
|
|
|
|
|
|
def main():
|
|
argv = sys.argv[1:]
|
|
if "--selftest" in argv:
|
|
return selftest()
|
|
return run(argv)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
sys.exit(main())
|