tools (volt): the "could not grade" class gets one code -- rc 2 -- and a gate that measures it on every tool

Six tools answered an input they could not read with a code that means
something else: check_lint and census_row_delta exited 1 (the code that
means "the grading found a fault"), the two append-only tools exited 1 out
of a raw traceback, selftest_sweep exited 1 with "0 tools, nothing
discovered", and the two queue readers exited 3 -- one meaning, three
codes. Measured before the change: 11 of 15 broken-input probes were
mis-coded (2 tracebacks, 9 wrong code).

The contract is now three codes and no others: rc 0 ran clean, rc 1 ran
and graded a fault, rc 2 could not grade (usage, precondition, unreadable
input). rc 3 is retired.

And the class is gated, not merely fixed: selftest_sweep.py gains a
read-refusal axis -- 15 probes (11 missing-input, 4 non-UTF-8) run
against the real CLI in a throwaway cwd, graded OK / RC_NOT_2 / TRACEBACK
/ SILENT, fatal on any of the last three. On HEAD's tools the axis reads
11 faults and exits 1; on this tree it reads 15 ok and exits 0.

Also: doc_census_check.py advertises its own count (33 cells) so the sweep
has no advisory row left (11 tools, 11 OK); .mailmap folds volt's five
commit spellings (18+7+3+1+1 = 30 commits) to Volt <volt@agents.invalid>,
the identity 016d26e lost to the repo's Timmy config.

Selftests: 18/15/33/37/54/16/22, 0 failed. Sweep 11 tools, 11 OK, 0 fatal.
queue_index --check rc 0 (49 sections, 45 ids, 9 nested). landing_check
346/346.
This commit is contained in:
volt committed 2026-10-17 22:08:19 +00:00
1 parent 461eede57a
commit 6ee8fcee5a
9 files changed
+238 -35

No files matched your search

+4
View File
@@ -12,3 +12,7 @@ Zephyrus Sparkborn <zephyrus-sparkborn@agents.invalid> <zephyrus-sparkborn@users
Zephyrus Sparkborn <zephyrus-sparkborn@agents.invalid> <zephyrus-sparkborn@users.noreply.github.com>
Zephyrus Sparkborn <zephyrus-sparkborn@agents.invalid> <zephyrus-sparkborn@noreply.local>
Cassian Assayborn <cassian@agents.invalid> <cassian@forge.local>
Volt <volt@agents.invalid> volt <volt@troupe.invalid>
Volt <volt@agents.invalid> volt <volt@troupe.local>
Volt <volt@agents.invalid> Volt <volt@troupe.local>
Volt <volt@agents.invalid> Volt <volt@stackchain.local>
+11 -1
View File
@@ -462,7 +462,17 @@ def main():
a = ap.parse_args()
if a.selftest:
return selftest()
res = audit(a.repo, progress=a.progress)
# A repo that is not there, or is not a git work tree, is a PRECONDITION
# failure: rc 2 with a named message. Before 2026-10-17 this path escaped
# as a FileNotFoundError traceback with rc 1 out of the first `git` call.
if not os.path.isdir(a.repo):
print("cannot read repo: %s is not a directory" % a.repo, file=sys.stderr)
return 2
try:
res = audit(a.repo, progress=a.progress)
except (OSError, RuntimeError, subprocess.SubprocessError) as exc:
print("cannot read repo %s: %s" % (a.repo, exc), file=sys.stderr)
return 2
report(res)
if a.json:
with open(a.json, "w") as f:
+15 -1
View File
@@ -242,7 +242,21 @@ def main():
return selftest()
if not a.audit_json:
ap.error("need AUDIT.json (or --selftest)")
d = json.load(open(a.audit_json))
# A missing, non-UTF-8 or non-audit file is a PRECONDITION failure: rc 2
# with a named message. This line used to be a bare
# `json.load(open(...))`, so all three classes escaped as a traceback with
# rc 1 -- the code this tool never otherwise uses, and the same code the
# repo's graders use for a real fault.
try:
with open(a.audit_json, encoding="utf-8") as fh:
d = json.load(fh)
except (OSError, ValueError) as exc:
print("cannot read audit %s: %s" % (a.audit_json, exc), file=sys.stderr)
return 2
if not isinstance(d, dict) or not isinstance(d.get("rows"), list):
print("not an append_only_audit --json file (no 'rows' list): %s"
% a.audit_json, file=sys.stderr)
return 2
res = adjudicate(d["rows"])
report(res)
if a.json:
+22 -6
View File
@@ -53,6 +53,16 @@ import collections
import json
import sys
class ReadError(Exception):
"""Input this tool could not read or could not recognise: rc 2, never rc 1.
rc 1 is the tool's verdict code (a partition that does not close). A file
that cannot be opened, decoded or parsed has produced no verdict at all, so
it must not borrow the code that means "the grading found a fault".
"""
# A row is identified by a subset of these fields -- the names are the JSON keys.
FIELDS = ("tool", "block", "line", "rev", "unit", "claim", "token")
@@ -77,17 +87,19 @@ def load(path):
`rows` is every surviving verdict dict, each carrying its tool name; keying
happens in `keyed()` so one load serves every key shape.
An unreadable or non-census file is a usage error, not a traceback: this
used to raise the json decoder's error or a bare `KeyError: 'reports'`, with
rc 1. The message names the file and what was expected instead.
An unreadable or non-census file is a PRECONDITION failure, not a traceback
and not a verdict: this used to raise the json decoder's error or a bare
`KeyError: 'reports'`, with rc 1 -- the code that means "the grading found a
fault". `ReadError` is turned into rc 2 in main(); the message names the
file and what was expected instead.
"""
try:
with open(path, encoding="utf-8") as fh:
d = json.load(fh)
except (OSError, ValueError) as exc:
raise SystemExit("cannot read census %s: %s" % (path, exc))
raise ReadError("cannot read census %s: %s" % (path, exc))
if not isinstance(d, dict) or not isinstance(d.get("reports"), list):
raise SystemExit("not a doc_census_check census (no 'reports' list): %s" % path)
raise ReadError("not a doc_census_check census (no 'reports' list): %s" % path)
rows, fails, per = [], [], {}
for rep in d["reports"]:
for v in rep["verdicts"]:
@@ -402,7 +414,11 @@ def main():
print(" %-11s %s" % (name, " + ".join(KEYS[name])))
return 0
before, after = load(a.before), load(a.after)
try:
before, after = load(a.before), load(a.after)
except ReadError as exc:
print(str(exc), file=sys.stderr)
return 2
print(banner("before", before))
print(banner("after", after))
print("verdict totals before: %s" % totals(before))
+8 -4
View File
@@ -515,10 +515,14 @@ def main(argv=None) -> int:
with open(a.log, encoding="utf-8", errors="replace", newline="") as fh:
text = fh.read()
except OSError as exc:
# A log that is not there is a usage error, not a grading failure:
# it used to escape as a FileNotFoundError traceback with rc 1,
# which is the code this tool uses for "the check has a fault".
raise SystemExit("cannot read log: %s" % exc)
# A log that is not there is a PRECONDITION failure, not a grading
# result. It used to escape as a FileNotFoundError traceback with
# rc 1, which is the code this tool uses for "the check has a
# fault"; since 2026-10-17 the repo-wide contract is rc 2 for
# "could not grade" (see tools/selftest_sweep.py's read-refusal
# axis, which now measures this row on every reader).
print(f"cannot read log: {exc}", file=sys.stderr)
return 2
cmd = extract_check(text)
primary, faults, detail = classify(cmd)
if not a.quiet:
+1 -1
View File
@@ -123,7 +123,7 @@ SCOPE, STATED RATHER THAN IMPLIED
USAGE
python3 doc_census_check.py [--repo DIR] [--rev HEAD] [--tools NAME...]
[--json OUT] [--no-run] [--selftest]
[--json OUT] [--no-run] [--selftest] # selftest: 33 cells, one per claim class
"""
import argparse
import ast
+7 -7
View File
@@ -527,7 +527,7 @@ def selftest(verbose=True):
if not ok:
failures.append(name)
# A read failure is rc 3, not a traceback. `--queue` at a missing path or at
# A read failure is rc 2, not a traceback. `--queue` at a missing path or at
# bytes that are not UTF-8 escaped main() as FileNotFoundError /
# UnicodeDecodeError; the first was already caught, the second was not, and
# rc 1 is what this corpus uses for "a check failed". Two cells, because the
@@ -538,22 +538,22 @@ def selftest(verbose=True):
with open(bad, 'wb') as fh:
fh.write(b'\xff\xfe\x80\n')
for name, path, want in (
('read cell 1: a missing queue path exits 3, not a traceback',
('read cell 1: a missing queue path exits 2, not a traceback',
os.path.join(td, 'nope.md'), 'No such file'),
('read cell 2: a queue that is not UTF-8 exits 3, not a traceback',
('read cell 2: a queue that is not UTF-8 exits 2, not a traceback',
bad, 'utf-8')):
n_read += 1
err = io.StringIO()
with contextlib.redirect_stderr(err):
rc = main(['--queue', path])
msg = err.getvalue()
ok = rc == 3 and 'cannot read queue:' in msg
ok = rc == 2 and 'cannot read queue:' in msg
if ok:
ok = want.lower() in msg.lower()
if verbose:
print('%s %s' % ('PASS' if ok else 'FAIL', name))
if not ok:
print(' expected rc 3 and `cannot read queue: ...%s...`\n'
print(' expected rc 2 and `cannot read queue: ...%s...`\n'
' got rc %d, stderr %r' % (want, rc, msg.strip()[:160]))
if not ok:
failures.append(name)
@@ -598,9 +598,9 @@ def main(argv=None):
# UnicodeDecodeError is not an OSError: a binary or mis-encoded file at
# --queue used to escape main() as a traceback with rc 1, which is the
# code the fixture corpus uses for "a check failed". A read failure is
# rc 3 with the same message, whatever the reason.
# rc 2 with the same message, whatever the reason.
print('cannot read queue: %s' % exc, file=sys.stderr)
return 3
return 2
res = census(text)
git_dir = os.path.dirname(os.path.abspath(path))
+9 -9
View File
@@ -489,7 +489,7 @@ def run(args):
# missing path or at bytes that are not UTF-8 raised FileNotFoundError /
# UnicodeDecodeError straight out of run(), so the operator got a stack and
# rc 1 while every other reader in this repo (`queue_claim_check.py:558-563`)
# says `cannot read queue:` and exits 3. The same contract is owed to the
# says `cannot read queue:` and exits 2. The same contract is owed to the
# --check document: a gate that cannot read its input has verified nothing,
# and must not report that as a stale index (rc 1) or a crash.
try:
@@ -497,7 +497,7 @@ def run(args):
text = fh.read()
except (OSError, UnicodeDecodeError) as exc:
print("cannot read queue: %s" % exc, file=sys.stderr)
return 3
return 2
payload = parse(text)
if args.check:
@@ -505,7 +505,7 @@ def run(args):
doc = open(args.check, encoding="utf-8").read()
except (OSError, UnicodeDecodeError) as exc:
print("cannot read index document: %s" % exc, file=sys.stderr)
return 3
return 2
pinned = extract_pinned(doc)
if pinned is None:
print("UNVERIFIABLE: no parseable pinned block in %s" % args.check)
@@ -772,9 +772,9 @@ def selftest():
# --- a read error is an exit code, not a traceback -----------------------
# `--queue` at a missing path or at bytes that are not UTF-8 used to raise
# straight out of run() (FileNotFoundError / UnicodeDecodeError, rc 1). The
# repo's other reader exits 3 with `cannot read queue:`; this tool now does
# repo's other reader exits 2 with `cannot read queue:`; this tool now does
# the same, and owes the --check document the same answer: an unreadable
# gate input is rc 3, never the rc 1 that means "stale".
# gate input is rc 2, never the rc 1 that means "stale".
bad_q = os.path.join(root, "bad-utf8.md")
with open(bad_q, "wb") as fh:
fh.write(b"\xff\xfe\x80\n")
@@ -782,14 +782,14 @@ def selftest():
with contextlib.redirect_stderr(err):
rc = run(argparse.Namespace(repo=root, queue="missing-queue.md", index=False,
json=None, write=None, check=None, outside=False))
check("missing queue -> exit 3", rc, 3)
check("missing queue -> exit 2", rc, 2)
check("missing queue -> says cannot read queue",
err.getvalue().startswith("cannot read queue:"), True)
err = io.StringIO()
with contextlib.redirect_stderr(err):
rc = run(argparse.Namespace(repo=root, queue="bad-utf8.md", index=False,
json=None, write=None, check=None, outside=False))
check("non-UTF-8 queue -> exit 3", rc, 3)
check("non-UTF-8 queue -> exit 2", rc, 2)
check("non-UTF-8 queue -> says cannot read queue",
"cannot read queue:" in err.getvalue(), True)
err = io.StringIO()
@@ -798,14 +798,14 @@ def selftest():
json=None, write=None,
check=os.path.join(repo, "missing-index.md"),
outside=False))
check("missing index document -> exit 3", rc, 3)
check("missing index document -> exit 2", rc, 2)
check("missing index document -> names the document",
"cannot read index document:" in err.getvalue(), True)
err = io.StringIO()
with contextlib.redirect_stderr(err):
rc = run(argparse.Namespace(repo=repo, queue="WORK-QUEUE.md", index=False,
json=None, write=None, check=bad_q, outside=False))
check("non-UTF-8 index document -> exit 3", rc, 3)
check("non-UTF-8 index document -> exit 2", rc, 2)
check("non-UTF-8 index document -> names the document",
"cannot read index document:" in err.getvalue(), True)
+161 -6
View File
@@ -38,6 +38,17 @@ WHAT IT MEASURES -- one row per tracked tool, one verdict per row
FATAL = every class above except NO_CLAIM. Exit 1 if any row is fatal, else 0.
THE READ-REFUSAL AXIS, ADDED 2026-10-17
The same command asks every tool the second question this archive kept
getting wrong: what exit code does it give an input it cannot read? The
contract is three codes and no others -- rc 0 ran clean, rc 1 ran and graded
a fault, rc 2 could not grade. The axis runs each tool against a path that
does not exist (and, for the four file readers, against bytes that are not
UTF-8) and grades the answer OK / RC_NOT_2 / TRACEBACK / SILENT; the last
three are fatal. rc 3 is retired: the two queue readers used it for "cannot
read the queue", so one meaning carried two codes and a stale index (rc 1)
could be confused with a queue that was never read.
WHAT COUNTS AS A CLAIM, PRECISELY
Only the module docstring (read through `ast.get_docstring`, so a fixture
string in the BODY is not a claim -- `doc_census_check.py` contains the literal
@@ -61,7 +72,7 @@ does not decide that a claim SHOULD exist -- NO_CLAIM is advisory by design.
USAGE:
python3 selftest_sweep.py --repo /root/troupe-standup
python3 selftest_sweep.py --tools-dir /tmp/fake --json /tmp/sweep.json
python3 selftest_sweep.py --selftest # 18 cells, parser and sweep
python3 selftest_sweep.py --selftest # 22 cells, parser, sweep and read axis
"""
from __future__ import annotations
@@ -98,6 +109,42 @@ FATAL = (
"UNREADABLE",
)
# --------------------------------------------------------------- read axis
# THE SECOND GATE, ADDED 2026-10-17: the exit code a tool gives an input it
# cannot read. The repo-wide contract is three codes and no others --
# rc 0 ran, nothing to report
# rc 1 ran, and GRADED something as faulty (STALE, TAUTOLOGY, MISMATCH, ...)
# rc 2 COULD NOT GRADE: usage error, precondition unmet, input unreadable
# Before this axis existed the third class was spread over rc 1 (check_lint,
# census_row_delta, both append-only tools as a raw traceback, this sweep) and
# rc 3 (the two queue readers), so "1" meant both "bad verdict" and "no
# verdict" and an operator could not tell them apart. The axis re-measures the
# class on every tool, from the outside, against the real CLI.
READ_AXIS_UNIT = 2
FATAL_READ = ("TRACEBACK", "RC_NOT_2", "SILENT", "NO_TOOL")
NO_PROBE = "NO_PROBE" # advisory: a tool with no probe defined
# {nope} = a path that does not exist; {bin} = a file that is not UTF-8.
READ_PROBES = (
("append_only_audit.py", ["--repo", "{nope}"]),
("append_only_owners.py", ["{nope}/audit.json"]),
("census_row_delta.py", ["{nope}/a.json", "{nope}/b.json"]),
("check_lint.py", ["--log", "{nope}/log.txt"]),
("citation_hash_class.py", ["--repo", "{nope}"]),
("citation_resolve_check.py", ["--repo", "{nope}"]),
("doc_census_check.py", ["--repo", "{nope}", "--no-run"]),
("landing_check.py", ["--repo", "{nope}"]),
("queue_claim_check.py", ["--queue", "{nope}/q.md"]),
("queue_index.py", ["--queue", "{nope}/q.md", "--check", "{nope}/i.md"]),
("selftest_sweep.py", ["--repo", "{nope}"]),
)
NOT_UTF8_PROBES = (
("append_only_owners.py", ["{bin}"]),
("census_row_delta.py", ["{bin}", "{bin}"]),
("queue_claim_check.py", ["--queue", "{bin}"]),
("queue_index.py", ["--queue", "{bin}", "--check", "{bin}"]),
)
def _int(text: str) -> int:
return int(text.replace(",", ""))
@@ -301,6 +348,89 @@ def report(results, source, untracked, quiet=False):
return 1 if fatal else 0
# ------------------------------------------------------- read-refusal axis
def _first_line(text: str) -> str:
for line in (text or "").splitlines():
if line.strip():
return line.strip()[:96]
return ""
def judge_read(rc, text):
"""One probe's verdict. Three classes, in this order on purpose: a
traceback is a defect whatever its exit code, and a silent rc 2 is a
refusal an operator cannot read."""
if "Traceback (most recent call last)" in (text or ""):
return {"verdict": "TRACEBACK", "rc": rc, "detail": _first_line(text)}
if rc != READ_AXIS_UNIT:
return {"verdict": "RC_NOT_2", "rc": rc,
"detail": _first_line(text) or f"rc {rc}, no output"}
if not (text or "").strip():
return {"verdict": "SILENT", "rc": rc, "detail": "rc 2 with no message"}
return {"verdict": "OK", "rc": rc, "detail": _first_line(text)}
def run_read_probe(path: str, argv, timeout: int):
"""(rc, combined output) for one probe, run in a throwaway cwd."""
with tempfile.TemporaryDirectory() as td:
binpath = os.path.join(td, "not-utf8.bin")
with open(binpath, "wb") as fh:
fh.write(b"\xff\xfe\x00binary\x00\n")
nope = os.path.join(td, "there-is-no-such-path")
real = [a.replace("{nope}", nope).replace("{bin}", binpath) for a in argv]
try:
p = subprocess.run([sys.executable, path] + real, cwd=td,
capture_output=True, text=True, errors="replace",
timeout=timeout)
except subprocess.TimeoutExpired:
return None, "<timeout>"
except OSError as exc:
return 127, f"<exec failed: {exc}>"
return p.returncode, (p.stdout or "") + (p.stderr or "")
def read_axis(rows, timeout: int):
"""One row per (tool, probe). Tools the sweep discovered are probed by path;
a probe naming a tool that was not discovered is NO_TOOL (fatal -- the table
and the probes have drifted apart)."""
paths = dict(rows)
out = []
for probe, specs in (("missing", READ_PROBES), ("not-utf8", NOT_UTF8_PROBES)):
for name, argv in specs:
rec: dict = {"tool": name, "probe": probe}
if name not in paths:
rec.update({"verdict": "NO_TOOL", "rc": None,
"detail": "probe names a tool the sweep did not discover"})
else:
rc, text = run_read_probe(paths[name], argv, timeout)
rec.update(judge_read(rc, text))
out.append(rec)
probed = {n for n, _ in READ_PROBES}
for name, _p in rows:
if name not in probed:
out.append({"tool": name, "probe": "missing", "verdict": NO_PROBE,
"rc": None, "detail": "no read probe defined (advisory)"})
return out
def report_read(axis, quiet=False):
fatal = [r for r in axis if r["verdict"] in FATAL_READ]
if not quiet:
print()
print(f"read-refusal axis: an unreadable input exits {READ_AXIS_UNIT}, "
"never 1 (a graded fault)")
width = max([len(r["tool"]) for r in axis] + [4])
for r in axis:
rc = "-" if r["rc"] is None else str(r["rc"])
print(f"{r['tool']:<{width}} {r['probe']:<9} rc={rc:<4} "
f"{r['verdict']:<10} {r['detail']}")
ok = sum(1 for r in axis if r["verdict"] == "OK")
print(f"read-refusal: {len(axis)} probes, {ok} ok, {len(fatal)} fault"
+ (f" ({', '.join(sorted({r['verdict'] for r in fatal}))})" if fatal else ""))
return 1 if fatal else 0
# ------------------------------------------------------------------- selftest
def selftest():
@@ -393,7 +523,21 @@ def selftest():
ok_json = False
cell("c18 --json carries every row and the fatal count", ok_json)
print(f"selftest: 18 cells, {18 - len(bad)} passed, {len(bad)} failed")
# c19-c22: the read-refusal axis's judgement, one cell per class. The axis
# is the gate on "an input I cannot read must not exit 1"; these cells are
# the gate on the gate.
jr = judge_read(2, "cannot read census /nope/a.json: [Errno 2]")
cell("c19 a refusal at rc 2 with a message -> OK", jr["verdict"] == "OK", jr["verdict"])
jr = judge_read(1, "MISSING: the check has a fault")
cell("c20 a graded fault's rc 1 on an unreadable input -> RC_NOT_2",
jr["verdict"] == "RC_NOT_2", jr["verdict"])
jr = judge_read(2, "Traceback (most recent call last):\n File x")
cell("c21 a traceback is fatal even when it exits 2",
jr["verdict"] == "TRACEBACK", jr["verdict"])
jr = judge_read(2, "")
cell("c22 rc 2 with no message -> SILENT", jr["verdict"] == "SILENT", jr["verdict"])
print(f"selftest: 22 cells, {22 - len(bad)} passed, {len(bad)} failed")
return 1 if bad else 0
@@ -417,11 +561,19 @@ def main(argv=None):
rows, source, untracked = discover(a.repo, a.tools_dir)
if not rows:
print("sweep: 0 tools, nothing discovered")
return 1
# A sweep that discovered nothing has verified nothing. It is not a
# clean run (rc 0) and not a fatal verdict (rc 1): it is rc 2, the
# repo-wide code for "could not grade" -- the same contract the
# read-refusal axis below measures on every other tool.
print("cannot read: no tools discovered in %s (%s)"
% (a.tools_dir or os.path.join(a.repo, "tools"), source),
file=sys.stderr)
return 2
results = sweep(rows, a.repo if not a.tools_dir else os.path.abspath(a.tools_dir),
a.timeout, a.verbose)
rc = report(results, source, untracked, a.quiet)
axis = read_axis(rows, a.timeout)
rrc = report_read(axis, a.quiet)
if a.json:
fatal = [r for r in results if r["verdict"] in FATAL]
@@ -432,8 +584,11 @@ def main(argv=None):
"untracked": untracked,
"tools": results,
"fatal": len(fatal),
"ok": sum(1 for r in results if r["verdict"] == "OK")}, fh, indent=1)
return rc
"ok": sum(1 for r in results if r["verdict"] == "OK"),
"read_axis": axis,
"read_fatal": sum(1 for r in axis
if r["verdict"] in FATAL_READ)}, fh, indent=1)
return 1 if (rc or rrc) else 0
if __name__ == "__main__":