stackchain/troupe-standup
2
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

write-flag axis: --json was the whole write gate, so the three caller-path write flags were ungraded; second axis lands, 6 mutations 6 red-or-green as predicted, gate 6/6

Browse Source 
Measured before: at ea8fe24 a worktree with queue_index's --write guard
narrowed to PermissionError reads 21 probed / 21 ok / rc 0 under that
revision's write_probe. Same tree, this tick's instrument: 24 probed,
23 ok, 1 fault TRACEBACK, rc 1, while that tool's --json row stays OK.

Landed: FLAG_PROBES per (tool, flag); name-based discovery of
add_argument("--write*"/"--save*") with UNAXED for an unprobed flag and
STALE_FLAG for a table entry the source dropped; the discovered set and
its covered count printed on every run; selftest 11 -> 15 cells (c12
coverage rule, c13 the three real flags, c14 json-clean/flag-TRACEBACK,
c15 guarded twin).

Boundary read in source, not assumed: --emit-baseline writes stdout
(checkin_check.py:935, only file write inside c33), check_lint --log is a
read.

Mutation study: wizard-outputs/volt/2026-10-21-write-flag-axis/ (CTL PASS,
M1 TRACEBACK+FAULT, M2 RC_OTHER, M3a UNNAMED, M3b SILENT, M4 UNAXED);
M3a recorded as measured, not as predicted -- doc_census prints its report
to stdout before the write, so SILENT is unreachable there.

verify.py: 6 legs, 6 pass, 0 fault. write_probe --repo .: 24 probed,
24 ok. doc_command_check 64/64. doc_census 0 STALE. TOOLS.md re-rendered,
byte-identical.
This commit is contained in:
volt committed 2026-10-21 17:28:26 +00:00
1 parent ea8fe24563
commit 4602bd59a0
5 files changed
+354 -38
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

tools/write_probe.py
+288 -25
View File
@@ -57,6 +57,50 @@ VERDICTS, IN THIS ORDER (a traceback is a defect whatever the exit code)
`FAULT: append_only_audit.py OK 2 cannot write --json ...` while the tool that
carried the defect was doc_census_check.py.
THE SECOND AXIS -- caller-path write flags other than --json
Until 2026-10-21 this instrument graded `--json` and nothing else. Every tool
that writes a caller-supplied path through a flag of its own
(`queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab`)
was outside the gate: its guard could be deleted and `verify.py` would still
print `gate: OK`. The control plane named that as this lane's next tool change
(reply 46 to volt: "the next tool to add a `--write` is ungraded until the
table grows a second axis"), and the fix is a second axis, not three rows
bolted onto the first.
Discovery is by name and the coverage requirement is derived from it: every
`<parser>.add_argument("--write...")` / `("--save...")` call in a tracked tool
is a caller-path write flag, and every one discovered must have an entry in
FLAG_PROBES. The rule is name-based on purpose, and its boundary is stated
rather than hidden: `--emit-baseline` is `store_true` and writes stdout (its
one file write is inside `checkin_check.py`'s selftest, at a path that
selftest owns), so it is not in the class; a future flag named outside the
`--write`/`--save` families would be missed by discovery -- which is why every
run prints the discovered set with its size, so a short list is visible on the
report instead of absent from it.
MEASURED, not asserted (2026-10-21, six throwaway copies of this clone, one
defect each, verdicts read out of this tool's own --json):
wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py. `CTL` the
clean tree -> the three flag rows OK, 24 probes, rc 0, and `verify.py --fast`
write-axis PASS; `M1` `queue_index --write`'s guard narrowed to
`PermissionError` -> TRACEBACK, probe rc 1, leg FAULT, while that same tool's
`--json` row still reads OK (the axis grades a path the old one could not see);
`M2` the refusal returning 1 -> RC_OTHER; `M3b` the message deleted from a tool
that prints nothing before it writes -> SILENT; `M4` a `--save-extra` flag on a
tracked tool with no table entry -> UNAXED. One prediction was wrong and the
case records the measurement rather than the expectation: `M3a` deletes the
same message from `doc_census_check --save-vocab`, which prints its census
report to stdout before it writes, so the judge still finds text and the class
is UNNAMED, not SILENT -- on that tool the two classes are not interchangeable,
and both are faults.
Two verdicts are this axis's own, and both are faults:
UNAXED a write flag discovered in the source with no FLAG_PROBES entry --
the path exists and nothing probes it (the NO_PROBE rule, applied
to a flag rather than a tool)
STALE_FLAG FLAG_PROBES names a flag the tool no longer defers -- the table
has outlived the code it describes
FIXTURES
Every probe runs in a throwaway tree, so the grade each tool performs is over
a few bytes rather than over this repository: the write path is what is being
@@ -68,8 +112,9 @@ READ-ONLY on the repo: the only files created are under a TemporaryDirectory.
USAGE
python3 tools/write_probe.py [--repo DIR] [--only TOOL] [--timeout S]
[--json PATH] [--verbose]
python3 tools/write_probe.py --selftest # 11 cells, judge, table,
# fault block, fixture
python3 tools/write_probe.py --selftest # 15 cells: judge, table,
# coverage, fault block,
# fixtures, both axes
EXIT
0 every probed tool is OK; 1 at least one is not (including NO_REACH and
NO_JSON: an ungraded write path may not be reported clean); 2 could not
@@ -84,6 +129,19 @@ import sys
import tempfile
JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']")
# A caller-path write flag other than --json. The class is the two name families
# this repo uses for "write the thing you were given a path for"; see THE SECOND
# AXIS above for the boundary and why --emit-baseline is not in it.
#
# The receiver is part of the pattern on purpose: every call site in this repo is
# `<parser>.add_argument("--flag")` on one line (21/21 tools at 2026-10-21), and
# requiring the receiver is what keeps this instrument from discovering its own
# fixtures -- `'ap.add_argument("--write")\n'` inside a selftest cell is a string
# literal at the start of a line, and a match there would have this tool
# reporting itself UNAXED for a flag it does not define.
WRITE_FLAG = re.compile(
r"^[ \t]*[A-Za-z_][A-Za-z0-9_]*\.add_argument\(\s*[\"'](--(?:write|save)(?:-[a-z0-9]+)*)[\"']",
re.M)
TRACEBACK = "Traceback (most recent call last)"
# (tool, argv) -- `{...}` placeholders are filled by fixture_paths(). One entry
@@ -121,6 +179,21 @@ WRITE_PROBES = (
("write_probe.py", ["--repo", "{tinyrepo}", "--json", "{target}"]),
)
# SECOND AXIS: caller-path write flags per tool -> (flag, argv template).
# `{flagtarget}` is a path whose parent does not exist, one per (tool, flag), so
# two tools' rows can never be confused in a fault line. Coverage is REQUIRED and
# discovered from the source (see flag_coverage): a tool that grows a `--write`
# and is not added here is UNAXED -- a fault -- not silently invisible.
FLAG_PROBES = {
"doc_census_check.py": ("--save-vocab",
["--repo", "{tinyrepo}", "--save-vocab", "{flagtarget}"]),
"queue_index.py": ("--write",
["--repo", "{tinyrepo}", "--queue", "{queue}",
"--write", "{flagtarget}"]),
"tool_map.py": ("--write",
["--repo", "{tinyrepo}", "--write", "{flagtarget}"]),
}
def _write(path, text):
d = os.path.dirname(path)
@@ -185,10 +258,67 @@ def build_fixtures(td, real_repo):
# The probe is meaningless if the target's parent exists: an existing
# directory would let the write succeed and grade the wrong thing.
assert not os.path.isdir(os.path.dirname(p["target"]))
# Second axis: one directory for every caller-path write flag's target, also
# nonexistent, so each flag probe grades a refusal rather than a delivery.
p["flagdir"] = os.path.join(td, "no-such-dir-flag")
assert not os.path.isdir(p["flagdir"])
p["repo"] = real_repo
return p
def flag_target(flagdir, tool, flag):
"""A per-(tool, flag) target inside the nonexistent flag directory."""
return os.path.join(flagdir, "%s%s" % (tool.replace(".py", ""), flag))
def flag_coverage(src, name):
"""-> (flags to probe, [(flag, verdict, detail)]) for one tool's source.
Pure, and that is the point: the UNAXED and STALE_FLAG verdicts are decided
from the source text alone, so the selftest can drive them without spawning
a process. A discovered flag with a table entry is probed; one without is
UNAXED; an entry the source has stopped defining is STALE_FLAG.
"""
disc = sorted(set(WRITE_FLAG.findall(src)))
cov = FLAG_PROBES.get(name)
covflag = cov[0] if cov else None
probe, faults = [], []
for f in disc:
if f == covflag:
probe.append(f)
else:
faults.append((f, "UNAXED",
"write flag discovered with no FLAG_PROBES entry -- ungraded"))
if covflag and covflag not in disc:
faults.append((covflag, "STALE_FLAG",
"FLAG_PROBES names a flag this tool no longer defines"))
return probe, faults
def probe_flags(name, src, path, fx, timeout, cwd, runner=None):
"""Every caller-path write flag this tool defines -> rows (axis `flag`)."""
runner = runner or run_probe
probe, faults = flag_coverage(src, name)
rows = [{"tool": name, "axis": "flag", "flag": f, "verdict": v, "rc": None,
"detail": d} for f, v, d in faults]
cov = FLAG_PROBES.get(name)
for flag in probe:
# probe is non-empty only when the discovered flag IS the covered one,
# so cov is never None here; the guard keeps that visible to a reader.
if not cov or cov[0] != flag:
continue
target = flag_target(fx["flagdir"], name, flag)
fx2 = dict(fx)
fx2["flagtarget"] = target
argv = [re.sub(r"\{(\w+)\}", lambda m: fx2[m.group(1)], x) for x in cov[1]]
rc, out, err = runner(path, argv, timeout, cwd)
rec = {"tool": name, "axis": "flag", "flag": flag, "probe": argv}
rec.update(judge(rc, out, err, target))
rec["detail"] = "%s: %s" % (flag, rec["detail"])
rows.append(rec)
return rows
def judge(rc, out, err, target):
text = (out or "") + (err or "")
if TRACEBACK in text:
@@ -340,7 +470,108 @@ def selftest(repo):
and all(ln.startswith("FAULT ") for ln in blk),
str(blk))
print("selftest: 11 cells, %d passed, %d failed" % (11 - len(bad), len(bad)))
# ---- the second axis (2026-10-21) -------------------------------------
# c12 is the coverage RULE, decided from source text alone: the class is
# discovered, so a tool that grows a --write is a fault until it is probed.
probe_ok, faults_ok = flag_coverage('ap.add_argument("--write", metavar="PATH")\n',
"queue_index.py")
probe_un, faults_un = flag_coverage('ap.add_argument("--save-vocab")\n',
"fake_tool.py")
probe_no, faults_no = flag_coverage(
'ap.add_argument("--json")\nap.add_argument("--emit-baseline", '
'action="store_true")\n', "queue_index.py")
cell("c12 flag coverage: covered -> probed, uncovered -> UNAXED, gone -> "
"STALE_FLAG, and --json/--emit-baseline are outside the class",
probe_ok == ["--write"] and not faults_ok
and not probe_un and [f[1] for f in faults_un] == ["UNAXED"]
and not probe_no and [f[1] for f in faults_no] == ["STALE_FLAG"],
"covered=%s/%s uncovered=%s/%s outside-class=%s/%s"
% (probe_ok, faults_ok, probe_un, faults_un, probe_no, faults_no))
# c13 is the integration cell: this repo's three caller-path write flags are
# probed by the same code path main() uses, over the same fixtures.
real_rows = []
with tempfile.TemporaryDirectory(prefix="write-selftest-flag-") as tdf:
fxf = build_fixtures(tdf, repo)
for nm in ("doc_census_check.py", "queue_index.py", "tool_map.py"):
pf = os.path.join(repo, "tools", nm)
try:
with open(pf, encoding="utf-8") as fh:
s = fh.read()
except OSError as exc:
cell("c13 this repo's three caller-path write flags probe OK", False,
"unreadable %s: %s" % (pf, exc))
real_rows = None
break
real_rows.extend(probe_flags(nm, s, pf, fxf, 300, fxf["tinyrepo"]))
if real_rows is not None:
cell("c13 this repo's three caller-path write flags probe OK",
len(real_rows) == 3 and all(r["verdict"] == "OK" for r in real_rows),
str([(r["tool"], r["flag"], r["verdict"]) for r in real_rows]))
# c14/c15 are the blind spot the axis exists for, measured on a fixture tool
# rather than argued: the SAME tool passes the json axis and fails the flag
# axis, and its guarded twin passes both. Without c14 the axis would be three
# green rows with nothing to show it can go red.
for label, wrap, want in (
("c14 a bare --write is TRACEBACK on the flag axis while the json "
"axis alone reads clean", False, "TRACEBACK"),
("c15 a guarded --write is OK on the flag axis (the axis can pass)", True,
"OK")):
with tempfile.TemporaryDirectory(prefix="write-selftest-bare-") as tdb:
fxb = build_fixtures(tdb, repo)
body = ['#!/usr/bin/env python3', 'import argparse, sys, json',
'ap = argparse.ArgumentParser()',
'ap.add_argument("--json")',
'ap.add_argument("--write", metavar="PATH")',
'a = ap.parse_args()',
# The json half is GUARDED in both fixtures: c14's point is a
# tool that reads clean on the axis this instrument already
# had, and is ungraded on the axis it did not.
'if a.json:',
' try:',
' with open(a.json, "w", encoding="utf-8") as fh:',
' fh.write("{}")',
' except OSError as exc:',
' print("cannot write --json " + a.json + ": "',
' + str(exc), file=sys.stderr)',
' sys.exit(2)']
if wrap:
body += ['if a.write:',
' try:',
' with open(a.write, "w", encoding="utf-8") as fh:',
' fh.write("x")',
' except OSError as exc:',
' print("cannot write --write " + a.write + ": "',
' + str(exc), file=sys.stderr)',
' sys.exit(2)']
else:
body += ['if a.write:',
' with open(a.write, "w", encoding="utf-8") as fh:',
' fh.write("x")']
src = "\n".join(body) + "\n"
st = os.path.join(fxb["toolsdir"], "bare_write.py")
_write(st, src)
saved = dict(FLAG_PROBES)
FLAG_PROBES["bare_write.py"] = ("--write",
["--write", "{flagtarget}"])
try:
jrc, jout, jerr = run_probe(st, ["--json", fxb["target"]], 120,
fxb["tinyrepo"])
json_v = judge(jrc, jout, jerr, fxb["target"])["verdict"]
frows = probe_flags("bare_write.py", src, st, fxb, 120,
fxb["tinyrepo"])
finally:
FLAG_PROBES.clear()
FLAG_PROBES.update(saved)
got_v = frows[0]["verdict"] if len(frows) == 1 else "ROWCOUNT:%d" % len(frows)
ok = got_v == want and frows[0].get("axis") == "flag"
if want == "TRACEBACK":
ok = ok and json_v == "OK"
cell(label, ok, "flag=%s json=%s rc=%s" % (got_v, json_v,
frows[0].get("rc")))
print("selftest: 15 cells, %d passed, %d failed" % (15 - len(bad), len(bad)))
return 1 if bad else 0
@@ -368,6 +599,7 @@ def main(argv=None):
table = dict(WRITE_PROBES)
rows = []
disc = {}
with tempfile.TemporaryDirectory(prefix="write-probe-") as td:
fx = build_fixtures(td, repo)
for name in names:
@@ -378,25 +610,40 @@ def main(argv=None):
with open(path, encoding="utf-8") as fh:
src = fh.read()
except OSError as exc:
rows.append({"tool": name, "verdict": "UNREADABLE", "rc": None,
"detail": str(exc)[:110]})
rows.append({"tool": name, "axis": "json", "verdict": "UNREADABLE",
"rc": None, "detail": str(exc)[:110]})
continue
if name not in table:
rows.append({"tool": name, "verdict": "NO_PROBE", "rc": None,
rows.append({"tool": name, "axis": "json", "verdict": "NO_PROBE",
"rc": None,
"detail": "no write probe in the table -- ungraded"})
continue
if not JSON_FLAG.search(src):
rows.append({"tool": name, "verdict": "NO_JSON", "rc": None,
"detail": "no --json argument in source"})
continue
argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x)
for x in table[name]]
rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"])
rec = {"tool": name, "probe": table[name]}
rec.update(judge(rc, out, err, fx["target"]))
rows.append(rec)
if a.verbose:
print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc))
elif not JSON_FLAG.search(src):
rows.append({"tool": name, "axis": "json", "verdict": "NO_JSON",
"rc": None, "detail": "no --json argument in source"})
else:
argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x)
for x in table[name]]
rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"])
rec = {"tool": name, "axis": "json", "probe": table[name]}
rec.update(judge(rc, out, err, fx["target"]))
rows.append(rec)
if a.verbose:
print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc))
# Second axis: every caller-path write flag this tool defines.
found = set(WRITE_FLAG.findall(src))
if found:
disc[name] = sorted(found)
for rec in probe_flags(name, src, path, fx, a.timeout, fx["tinyrepo"]):
rows.append(rec)
if a.verbose:
print(" . %-24s %-9s rc=%s %s"
% (name, rec["verdict"], rec.get("rc"), rec["flag"]))
for name in sorted(set(FLAG_PROBES) - set(names)):
rows.append({"tool": name, "axis": "flag",
"flag": FLAG_PROBES[name][0], "verdict": "STALE_FLAG",
"rc": None,
"detail": "%s: FLAG_PROBES names a tool that is not tracked"
% FLAG_PROBES[name][0]})
if not rows:
print("cannot read: no tool matched --only " + str(a.only),
file=sys.stderr)
@@ -404,7 +651,7 @@ def main(argv=None):
bad = [r for r in rows if r["verdict"] != "OK"]
width = max([len(r["tool"]) for r in rows] + [4])
print("write probe -- %d of %d tracked tool(s) probed under %s (%s)"
print("write probe -- %d probe(s) over %d tracked tool(s) under %s (%s)"
% (len(rows), len(names), os.path.join(repo, "tools"), source))
print()
faults = fault_lines(rows, width)
@@ -412,20 +659,36 @@ def main(argv=None):
for ln in faults:
print(ln)
print()
print("%-*s %-9s %4s detail" % (width, "tool", "verdict", "rc"))
print("%-*s %-4s %-9s %4s detail" % (width, "tool", "axis", "verdict", "rc"))
for r in rows:
print("%-*s %-9s %4s %s"
% (width, r["tool"], r["verdict"],
print("%-*s %-4s %-9s %4s %s"
% (width, r["tool"], r.get("axis", "json"), r["verdict"],
"-" if r.get("rc") is None else r["rc"], r["detail"]))
print()
print("write axis: %d probed, %d ok, %d fault(s)"
% (len(rows), len(rows) - len(bad), len(bad))
n_json = len([r for r in rows if r.get("axis") == "json"])
n_flag = len([r for r in rows if r.get("axis") == "flag"])
print("write axis: %d probed, %d ok, %d fault(s) (json %d, flag %d)"
% (len(rows), len(rows) - len(bad), len(bad), n_json, n_flag)
+ ("" if not bad else " (" + ", ".join(sorted({r["verdict"]
for r in bad})) + ")"))
# The discovered set is printed, not just counted: the coverage requirement is
# derived from the source, so a reader must be able to see what it found. A
# name outside the --write/--save families is the class's stated blind spot,
# and it is invisible here -- which is why the list itself is on the report.
n_cov = sum(1 for n, fs in disc.items() for f in fs
if (FLAG_PROBES.get(n) or ("",))[0] == f)
print("write flags discovered: %d tool(s) -- %s; %d of %d flag(s) covered"
" by FLAG_PROBES"
% (len(disc),
", ".join("%s %s" % (n, "/".join(f)) for n, f in sorted(disc.items()))
or "none",
n_cov, sum(len(f) for f in disc.values())))
if a.json:
payload = {"repo": repo, "discovery": source, "probed": len(rows),
"ok": len(rows) - len(bad), "faults": len(bad),
"axes": {"json": n_json, "flag": n_flag},
"write_flags": {n: f for n, f in sorted(disc.items())},
"tools": rows}
try:
with open(a.json, "w", encoding="utf-8") as fh: