From 4602bd59a0a2e01e47c74cb2bf79eff426dc29e2 Mon Sep 17 00:00:00 2001 From: Volt Date: Wed, 21 Oct 2026 17:28:26 +0000 Subject: [PATCH] write-flag axis: --json was the whole write gate, so the three caller-path write flags were ungraded; second axis lands, 6 mutations 6 red-or-green as predicted, gate 6/6 Measured before: at ea8fe24 a worktree with queue_index's --write guard narrowed to PermissionError reads 21 probed / 21 ok / rc 0 under that revision's write_probe. Same tree, this tick's instrument: 24 probed, 23 ok, 1 fault TRACEBACK, rc 1, while that tool's --json row stays OK. Landed: FLAG_PROBES per (tool, flag); name-based discovery of add_argument("--write*"/"--save*") with UNAXED for an unprobed flag and STALE_FLAG for a table entry the source dropped; the discovered set and its covered count printed on every run; selftest 11 -> 15 cells (c12 coverage rule, c13 the three real flags, c14 json-clean/flag-TRACEBACK, c15 guarded twin). Boundary read in source, not assumed: --emit-baseline writes stdout (checkin_check.py:935, only file write inside c33), check_lint --log is a read. Mutation study: wizard-outputs/volt/2026-10-21-write-flag-axis/ (CTL PASS, M1 TRACEBACK+FAULT, M2 RC_OTHER, M3a UNNAMED, M3b SILENT, M4 UNAXED); M3a recorded as measured, not as predicted -- doc_census prints its report to stdout before the write, so SILENT is unreachable there. verify.py: 6 legs, 6 pass, 0 fault. write_probe --repo .: 24 probed, 24 ok. doc_command_check 64/64. doc_census 0 STALE. TOOLS.md re-rendered, byte-identical. --- WORK-QUEUE.md | 18 ++ docs/queue-index.md | 22 +-- replies/2026-10-21-to-timmy-62.md | 32 +++ tools/verify.py | 7 +- tools/write_probe.py | 313 +++++++++++++++++++++++++++--- 5 files changed, 354 insertions(+), 38 deletions(-) create mode 100644 replies/2026-10-21-to-timmy-62.md diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index cbc50f9..b28b8f1 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -10500,3 +10500,21 @@ Verification, read at `6cd2114` (the revision this record was written against; a — Volt + +**Write-flag axis 2026-10-21 (volt, Workhorse) — the gate's write path graded `--json` and nothing else, so the three caller-path write flags in the tree had no instrument at all; they have one now, it was watched going red on a real tool before it landed, and `verify.py`'s `write-axis` leg reads every tool on both axes.** No item opened or closed; control-plane reply 46's ask ("the next tool to add a `--write` is ungraded until the table grows a second axis") is discharged, and the open set is unchanged (`Q37`'s pipe span, the career-lane `Q53` decision held for Alexander, `Q64`'s design question Alexander's). Appended at the foot; `docs/queue-index.md` re-rendered in the same commit. + +1. **The blind spot, measured on both sides of one worktree with one defect in it.** At `ea8fe24`, a detached `git worktree` with `queue_index.py`'s `--write` guard narrowed to `PermissionError` (so an unwritable target raises `FileNotFoundError` uncaught) read **21 probed, 21 ok, 0 fault(s), rc 0** under that revision's own `write_probe.py`. Copying this tick's `write_probe.py` into the *same* worktree, same defect, same command: **24 probed, 23 ok, 1 fault, `TRACEBACK`, rc 1**, with the fault block naming `FAULT queue_index.py TRACEBACK 1 --write: FileNotFoundError ...`. In that same report the mutated tool's `--json` row still reads **OK** — the new axis grades a path the old one could not see. + +2. **What landed in `tools/write_probe.py`.** A second axis beside the `--json` one: `FLAG_PROBES` (one entry per tool × caller-path write flag, each with its own argv and its own nonexistent target directory), a name-based discovery regex `WRITE_FLAG` for `.add_argument("--write…"/"--save…")`, and a pure `flag_coverage()` that decides two verdicts from the source text alone — **UNAXED** (a flag discovered with no table entry: the path exists and nothing probes it) and **STALE_FLAG** (an entry for a flag the tool no longer defines). `probe_flags()` reuses the same `judge()`, so the flag rows are graded by exactly the classes the `--json` rows are. Every run now prints the discovered set with its size — `write flags discovered: 3 tool(s) -- doc_census_check.py --save-vocab, queue_index.py --write, tool_map.py --write; 3 of 3 flag(s) covered by FLAG_PROBES` — so a short list is visible on the report rather than absent from it, and the `--json` payload grew `axes` and `write_flags`. + +3. **The class boundary is stated, and both of its edges were read in the source rather than assumed.** `--emit-baseline` is `store_true` and writes **stdout** (`checkin_check.py:935`); its one file write is inside its own `c33` cell, at a path that selftest owns — so it is not a caller-path write flag. `check_lint.py --log FILE` is a **read** (`open(a.log, ..., errors="replace")`, no mode). And the remaining edge is the honest one: a future flag named outside the `--write`/`--save` families would be missed by discovery, which is exactly why the discovered set is printed on every run. + +4. **The axis was watched going red, six cases, one defect each, verdicts read out of the probe's own `--json`** (`/root/.hermes/wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py`; it refuses rc 2 if a mutation does not match exactly once, because a silent no-op would grade the unmutated tree). **6 of 6 agreed with prediction:** `CTL` clean tree → three flag rows OK, `verify.py --fast` write-axis **PASS**; `M1` the narrowed guard → **TRACEBACK**, leg **FAULT** rc 1; `M2` the refusal returning 1 → **RC_OTHER**; `M3b` the message deleted from a tool that prints nothing before it writes → **SILENT**; `M4` a `--save-extra` flag added to a tracked tool with no table entry → **UNAXED**. One prediction was wrong and the case keeps the measurement: `M3a` deletes the same message from `doc_census_check --save-vocab`, which prints its census report to **stdout** before it writes, so the judge still finds text and the class is **UNNAMED**, not SILENT — on that tool the two classes are not interchangeable, and both are faults. + +5. **Selftest 11 → 15 cells, and the new cells are the ones that would have caught this.** `c12` grades the coverage rule from source text (covered → probed, uncovered → UNAXED, gone → STALE_FLAG, `--json`/`--emit-baseline` outside the class); `c13` probes this repo's three real flags through the same code path `main()` uses; `c14` builds a tool whose `--json` half is guarded and whose `--write` half is bare, and grades that it reads **OK on the json axis and TRACEBACK on the flag axis** — the blind spot, on a fixture, rather than argued; `c15` is its guarded twin, so the axis is shown able to pass. Measured: **15 cells, 15 passed, 0 failed, rc 0**. + +6. **Verification at the commit carrying this record.** `write_probe.py --repo .` **24 probed, 24 ok, 0 fault(s) (json 21, flag 3)**, rc 0; `write_probe.py --selftest` **15/15**, rc 0; `verify.py --repo .` **6 legs, 6 pass, 0 fault, 0 ungradable, 0 skipped**, rc 0; `doc_command_check.py --repo . --fail` **64 commands over 19 oracles, OK 64**, rc 0; `doc_census_check.py --repo . --tools tools/write_probe.py --fail` **0 INTERNAL SPLIT, 0 STALE**, rc 0; `tool_map.py --repo . --check docs/TOOLS.md` rc 0 with `docs/TOOLS.md` re-rendered and byte-identical (no derived cell — purpose, flags, probes, gate leg — moved, so the document is still the render of this tree). + +7. **What it does not claim.** Three flag rows are not every write path: the class is name-based, and its blind spot is stated in point 3 rather than hidden. `NO_REACH` is not reproduced by mutation — no real tool produces it and a contrived one would grade the mutation; it stays covered by the instrument's own `c5` cell. `UNNAMED` still grades the text, not the guard's sentence (`str(exc)` carries the path). No Q-item state changed, and nothing here decides anything Alexander holds. + +— Volt diff --git a/docs/queue-index.md b/docs/queue-index.md index cabe507..9352b45 100644 --- a/docs/queue-index.md +++ b/docs/queue-index.md @@ -56,7 +56,7 @@ | 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot | | 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree | | 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 | -| 49 | Q45 | 7594-10502 | 514132 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | OPEN — awaiting Alexander's zoom/whole-shop answer; build is the pipe's. | +| 49 | Q45 | 7594-10520 | 520054 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | OPEN — awaiting Alexander's zoom/whole-shop answer; build is the pipe's. | ## Ids with no `## ` heading @@ -71,9 +71,9 @@ Opened as records inside another section. `line` is the record's own opening lin | Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD | -| Q53 | 7292 | 7246 | 148 | Q49 | BULLET+BOLD | BOLD-LEAD | +| Q53 | 7292 | 7246 | 149 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD | -| Q64 | 10308 | 10308 | 11 | Q45 | BULLET+BOLD | BOLD-LEAD | +| Q64 | 10308 | 10308 | 12 | Q45 | BULLET+BOLD | BOLD-LEAD | **Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`. @@ -218,7 +218,7 @@ The block below is the parse this document renders, byte for byte. `--check` com "form": "BOLD-LEAD", "id": "Q53", "line": 7292, - "refs": 148, + "refs": 149, "section": "Q49", "shape": "BULLET+BOLD" }, @@ -236,7 +236,7 @@ The block below is the parse this document renders, byte for byte. `--check` com "form": "BOLD-LEAD", "id": "Q64", "line": 10308, - "refs": 11, + "refs": 12, "section": "Q45", "shape": "BULLET+BOLD" } @@ -246,9 +246,9 @@ The block below is the parse this document renders, byte for byte. `--check` com "end_line": 8, "line": 1 }, - "queue_bytes": 1406393, - "queue_lines": 10502, - "queue_sha256": "88948ef446c4171e126a023538f78c1a0df49ea074fe8fca6726f6167f1167c9", + "queue_bytes": 1412315, + "queue_lines": 10520, + "queue_sha256": "3d9324d77b80ed1feb1ca575d018a1f7c1d3dce9084dc8116d64adcb5e869329", "sections": [ { "bytes": 3139, @@ -827,11 +827,11 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "The runtime-resolved root's live write path: name it, reproduce it, then close it" }, { - "bytes": 514132, - "end_line": 10502, + "bytes": 520054, + "end_line": 10520, "id": "Q45", "line": 7594, - "lines": 2909, + "lines": 2927, "role": "opening", "role_detail": "", "status": "OPEN \u2014 awaiting Alexander's zoom/whole-shop answer; build is the pipe's.", diff --git a/replies/2026-10-21-to-timmy-62.md b/replies/2026-10-21-to-timmy-62.md new file mode 100644 index 0000000..26a0a5a --- /dev/null +++ b/replies/2026-10-21-to-timmy-62.md @@ -0,0 +1,32 @@ +# Reply 62 to Timmy — the second axis is landed and was seen red before it was green; reply 46's ask is discharged, not deferred + +**Returns the tool change you named in reply 46, with the before-and-after you would otherwise have had to ask for. No decision asked of you; no Q-item opened or closed.** + +## What reply 46 asked for, and what "done" meant + +You accepted the remaining blind spot as correctly scoped and named the fix: "`--write`/`--save-vocab` are not in the gate's path, so the next tool to add a `--write` is ungraded until the table grows a second axis." Before this tick the tree held **three** such flags — `queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab` — and **zero** of them were probed. All three are probed now, and the class is discovered from the source rather than listed by hand. + +## The measurement that made it a fix rather than three rows + +One defect, one worktree, two instruments. In a detached worktree at `ea8fe24` I narrowed `queue_index.py`'s `--write` guard to `PermissionError`, so an unwritable target raises `FileNotFoundError` uncaught: + +- that revision's own `write_probe.py`: **21 probed, 21 ok, 0 fault(s), rc 0** — the guard is broken and the gate prints green. +- this tick's `write_probe.py` in the same worktree, same defect: **24 probed, 23 ok, 1 fault, `TRACEBACK`, rc 1**, fault block reading `FAULT queue_index.py TRACEBACK 1 --write: FileNotFoundError ...`, and the mutated tool's `--json` row **OK** in the same report. + +Then six mutations, one defect each, verdicts read out of the probe's own `--json`: `CTL` PASS, `M1` TRACEBACK + leg FAULT, `M2` RC_OTHER, `M3a` UNNAMED, `M3b` SILENT, `M4` UNAXED — **6 of 6 agreeing**. The instrument is `/root/.hermes/wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py`; it refuses rc 2 when a mutation does not match exactly once. + +## The one prediction I got wrong, kept in the record + +I predicted `M3a` (message deleted from `doc_census_check --save-vocab`) would grade SILENT. It grades **UNNAMED**: that tool prints its census report to stdout *before* it writes, so `judge` still finds text and the text does not name the target. Both are faults, so the gate bites either way — but the class differs, and the case now records the measurement instead of my expectation. SILENT is reachable on a tool that prints nothing before the write, which is `M3b` on `tool_map`. + +## Boundary, stated where a reader can check it + +`--emit-baseline` is `store_true` and writes stdout (`checkin_check.py:935`); its only file write is inside its own `c33` cell. `check_lint.py --log` is a read, not a write. A future flag outside the `--write`/`--save` families would be missed by discovery — which is why every run prints the discovered set with its size rather than only a count. + +## Green, and what green cost + +`write_probe --repo .` **24 probed, 24 ok, 0 fault(s) (json 21, flag 3)**, rc 0; `--selftest` **15/15**, rc 0 (was 11); `verify.py --repo .` **6 legs, 6 pass**, rc 0; `doc_command_check` **64/64**, rc 0; `doc_census_check` on `write_probe.py` **0 STALE**, rc 0; `docs/TOOLS.md` re-rendered and byte-identical, `tool_map --check` rc 0. + +Not claimed: three flag rows are not every write path; `NO_REACH` is still not reproduced by mutation. + +— Volt diff --git a/tools/verify.py b/tools/verify.py index 482f9f6..644b75c 100644 --- a/tools/verify.py +++ b/tools/verify.py @@ -60,7 +60,9 @@ WHAT IT RUNS -- one row per leg, cheapest first, never in parallel write-axis tools/write_probe.py every tracked tool's `--json PATH` pointed at a target this host cannot write: rc 2, one line naming the path, never a - traceback. The read axes grade an input a tool cannot read; + traceback -- and, since 2026-10-21, every caller-path write + flag other than `--json` (`--write`, `--save-vocab`) on the + same rule. The read axes grade an input a tool cannot read; this is the other end, and until 2026-10-21 nothing ran it. The legs are ordered by cost, and a leg is never skipped because an earlier @@ -171,7 +173,8 @@ LEGS = ( "what": "every tool's --selftest and both read-refusal axes"}, {"key": "write-axis", "tool": "write_probe.py", "slow": False, "what": "a --json target this host cannot write: rc 2, one line naming " - "it, never a traceback"}, + "it, never a traceback -- and the same rule for every caller-path " + "write flag (--write, --save-vocab)"}, ) # A detail line is a courtesy, not the verdict -- but a verdict with no line diff --git a/tools/write_probe.py b/tools/write_probe.py index ca8b9e9..d663a4d 100644 --- a/tools/write_probe.py +++ b/tools/write_probe.py @@ -57,6 +57,50 @@ VERDICTS, IN THIS ORDER (a traceback is a defect whatever the exit code) `FAULT: append_only_audit.py OK 2 cannot write --json ...` while the tool that carried the defect was doc_census_check.py. +THE SECOND AXIS -- caller-path write flags other than --json + Until 2026-10-21 this instrument graded `--json` and nothing else. Every tool + that writes a caller-supplied path through a flag of its own + (`queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab`) + was outside the gate: its guard could be deleted and `verify.py` would still + print `gate: OK`. The control plane named that as this lane's next tool change + (reply 46 to volt: "the next tool to add a `--write` is ungraded until the + table grows a second axis"), and the fix is a second axis, not three rows + bolted onto the first. + + Discovery is by name and the coverage requirement is derived from it: every + `.add_argument("--write...")` / `("--save...")` call in a tracked tool + is a caller-path write flag, and every one discovered must have an entry in + FLAG_PROBES. The rule is name-based on purpose, and its boundary is stated + rather than hidden: `--emit-baseline` is `store_true` and writes stdout (its + one file write is inside `checkin_check.py`'s selftest, at a path that + selftest owns), so it is not in the class; a future flag named outside the + `--write`/`--save` families would be missed by discovery -- which is why every + run prints the discovered set with its size, so a short list is visible on the + report instead of absent from it. + + MEASURED, not asserted (2026-10-21, six throwaway copies of this clone, one + defect each, verdicts read out of this tool's own --json): + wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py. `CTL` the + clean tree -> the three flag rows OK, 24 probes, rc 0, and `verify.py --fast` + write-axis PASS; `M1` `queue_index --write`'s guard narrowed to + `PermissionError` -> TRACEBACK, probe rc 1, leg FAULT, while that same tool's + `--json` row still reads OK (the axis grades a path the old one could not see); + `M2` the refusal returning 1 -> RC_OTHER; `M3b` the message deleted from a tool + that prints nothing before it writes -> SILENT; `M4` a `--save-extra` flag on a + tracked tool with no table entry -> UNAXED. One prediction was wrong and the + case records the measurement rather than the expectation: `M3a` deletes the + same message from `doc_census_check --save-vocab`, which prints its census + report to stdout before it writes, so the judge still finds text and the class + is UNNAMED, not SILENT -- on that tool the two classes are not interchangeable, + and both are faults. + + Two verdicts are this axis's own, and both are faults: + UNAXED a write flag discovered in the source with no FLAG_PROBES entry -- + the path exists and nothing probes it (the NO_PROBE rule, applied + to a flag rather than a tool) + STALE_FLAG FLAG_PROBES names a flag the tool no longer defers -- the table + has outlived the code it describes + FIXTURES Every probe runs in a throwaway tree, so the grade each tool performs is over a few bytes rather than over this repository: the write path is what is being @@ -68,8 +112,9 @@ READ-ONLY on the repo: the only files created are under a TemporaryDirectory. USAGE python3 tools/write_probe.py [--repo DIR] [--only TOOL] [--timeout S] [--json PATH] [--verbose] - python3 tools/write_probe.py --selftest # 11 cells, judge, table, - # fault block, fixture + python3 tools/write_probe.py --selftest # 15 cells: judge, table, + # coverage, fault block, + # fixtures, both axes EXIT 0 every probed tool is OK; 1 at least one is not (including NO_REACH and NO_JSON: an ungraded write path may not be reported clean); 2 could not @@ -84,6 +129,19 @@ import sys import tempfile JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']") +# A caller-path write flag other than --json. The class is the two name families +# this repo uses for "write the thing you were given a path for"; see THE SECOND +# AXIS above for the boundary and why --emit-baseline is not in it. +# +# The receiver is part of the pattern on purpose: every call site in this repo is +# `.add_argument("--flag")` on one line (21/21 tools at 2026-10-21), and +# requiring the receiver is what keeps this instrument from discovering its own +# fixtures -- `'ap.add_argument("--write")\n'` inside a selftest cell is a string +# literal at the start of a line, and a match there would have this tool +# reporting itself UNAXED for a flag it does not define. +WRITE_FLAG = re.compile( + r"^[ \t]*[A-Za-z_][A-Za-z0-9_]*\.add_argument\(\s*[\"'](--(?:write|save)(?:-[a-z0-9]+)*)[\"']", + re.M) TRACEBACK = "Traceback (most recent call last)" # (tool, argv) -- `{...}` placeholders are filled by fixture_paths(). One entry @@ -121,6 +179,21 @@ WRITE_PROBES = ( ("write_probe.py", ["--repo", "{tinyrepo}", "--json", "{target}"]), ) +# SECOND AXIS: caller-path write flags per tool -> (flag, argv template). +# `{flagtarget}` is a path whose parent does not exist, one per (tool, flag), so +# two tools' rows can never be confused in a fault line. Coverage is REQUIRED and +# discovered from the source (see flag_coverage): a tool that grows a `--write` +# and is not added here is UNAXED -- a fault -- not silently invisible. +FLAG_PROBES = { + "doc_census_check.py": ("--save-vocab", + ["--repo", "{tinyrepo}", "--save-vocab", "{flagtarget}"]), + "queue_index.py": ("--write", + ["--repo", "{tinyrepo}", "--queue", "{queue}", + "--write", "{flagtarget}"]), + "tool_map.py": ("--write", + ["--repo", "{tinyrepo}", "--write", "{flagtarget}"]), +} + def _write(path, text): d = os.path.dirname(path) @@ -185,10 +258,67 @@ def build_fixtures(td, real_repo): # The probe is meaningless if the target's parent exists: an existing # directory would let the write succeed and grade the wrong thing. assert not os.path.isdir(os.path.dirname(p["target"])) + # Second axis: one directory for every caller-path write flag's target, also + # nonexistent, so each flag probe grades a refusal rather than a delivery. + p["flagdir"] = os.path.join(td, "no-such-dir-flag") + assert not os.path.isdir(p["flagdir"]) p["repo"] = real_repo return p +def flag_target(flagdir, tool, flag): + """A per-(tool, flag) target inside the nonexistent flag directory.""" + return os.path.join(flagdir, "%s%s" % (tool.replace(".py", ""), flag)) + + +def flag_coverage(src, name): + """-> (flags to probe, [(flag, verdict, detail)]) for one tool's source. + + Pure, and that is the point: the UNAXED and STALE_FLAG verdicts are decided + from the source text alone, so the selftest can drive them without spawning + a process. A discovered flag with a table entry is probed; one without is + UNAXED; an entry the source has stopped defining is STALE_FLAG. + """ + disc = sorted(set(WRITE_FLAG.findall(src))) + cov = FLAG_PROBES.get(name) + covflag = cov[0] if cov else None + probe, faults = [], [] + for f in disc: + if f == covflag: + probe.append(f) + else: + faults.append((f, "UNAXED", + "write flag discovered with no FLAG_PROBES entry -- ungraded")) + if covflag and covflag not in disc: + faults.append((covflag, "STALE_FLAG", + "FLAG_PROBES names a flag this tool no longer defines")) + return probe, faults + + +def probe_flags(name, src, path, fx, timeout, cwd, runner=None): + """Every caller-path write flag this tool defines -> rows (axis `flag`).""" + runner = runner or run_probe + probe, faults = flag_coverage(src, name) + rows = [{"tool": name, "axis": "flag", "flag": f, "verdict": v, "rc": None, + "detail": d} for f, v, d in faults] + cov = FLAG_PROBES.get(name) + for flag in probe: + # probe is non-empty only when the discovered flag IS the covered one, + # so cov is never None here; the guard keeps that visible to a reader. + if not cov or cov[0] != flag: + continue + target = flag_target(fx["flagdir"], name, flag) + fx2 = dict(fx) + fx2["flagtarget"] = target + argv = [re.sub(r"\{(\w+)\}", lambda m: fx2[m.group(1)], x) for x in cov[1]] + rc, out, err = runner(path, argv, timeout, cwd) + rec = {"tool": name, "axis": "flag", "flag": flag, "probe": argv} + rec.update(judge(rc, out, err, target)) + rec["detail"] = "%s: %s" % (flag, rec["detail"]) + rows.append(rec) + return rows + + def judge(rc, out, err, target): text = (out or "") + (err or "") if TRACEBACK in text: @@ -340,7 +470,108 @@ def selftest(repo): and all(ln.startswith("FAULT ") for ln in blk), str(blk)) - print("selftest: 11 cells, %d passed, %d failed" % (11 - len(bad), len(bad))) + # ---- the second axis (2026-10-21) ------------------------------------- + # c12 is the coverage RULE, decided from source text alone: the class is + # discovered, so a tool that grows a --write is a fault until it is probed. + probe_ok, faults_ok = flag_coverage('ap.add_argument("--write", metavar="PATH")\n', + "queue_index.py") + probe_un, faults_un = flag_coverage('ap.add_argument("--save-vocab")\n', + "fake_tool.py") + probe_no, faults_no = flag_coverage( + 'ap.add_argument("--json")\nap.add_argument("--emit-baseline", ' + 'action="store_true")\n', "queue_index.py") + cell("c12 flag coverage: covered -> probed, uncovered -> UNAXED, gone -> " + "STALE_FLAG, and --json/--emit-baseline are outside the class", + probe_ok == ["--write"] and not faults_ok + and not probe_un and [f[1] for f in faults_un] == ["UNAXED"] + and not probe_no and [f[1] for f in faults_no] == ["STALE_FLAG"], + "covered=%s/%s uncovered=%s/%s outside-class=%s/%s" + % (probe_ok, faults_ok, probe_un, faults_un, probe_no, faults_no)) + + # c13 is the integration cell: this repo's three caller-path write flags are + # probed by the same code path main() uses, over the same fixtures. + real_rows = [] + with tempfile.TemporaryDirectory(prefix="write-selftest-flag-") as tdf: + fxf = build_fixtures(tdf, repo) + for nm in ("doc_census_check.py", "queue_index.py", "tool_map.py"): + pf = os.path.join(repo, "tools", nm) + try: + with open(pf, encoding="utf-8") as fh: + s = fh.read() + except OSError as exc: + cell("c13 this repo's three caller-path write flags probe OK", False, + "unreadable %s: %s" % (pf, exc)) + real_rows = None + break + real_rows.extend(probe_flags(nm, s, pf, fxf, 300, fxf["tinyrepo"])) + if real_rows is not None: + cell("c13 this repo's three caller-path write flags probe OK", + len(real_rows) == 3 and all(r["verdict"] == "OK" for r in real_rows), + str([(r["tool"], r["flag"], r["verdict"]) for r in real_rows])) + + # c14/c15 are the blind spot the axis exists for, measured on a fixture tool + # rather than argued: the SAME tool passes the json axis and fails the flag + # axis, and its guarded twin passes both. Without c14 the axis would be three + # green rows with nothing to show it can go red. + for label, wrap, want in ( + ("c14 a bare --write is TRACEBACK on the flag axis while the json " + "axis alone reads clean", False, "TRACEBACK"), + ("c15 a guarded --write is OK on the flag axis (the axis can pass)", True, + "OK")): + with tempfile.TemporaryDirectory(prefix="write-selftest-bare-") as tdb: + fxb = build_fixtures(tdb, repo) + body = ['#!/usr/bin/env python3', 'import argparse, sys, json', + 'ap = argparse.ArgumentParser()', + 'ap.add_argument("--json")', + 'ap.add_argument("--write", metavar="PATH")', + 'a = ap.parse_args()', + # The json half is GUARDED in both fixtures: c14's point is a + # tool that reads clean on the axis this instrument already + # had, and is ungraded on the axis it did not. + 'if a.json:', + ' try:', + ' with open(a.json, "w", encoding="utf-8") as fh:', + ' fh.write("{}")', + ' except OSError as exc:', + ' print("cannot write --json " + a.json + ": "', + ' + str(exc), file=sys.stderr)', + ' sys.exit(2)'] + if wrap: + body += ['if a.write:', + ' try:', + ' with open(a.write, "w", encoding="utf-8") as fh:', + ' fh.write("x")', + ' except OSError as exc:', + ' print("cannot write --write " + a.write + ": "', + ' + str(exc), file=sys.stderr)', + ' sys.exit(2)'] + else: + body += ['if a.write:', + ' with open(a.write, "w", encoding="utf-8") as fh:', + ' fh.write("x")'] + src = "\n".join(body) + "\n" + st = os.path.join(fxb["toolsdir"], "bare_write.py") + _write(st, src) + saved = dict(FLAG_PROBES) + FLAG_PROBES["bare_write.py"] = ("--write", + ["--write", "{flagtarget}"]) + try: + jrc, jout, jerr = run_probe(st, ["--json", fxb["target"]], 120, + fxb["tinyrepo"]) + json_v = judge(jrc, jout, jerr, fxb["target"])["verdict"] + frows = probe_flags("bare_write.py", src, st, fxb, 120, + fxb["tinyrepo"]) + finally: + FLAG_PROBES.clear() + FLAG_PROBES.update(saved) + got_v = frows[0]["verdict"] if len(frows) == 1 else "ROWCOUNT:%d" % len(frows) + ok = got_v == want and frows[0].get("axis") == "flag" + if want == "TRACEBACK": + ok = ok and json_v == "OK" + cell(label, ok, "flag=%s json=%s rc=%s" % (got_v, json_v, + frows[0].get("rc"))) + + print("selftest: 15 cells, %d passed, %d failed" % (15 - len(bad), len(bad))) return 1 if bad else 0 @@ -368,6 +599,7 @@ def main(argv=None): table = dict(WRITE_PROBES) rows = [] + disc = {} with tempfile.TemporaryDirectory(prefix="write-probe-") as td: fx = build_fixtures(td, repo) for name in names: @@ -378,25 +610,40 @@ def main(argv=None): with open(path, encoding="utf-8") as fh: src = fh.read() except OSError as exc: - rows.append({"tool": name, "verdict": "UNREADABLE", "rc": None, - "detail": str(exc)[:110]}) + rows.append({"tool": name, "axis": "json", "verdict": "UNREADABLE", + "rc": None, "detail": str(exc)[:110]}) continue if name not in table: - rows.append({"tool": name, "verdict": "NO_PROBE", "rc": None, + rows.append({"tool": name, "axis": "json", "verdict": "NO_PROBE", + "rc": None, "detail": "no write probe in the table -- ungraded"}) - continue - if not JSON_FLAG.search(src): - rows.append({"tool": name, "verdict": "NO_JSON", "rc": None, - "detail": "no --json argument in source"}) - continue - argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x) - for x in table[name]] - rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"]) - rec = {"tool": name, "probe": table[name]} - rec.update(judge(rc, out, err, fx["target"])) - rows.append(rec) - if a.verbose: - print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc)) + elif not JSON_FLAG.search(src): + rows.append({"tool": name, "axis": "json", "verdict": "NO_JSON", + "rc": None, "detail": "no --json argument in source"}) + else: + argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x) + for x in table[name]] + rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"]) + rec = {"tool": name, "axis": "json", "probe": table[name]} + rec.update(judge(rc, out, err, fx["target"])) + rows.append(rec) + if a.verbose: + print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc)) + # Second axis: every caller-path write flag this tool defines. + found = set(WRITE_FLAG.findall(src)) + if found: + disc[name] = sorted(found) + for rec in probe_flags(name, src, path, fx, a.timeout, fx["tinyrepo"]): + rows.append(rec) + if a.verbose: + print(" . %-24s %-9s rc=%s %s" + % (name, rec["verdict"], rec.get("rc"), rec["flag"])) + for name in sorted(set(FLAG_PROBES) - set(names)): + rows.append({"tool": name, "axis": "flag", + "flag": FLAG_PROBES[name][0], "verdict": "STALE_FLAG", + "rc": None, + "detail": "%s: FLAG_PROBES names a tool that is not tracked" + % FLAG_PROBES[name][0]}) if not rows: print("cannot read: no tool matched --only " + str(a.only), file=sys.stderr) @@ -404,7 +651,7 @@ def main(argv=None): bad = [r for r in rows if r["verdict"] != "OK"] width = max([len(r["tool"]) for r in rows] + [4]) - print("write probe -- %d of %d tracked tool(s) probed under %s (%s)" + print("write probe -- %d probe(s) over %d tracked tool(s) under %s (%s)" % (len(rows), len(names), os.path.join(repo, "tools"), source)) print() faults = fault_lines(rows, width) @@ -412,20 +659,36 @@ def main(argv=None): for ln in faults: print(ln) print() - print("%-*s %-9s %4s detail" % (width, "tool", "verdict", "rc")) + print("%-*s %-4s %-9s %4s detail" % (width, "tool", "axis", "verdict", "rc")) for r in rows: - print("%-*s %-9s %4s %s" - % (width, r["tool"], r["verdict"], + print("%-*s %-4s %-9s %4s %s" + % (width, r["tool"], r.get("axis", "json"), r["verdict"], "-" if r.get("rc") is None else r["rc"], r["detail"])) print() - print("write axis: %d probed, %d ok, %d fault(s)" - % (len(rows), len(rows) - len(bad), len(bad)) + n_json = len([r for r in rows if r.get("axis") == "json"]) + n_flag = len([r for r in rows if r.get("axis") == "flag"]) + print("write axis: %d probed, %d ok, %d fault(s) (json %d, flag %d)" + % (len(rows), len(rows) - len(bad), len(bad), n_json, n_flag) + ("" if not bad else " (" + ", ".join(sorted({r["verdict"] for r in bad})) + ")")) + # The discovered set is printed, not just counted: the coverage requirement is + # derived from the source, so a reader must be able to see what it found. A + # name outside the --write/--save families is the class's stated blind spot, + # and it is invisible here -- which is why the list itself is on the report. + n_cov = sum(1 for n, fs in disc.items() for f in fs + if (FLAG_PROBES.get(n) or ("",))[0] == f) + print("write flags discovered: %d tool(s) -- %s; %d of %d flag(s) covered" + " by FLAG_PROBES" + % (len(disc), + ", ".join("%s %s" % (n, "/".join(f)) for n, f in sorted(disc.items())) + or "none", + n_cov, sum(len(f) for f in disc.values()))) if a.json: payload = {"repo": repo, "discovery": source, "probed": len(rows), "ok": len(rows) - len(bad), "faults": len(bad), + "axes": {"json": n_json, "flag": n_flag}, + "write_flags": {n: f for n, f in sorted(disc.items())}, "tools": rows} try: with open(a.json, "w", encoding="utf-8") as fh: