write-flag axis: --json was the whole write gate, so the three caller-path write flags were ungraded; second axis lands, 6 mutations 6 red-or-green as predicted, gate 6/6

Measured before: at ea8fe24 a worktree with queue_index's --write guard
narrowed to PermissionError reads 21 probed / 21 ok / rc 0 under that
revision's write_probe. Same tree, this tick's instrument: 24 probed,
23 ok, 1 fault TRACEBACK, rc 1, while that tool's --json row stays OK.

Landed: FLAG_PROBES per (tool, flag); name-based discovery of
add_argument("--write*"/"--save*") with UNAXED for an unprobed flag and
STALE_FLAG for a table entry the source dropped; the discovered set and
its covered count printed on every run; selftest 11 -> 15 cells (c12
coverage rule, c13 the three real flags, c14 json-clean/flag-TRACEBACK,
c15 guarded twin).

Boundary read in source, not assumed: --emit-baseline writes stdout
(checkin_check.py:935, only file write inside c33), check_lint --log is a
read.

Mutation study: wizard-outputs/volt/2026-10-21-write-flag-axis/ (CTL PASS,
M1 TRACEBACK+FAULT, M2 RC_OTHER, M3a UNNAMED, M3b SILENT, M4 UNAXED);
M3a recorded as measured, not as predicted -- doc_census prints its report
to stdout before the write, so SILENT is unreachable there.

verify.py: 6 legs, 6 pass, 0 fault. write_probe --repo .: 24 probed,
24 ok. doc_command_check 64/64. doc_census 0 STALE. TOOLS.md re-rendered,
byte-identical.
This commit is contained in:
volt committed 2026-10-21 17:28:26 +00:00
1 parent ea8fe24563
commit 4602bd59a0
5 files changed
+347 -31

No files matched your search

+18
View File
@@ -10500,3 +10500,21 @@ Verification, read at `6cd2114` (the revision this record was written against; a
— Volt — Volt
**Write-flag axis 2026-10-21 (volt, Workhorse) — the gate's write path graded `--json` and nothing else, so the three caller-path write flags in the tree had no instrument at all; they have one now, it was watched going red on a real tool before it landed, and `verify.py`'s `write-axis` leg reads every tool on both axes.** No item opened or closed; control-plane reply 46's ask ("the next tool to add a `--write` is ungraded until the table grows a second axis") is discharged, and the open set is unchanged (`Q37`'s pipe span, the career-lane `Q53` decision held for Alexander, `Q64`'s design question Alexander's). Appended at the foot; `docs/queue-index.md` re-rendered in the same commit.
1. **The blind spot, measured on both sides of one worktree with one defect in it.** At `ea8fe24`, a detached `git worktree` with `queue_index.py`'s `--write` guard narrowed to `PermissionError` (so an unwritable target raises `FileNotFoundError` uncaught) read **21 probed, 21 ok, 0 fault(s), rc 0** under that revision's own `write_probe.py`. Copying this tick's `write_probe.py` into the *same* worktree, same defect, same command: **24 probed, 23 ok, 1 fault, `TRACEBACK`, rc 1**, with the fault block naming `FAULT queue_index.py TRACEBACK 1 --write: FileNotFoundError ...`. In that same report the mutated tool's `--json` row still reads **OK** — the new axis grades a path the old one could not see.
2. **What landed in `tools/write_probe.py`.** A second axis beside the `--json` one: `FLAG_PROBES` (one entry per tool × caller-path write flag, each with its own argv and its own nonexistent target directory), a name-based discovery regex `WRITE_FLAG` for `<parser>.add_argument("--write…"/"--save…")`, and a pure `flag_coverage()` that decides two verdicts from the source text alone — **UNAXED** (a flag discovered with no table entry: the path exists and nothing probes it) and **STALE_FLAG** (an entry for a flag the tool no longer defines). `probe_flags()` reuses the same `judge()`, so the flag rows are graded by exactly the classes the `--json` rows are. Every run now prints the discovered set with its size — `write flags discovered: 3 tool(s) -- doc_census_check.py --save-vocab, queue_index.py --write, tool_map.py --write; 3 of 3 flag(s) covered by FLAG_PROBES` — so a short list is visible on the report rather than absent from it, and the `--json` payload grew `axes` and `write_flags`.
3. **The class boundary is stated, and both of its edges were read in the source rather than assumed.** `--emit-baseline` is `store_true` and writes **stdout** (`checkin_check.py:935`); its one file write is inside its own `c33` cell, at a path that selftest owns — so it is not a caller-path write flag. `check_lint.py --log FILE` is a **read** (`open(a.log, ..., errors="replace")`, no mode). And the remaining edge is the honest one: a future flag named outside the `--write`/`--save` families would be missed by discovery, which is exactly why the discovered set is printed on every run.
4. **The axis was watched going red, six cases, one defect each, verdicts read out of the probe's own `--json`** (`/root/.hermes/wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py`; it refuses rc 2 if a mutation does not match exactly once, because a silent no-op would grade the unmutated tree). **6 of 6 agreed with prediction:** `CTL` clean tree → three flag rows OK, `verify.py --fast` write-axis **PASS**; `M1` the narrowed guard → **TRACEBACK**, leg **FAULT** rc 1; `M2` the refusal returning 1 → **RC_OTHER**; `M3b` the message deleted from a tool that prints nothing before it writes → **SILENT**; `M4` a `--save-extra` flag added to a tracked tool with no table entry → **UNAXED**. One prediction was wrong and the case keeps the measurement: `M3a` deletes the same message from `doc_census_check --save-vocab`, which prints its census report to **stdout** before it writes, so the judge still finds text and the class is **UNNAMED**, not SILENT — on that tool the two classes are not interchangeable, and both are faults.
5. **Selftest 11 → 15 cells, and the new cells are the ones that would have caught this.** `c12` grades the coverage rule from source text (covered → probed, uncovered → UNAXED, gone → STALE_FLAG, `--json`/`--emit-baseline` outside the class); `c13` probes this repo's three real flags through the same code path `main()` uses; `c14` builds a tool whose `--json` half is guarded and whose `--write` half is bare, and grades that it reads **OK on the json axis and TRACEBACK on the flag axis** — the blind spot, on a fixture, rather than argued; `c15` is its guarded twin, so the axis is shown able to pass. Measured: **15 cells, 15 passed, 0 failed, rc 0**.
6. **Verification at the commit carrying this record.** `write_probe.py --repo .` **24 probed, 24 ok, 0 fault(s) (json 21, flag 3)**, rc 0; `write_probe.py --selftest` **15/15**, rc 0; `verify.py --repo .` **6 legs, 6 pass, 0 fault, 0 ungradable, 0 skipped**, rc 0; `doc_command_check.py --repo . --fail` **64 commands over 19 oracles, OK 64**, rc 0; `doc_census_check.py --repo . --tools tools/write_probe.py --fail` **0 INTERNAL SPLIT, 0 STALE**, rc 0; `tool_map.py --repo . --check docs/TOOLS.md` rc 0 with `docs/TOOLS.md` re-rendered and byte-identical (no derived cell — purpose, flags, probes, gate leg — moved, so the document is still the render of this tree).
7. **What it does not claim.** Three flag rows are not every write path: the class is name-based, and its blind spot is stated in point 3 rather than hidden. `NO_REACH` is not reproduced by mutation — no real tool produces it and a contrived one would grade the mutation; it stays covered by the instrument's own `c5` cell. `UNNAMED` still grades the text, not the guard's sentence (`str(exc)` carries the path). No Q-item state changed, and nothing here decides anything Alexander holds.
— Volt
+11 -11
View File
@@ -56,7 +56,7 @@
| 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot | | 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot |
| 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree | | 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree |
| 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 | | 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 |
| 49 | Q45 | 7594-10502 | 514132 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | OPEN — awaiting Alexander's zoom/whole-shop answer; build is the pipe's. | | 49 | Q45 | 7594-10520 | 520054 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | OPEN — awaiting Alexander's zoom/whole-shop answer; build is the pipe's. |
## Ids with no `## ` heading ## Ids with no `## ` heading
@@ -71,9 +71,9 @@ Opened as records inside another section. `line` is the record's own opening lin
| Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q53 | 7292 | 7246 | 148 | Q49 | BULLET+BOLD | BOLD-LEAD | | Q53 | 7292 | 7246 | 149 | Q49 | BULLET+BOLD | BOLD-LEAD |
| Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD | | Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD |
| Q64 | 10308 | 10308 | 11 | Q45 | BULLET+BOLD | BOLD-LEAD | | Q64 | 10308 | 10308 | 12 | Q45 | BULLET+BOLD | BOLD-LEAD |
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`. **Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
@@ -218,7 +218,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
"form": "BOLD-LEAD", "form": "BOLD-LEAD",
"id": "Q53", "id": "Q53",
"line": 7292, "line": 7292,
"refs": 148, "refs": 149,
"section": "Q49", "section": "Q49",
"shape": "BULLET+BOLD" "shape": "BULLET+BOLD"
}, },
@@ -236,7 +236,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
"form": "BOLD-LEAD", "form": "BOLD-LEAD",
"id": "Q64", "id": "Q64",
"line": 10308, "line": 10308,
"refs": 11, "refs": 12,
"section": "Q45", "section": "Q45",
"shape": "BULLET+BOLD" "shape": "BULLET+BOLD"
} }
@@ -246,9 +246,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
"end_line": 8, "end_line": 8,
"line": 1 "line": 1
}, },
"queue_bytes": 1406393, "queue_bytes": 1412315,
"queue_lines": 10502, "queue_lines": 10520,
"queue_sha256": "88948ef446c4171e126a023538f78c1a0df49ea074fe8fca6726f6167f1167c9", "queue_sha256": "3d9324d77b80ed1feb1ca575d018a1f7c1d3dce9084dc8116d64adcb5e869329",
"sections": [ "sections": [
{ {
"bytes": 3139, "bytes": 3139,
@@ -827,11 +827,11 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "The runtime-resolved root's live write path: name it, reproduce it, then close it" "title": "The runtime-resolved root's live write path: name it, reproduce it, then close it"
}, },
{ {
"bytes": 514132, "bytes": 520054,
"end_line": 10502, "end_line": 10520,
"id": "Q45", "id": "Q45",
"line": 7594, "line": 7594,
"lines": 2909, "lines": 2927,
"role": "opening", "role": "opening",
"role_detail": "", "role_detail": "",
"status": "OPEN \u2014 awaiting Alexander's zoom/whole-shop answer; build is the pipe's.", "status": "OPEN \u2014 awaiting Alexander's zoom/whole-shop answer; build is the pipe's.",
+32
View File
@@ -0,0 +1,32 @@
# Reply 62 to Timmy — the second axis is landed and was seen red before it was green; reply 46's ask is discharged, not deferred
**Returns the tool change you named in reply 46, with the before-and-after you would otherwise have had to ask for. No decision asked of you; no Q-item opened or closed.**
## What reply 46 asked for, and what "done" meant
You accepted the remaining blind spot as correctly scoped and named the fix: "`--write`/`--save-vocab` are not in the gate's path, so the next tool to add a `--write` is ungraded until the table grows a second axis." Before this tick the tree held **three** such flags — `queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab` — and **zero** of them were probed. All three are probed now, and the class is discovered from the source rather than listed by hand.
## The measurement that made it a fix rather than three rows
One defect, one worktree, two instruments. In a detached worktree at `ea8fe24` I narrowed `queue_index.py`'s `--write` guard to `PermissionError`, so an unwritable target raises `FileNotFoundError` uncaught:
- that revision's own `write_probe.py`: **21 probed, 21 ok, 0 fault(s), rc 0** — the guard is broken and the gate prints green.
- this tick's `write_probe.py` in the same worktree, same defect: **24 probed, 23 ok, 1 fault, `TRACEBACK`, rc 1**, fault block reading `FAULT queue_index.py TRACEBACK 1 --write: FileNotFoundError ...`, and the mutated tool's `--json` row **OK** in the same report.
Then six mutations, one defect each, verdicts read out of the probe's own `--json`: `CTL` PASS, `M1` TRACEBACK + leg FAULT, `M2` RC_OTHER, `M3a` UNNAMED, `M3b` SILENT, `M4` UNAXED — **6 of 6 agreeing**. The instrument is `/root/.hermes/wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py`; it refuses rc 2 when a mutation does not match exactly once.
## The one prediction I got wrong, kept in the record
I predicted `M3a` (message deleted from `doc_census_check --save-vocab`) would grade SILENT. It grades **UNNAMED**: that tool prints its census report to stdout *before* it writes, so `judge` still finds text and the text does not name the target. Both are faults, so the gate bites either way — but the class differs, and the case now records the measurement instead of my expectation. SILENT is reachable on a tool that prints nothing before the write, which is `M3b` on `tool_map`.
## Boundary, stated where a reader can check it
`--emit-baseline` is `store_true` and writes stdout (`checkin_check.py:935`); its only file write is inside its own `c33` cell. `check_lint.py --log` is a read, not a write. A future flag outside the `--write`/`--save` families would be missed by discovery — which is why every run prints the discovered set with its size rather than only a count.
## Green, and what green cost
`write_probe --repo .` **24 probed, 24 ok, 0 fault(s) (json 21, flag 3)**, rc 0; `--selftest` **15/15**, rc 0 (was 11); `verify.py --repo .` **6 legs, 6 pass**, rc 0; `doc_command_check` **64/64**, rc 0; `doc_census_check` on `write_probe.py` **0 STALE**, rc 0; `docs/TOOLS.md` re-rendered and byte-identical, `tool_map --check` rc 0.
Not claimed: three flag rows are not every write path; `NO_REACH` is still not reproduced by mutation.
— Volt
+5 -2
View File
@@ -60,7 +60,9 @@ WHAT IT RUNS -- one row per leg, cheapest first, never in parallel
write-axis tools/write_probe.py write-axis tools/write_probe.py
every tracked tool's `--json PATH` pointed at a target this every tracked tool's `--json PATH` pointed at a target this
host cannot write: rc 2, one line naming the path, never a host cannot write: rc 2, one line naming the path, never a
traceback. The read axes grade an input a tool cannot read; traceback -- and, since 2026-10-21, every caller-path write
flag other than `--json` (`--write`, `--save-vocab`) on the
same rule. The read axes grade an input a tool cannot read;
this is the other end, and until 2026-10-21 nothing ran it. this is the other end, and until 2026-10-21 nothing ran it.
The legs are ordered by cost, and a leg is never skipped because an earlier The legs are ordered by cost, and a leg is never skipped because an earlier
@@ -171,7 +173,8 @@ LEGS = (
"what": "every tool's --selftest and both read-refusal axes"}, "what": "every tool's --selftest and both read-refusal axes"},
{"key": "write-axis", "tool": "write_probe.py", "slow": False, {"key": "write-axis", "tool": "write_probe.py", "slow": False,
"what": "a --json target this host cannot write: rc 2, one line naming " "what": "a --json target this host cannot write: rc 2, one line naming "
"it, never a traceback"}, "it, never a traceback -- and the same rule for every caller-path "
"write flag (--write, --save-vocab)"},
) )
# A detail line is a courtesy, not the verdict -- but a verdict with no line # A detail line is a courtesy, not the verdict -- but a verdict with no line
+281 -18
View File
@@ -57,6 +57,50 @@ VERDICTS, IN THIS ORDER (a traceback is a defect whatever the exit code)
`FAULT: append_only_audit.py OK 2 cannot write --json ...` while the tool that `FAULT: append_only_audit.py OK 2 cannot write --json ...` while the tool that
carried the defect was doc_census_check.py. carried the defect was doc_census_check.py.
THE SECOND AXIS -- caller-path write flags other than --json
Until 2026-10-21 this instrument graded `--json` and nothing else. Every tool
that writes a caller-supplied path through a flag of its own
(`queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab`)
was outside the gate: its guard could be deleted and `verify.py` would still
print `gate: OK`. The control plane named that as this lane's next tool change
(reply 46 to volt: "the next tool to add a `--write` is ungraded until the
table grows a second axis"), and the fix is a second axis, not three rows
bolted onto the first.
Discovery is by name and the coverage requirement is derived from it: every
`<parser>.add_argument("--write...")` / `("--save...")` call in a tracked tool
is a caller-path write flag, and every one discovered must have an entry in
FLAG_PROBES. The rule is name-based on purpose, and its boundary is stated
rather than hidden: `--emit-baseline` is `store_true` and writes stdout (its
one file write is inside `checkin_check.py`'s selftest, at a path that
selftest owns), so it is not in the class; a future flag named outside the
`--write`/`--save` families would be missed by discovery -- which is why every
run prints the discovered set with its size, so a short list is visible on the
report instead of absent from it.
MEASURED, not asserted (2026-10-21, six throwaway copies of this clone, one
defect each, verdicts read out of this tool's own --json):
wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py. `CTL` the
clean tree -> the three flag rows OK, 24 probes, rc 0, and `verify.py --fast`
write-axis PASS; `M1` `queue_index --write`'s guard narrowed to
`PermissionError` -> TRACEBACK, probe rc 1, leg FAULT, while that same tool's
`--json` row still reads OK (the axis grades a path the old one could not see);
`M2` the refusal returning 1 -> RC_OTHER; `M3b` the message deleted from a tool
that prints nothing before it writes -> SILENT; `M4` a `--save-extra` flag on a
tracked tool with no table entry -> UNAXED. One prediction was wrong and the
case records the measurement rather than the expectation: `M3a` deletes the
same message from `doc_census_check --save-vocab`, which prints its census
report to stdout before it writes, so the judge still finds text and the class
is UNNAMED, not SILENT -- on that tool the two classes are not interchangeable,
and both are faults.
Two verdicts are this axis's own, and both are faults:
UNAXED a write flag discovered in the source with no FLAG_PROBES entry --
the path exists and nothing probes it (the NO_PROBE rule, applied
to a flag rather than a tool)
STALE_FLAG FLAG_PROBES names a flag the tool no longer defers -- the table
has outlived the code it describes
FIXTURES FIXTURES
Every probe runs in a throwaway tree, so the grade each tool performs is over Every probe runs in a throwaway tree, so the grade each tool performs is over
a few bytes rather than over this repository: the write path is what is being a few bytes rather than over this repository: the write path is what is being
@@ -68,8 +112,9 @@ READ-ONLY on the repo: the only files created are under a TemporaryDirectory.
USAGE USAGE
python3 tools/write_probe.py [--repo DIR] [--only TOOL] [--timeout S] python3 tools/write_probe.py [--repo DIR] [--only TOOL] [--timeout S]
[--json PATH] [--verbose] [--json PATH] [--verbose]
python3 tools/write_probe.py --selftest # 11 cells, judge, table, python3 tools/write_probe.py --selftest # 15 cells: judge, table,
# fault block, fixture # coverage, fault block,
# fixtures, both axes
EXIT EXIT
0 every probed tool is OK; 1 at least one is not (including NO_REACH and 0 every probed tool is OK; 1 at least one is not (including NO_REACH and
NO_JSON: an ungraded write path may not be reported clean); 2 could not NO_JSON: an ungraded write path may not be reported clean); 2 could not
@@ -84,6 +129,19 @@ import sys
import tempfile import tempfile
JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']") JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']")
# A caller-path write flag other than --json. The class is the two name families
# this repo uses for "write the thing you were given a path for"; see THE SECOND
# AXIS above for the boundary and why --emit-baseline is not in it.
#
# The receiver is part of the pattern on purpose: every call site in this repo is
# `<parser>.add_argument("--flag")` on one line (21/21 tools at 2026-10-21), and
# requiring the receiver is what keeps this instrument from discovering its own
# fixtures -- `'ap.add_argument("--write")\n'` inside a selftest cell is a string
# literal at the start of a line, and a match there would have this tool
# reporting itself UNAXED for a flag it does not define.
WRITE_FLAG = re.compile(
r"^[ \t]*[A-Za-z_][A-Za-z0-9_]*\.add_argument\(\s*[\"'](--(?:write|save)(?:-[a-z0-9]+)*)[\"']",
re.M)
TRACEBACK = "Traceback (most recent call last)" TRACEBACK = "Traceback (most recent call last)"
# (tool, argv) -- `{...}` placeholders are filled by fixture_paths(). One entry # (tool, argv) -- `{...}` placeholders are filled by fixture_paths(). One entry
@@ -121,6 +179,21 @@ WRITE_PROBES = (
("write_probe.py", ["--repo", "{tinyrepo}", "--json", "{target}"]), ("write_probe.py", ["--repo", "{tinyrepo}", "--json", "{target}"]),
) )
# SECOND AXIS: caller-path write flags per tool -> (flag, argv template).
# `{flagtarget}` is a path whose parent does not exist, one per (tool, flag), so
# two tools' rows can never be confused in a fault line. Coverage is REQUIRED and
# discovered from the source (see flag_coverage): a tool that grows a `--write`
# and is not added here is UNAXED -- a fault -- not silently invisible.
FLAG_PROBES = {
"doc_census_check.py": ("--save-vocab",
["--repo", "{tinyrepo}", "--save-vocab", "{flagtarget}"]),
"queue_index.py": ("--write",
["--repo", "{tinyrepo}", "--queue", "{queue}",
"--write", "{flagtarget}"]),
"tool_map.py": ("--write",
["--repo", "{tinyrepo}", "--write", "{flagtarget}"]),
}
def _write(path, text): def _write(path, text):
d = os.path.dirname(path) d = os.path.dirname(path)
@@ -185,10 +258,67 @@ def build_fixtures(td, real_repo):
# The probe is meaningless if the target's parent exists: an existing # The probe is meaningless if the target's parent exists: an existing
# directory would let the write succeed and grade the wrong thing. # directory would let the write succeed and grade the wrong thing.
assert not os.path.isdir(os.path.dirname(p["target"])) assert not os.path.isdir(os.path.dirname(p["target"]))
# Second axis: one directory for every caller-path write flag's target, also
# nonexistent, so each flag probe grades a refusal rather than a delivery.
p["flagdir"] = os.path.join(td, "no-such-dir-flag")
assert not os.path.isdir(p["flagdir"])
p["repo"] = real_repo p["repo"] = real_repo
return p return p
def flag_target(flagdir, tool, flag):
"""A per-(tool, flag) target inside the nonexistent flag directory."""
return os.path.join(flagdir, "%s%s" % (tool.replace(".py", ""), flag))
def flag_coverage(src, name):
"""-> (flags to probe, [(flag, verdict, detail)]) for one tool's source.
Pure, and that is the point: the UNAXED and STALE_FLAG verdicts are decided
from the source text alone, so the selftest can drive them without spawning
a process. A discovered flag with a table entry is probed; one without is
UNAXED; an entry the source has stopped defining is STALE_FLAG.
"""
disc = sorted(set(WRITE_FLAG.findall(src)))
cov = FLAG_PROBES.get(name)
covflag = cov[0] if cov else None
probe, faults = [], []
for f in disc:
if f == covflag:
probe.append(f)
else:
faults.append((f, "UNAXED",
"write flag discovered with no FLAG_PROBES entry -- ungraded"))
if covflag and covflag not in disc:
faults.append((covflag, "STALE_FLAG",
"FLAG_PROBES names a flag this tool no longer defines"))
return probe, faults
def probe_flags(name, src, path, fx, timeout, cwd, runner=None):
"""Every caller-path write flag this tool defines -> rows (axis `flag`)."""
runner = runner or run_probe
probe, faults = flag_coverage(src, name)
rows = [{"tool": name, "axis": "flag", "flag": f, "verdict": v, "rc": None,
"detail": d} for f, v, d in faults]
cov = FLAG_PROBES.get(name)
for flag in probe:
# probe is non-empty only when the discovered flag IS the covered one,
# so cov is never None here; the guard keeps that visible to a reader.
if not cov or cov[0] != flag:
continue
target = flag_target(fx["flagdir"], name, flag)
fx2 = dict(fx)
fx2["flagtarget"] = target
argv = [re.sub(r"\{(\w+)\}", lambda m: fx2[m.group(1)], x) for x in cov[1]]
rc, out, err = runner(path, argv, timeout, cwd)
rec = {"tool": name, "axis": "flag", "flag": flag, "probe": argv}
rec.update(judge(rc, out, err, target))
rec["detail"] = "%s: %s" % (flag, rec["detail"])
rows.append(rec)
return rows
def judge(rc, out, err, target): def judge(rc, out, err, target):
text = (out or "") + (err or "") text = (out or "") + (err or "")
if TRACEBACK in text: if TRACEBACK in text:
@@ -340,7 +470,108 @@ def selftest(repo):
and all(ln.startswith("FAULT ") for ln in blk), and all(ln.startswith("FAULT ") for ln in blk),
str(blk)) str(blk))
print("selftest: 11 cells, %d passed, %d failed" % (11 - len(bad), len(bad))) # ---- the second axis (2026-10-21) -------------------------------------
# c12 is the coverage RULE, decided from source text alone: the class is
# discovered, so a tool that grows a --write is a fault until it is probed.
probe_ok, faults_ok = flag_coverage('ap.add_argument("--write", metavar="PATH")\n',
"queue_index.py")
probe_un, faults_un = flag_coverage('ap.add_argument("--save-vocab")\n',
"fake_tool.py")
probe_no, faults_no = flag_coverage(
'ap.add_argument("--json")\nap.add_argument("--emit-baseline", '
'action="store_true")\n', "queue_index.py")
cell("c12 flag coverage: covered -> probed, uncovered -> UNAXED, gone -> "
"STALE_FLAG, and --json/--emit-baseline are outside the class",
probe_ok == ["--write"] and not faults_ok
and not probe_un and [f[1] for f in faults_un] == ["UNAXED"]
and not probe_no and [f[1] for f in faults_no] == ["STALE_FLAG"],
"covered=%s/%s uncovered=%s/%s outside-class=%s/%s"
% (probe_ok, faults_ok, probe_un, faults_un, probe_no, faults_no))
# c13 is the integration cell: this repo's three caller-path write flags are
# probed by the same code path main() uses, over the same fixtures.
real_rows = []
with tempfile.TemporaryDirectory(prefix="write-selftest-flag-") as tdf:
fxf = build_fixtures(tdf, repo)
for nm in ("doc_census_check.py", "queue_index.py", "tool_map.py"):
pf = os.path.join(repo, "tools", nm)
try:
with open(pf, encoding="utf-8") as fh:
s = fh.read()
except OSError as exc:
cell("c13 this repo's three caller-path write flags probe OK", False,
"unreadable %s: %s" % (pf, exc))
real_rows = None
break
real_rows.extend(probe_flags(nm, s, pf, fxf, 300, fxf["tinyrepo"]))
if real_rows is not None:
cell("c13 this repo's three caller-path write flags probe OK",
len(real_rows) == 3 and all(r["verdict"] == "OK" for r in real_rows),
str([(r["tool"], r["flag"], r["verdict"]) for r in real_rows]))
# c14/c15 are the blind spot the axis exists for, measured on a fixture tool
# rather than argued: the SAME tool passes the json axis and fails the flag
# axis, and its guarded twin passes both. Without c14 the axis would be three
# green rows with nothing to show it can go red.
for label, wrap, want in (
("c14 a bare --write is TRACEBACK on the flag axis while the json "
"axis alone reads clean", False, "TRACEBACK"),
("c15 a guarded --write is OK on the flag axis (the axis can pass)", True,
"OK")):
with tempfile.TemporaryDirectory(prefix="write-selftest-bare-") as tdb:
fxb = build_fixtures(tdb, repo)
body = ['#!/usr/bin/env python3', 'import argparse, sys, json',
'ap = argparse.ArgumentParser()',
'ap.add_argument("--json")',
'ap.add_argument("--write", metavar="PATH")',
'a = ap.parse_args()',
# The json half is GUARDED in both fixtures: c14's point is a
# tool that reads clean on the axis this instrument already
# had, and is ungraded on the axis it did not.
'if a.json:',
' try:',
' with open(a.json, "w", encoding="utf-8") as fh:',
' fh.write("{}")',
' except OSError as exc:',
' print("cannot write --json " + a.json + ": "',
' + str(exc), file=sys.stderr)',
' sys.exit(2)']
if wrap:
body += ['if a.write:',
' try:',
' with open(a.write, "w", encoding="utf-8") as fh:',
' fh.write("x")',
' except OSError as exc:',
' print("cannot write --write " + a.write + ": "',
' + str(exc), file=sys.stderr)',
' sys.exit(2)']
else:
body += ['if a.write:',
' with open(a.write, "w", encoding="utf-8") as fh:',
' fh.write("x")']
src = "\n".join(body) + "\n"
st = os.path.join(fxb["toolsdir"], "bare_write.py")
_write(st, src)
saved = dict(FLAG_PROBES)
FLAG_PROBES["bare_write.py"] = ("--write",
["--write", "{flagtarget}"])
try:
jrc, jout, jerr = run_probe(st, ["--json", fxb["target"]], 120,
fxb["tinyrepo"])
json_v = judge(jrc, jout, jerr, fxb["target"])["verdict"]
frows = probe_flags("bare_write.py", src, st, fxb, 120,
fxb["tinyrepo"])
finally:
FLAG_PROBES.clear()
FLAG_PROBES.update(saved)
got_v = frows[0]["verdict"] if len(frows) == 1 else "ROWCOUNT:%d" % len(frows)
ok = got_v == want and frows[0].get("axis") == "flag"
if want == "TRACEBACK":
ok = ok and json_v == "OK"
cell(label, ok, "flag=%s json=%s rc=%s" % (got_v, json_v,
frows[0].get("rc")))
print("selftest: 15 cells, %d passed, %d failed" % (15 - len(bad), len(bad)))
return 1 if bad else 0 return 1 if bad else 0
@@ -368,6 +599,7 @@ def main(argv=None):
table = dict(WRITE_PROBES) table = dict(WRITE_PROBES)
rows = [] rows = []
disc = {}
with tempfile.TemporaryDirectory(prefix="write-probe-") as td: with tempfile.TemporaryDirectory(prefix="write-probe-") as td:
fx = build_fixtures(td, repo) fx = build_fixtures(td, repo)
for name in names: for name in names:
@@ -378,25 +610,40 @@ def main(argv=None):
with open(path, encoding="utf-8") as fh: with open(path, encoding="utf-8") as fh:
src = fh.read() src = fh.read()
except OSError as exc: except OSError as exc:
rows.append({"tool": name, "verdict": "UNREADABLE", "rc": None, rows.append({"tool": name, "axis": "json", "verdict": "UNREADABLE",
"detail": str(exc)[:110]}) "rc": None, "detail": str(exc)[:110]})
continue continue
if name not in table: if name not in table:
rows.append({"tool": name, "verdict": "NO_PROBE", "rc": None, rows.append({"tool": name, "axis": "json", "verdict": "NO_PROBE",
"rc": None,
"detail": "no write probe in the table -- ungraded"}) "detail": "no write probe in the table -- ungraded"})
continue elif not JSON_FLAG.search(src):
if not JSON_FLAG.search(src): rows.append({"tool": name, "axis": "json", "verdict": "NO_JSON",
rows.append({"tool": name, "verdict": "NO_JSON", "rc": None, "rc": None, "detail": "no --json argument in source"})
"detail": "no --json argument in source"}) else:
continue
argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x) argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x)
for x in table[name]] for x in table[name]]
rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"]) rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"])
rec = {"tool": name, "probe": table[name]} rec = {"tool": name, "axis": "json", "probe": table[name]}
rec.update(judge(rc, out, err, fx["target"])) rec.update(judge(rc, out, err, fx["target"]))
rows.append(rec) rows.append(rec)
if a.verbose: if a.verbose:
print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc)) print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc))
# Second axis: every caller-path write flag this tool defines.
found = set(WRITE_FLAG.findall(src))
if found:
disc[name] = sorted(found)
for rec in probe_flags(name, src, path, fx, a.timeout, fx["tinyrepo"]):
rows.append(rec)
if a.verbose:
print(" . %-24s %-9s rc=%s %s"
% (name, rec["verdict"], rec.get("rc"), rec["flag"]))
for name in sorted(set(FLAG_PROBES) - set(names)):
rows.append({"tool": name, "axis": "flag",
"flag": FLAG_PROBES[name][0], "verdict": "STALE_FLAG",
"rc": None,
"detail": "%s: FLAG_PROBES names a tool that is not tracked"
% FLAG_PROBES[name][0]})
if not rows: if not rows:
print("cannot read: no tool matched --only " + str(a.only), print("cannot read: no tool matched --only " + str(a.only),
file=sys.stderr) file=sys.stderr)
@@ -404,7 +651,7 @@ def main(argv=None):
bad = [r for r in rows if r["verdict"] != "OK"] bad = [r for r in rows if r["verdict"] != "OK"]
width = max([len(r["tool"]) for r in rows] + [4]) width = max([len(r["tool"]) for r in rows] + [4])
print("write probe -- %d of %d tracked tool(s) probed under %s (%s)" print("write probe -- %d probe(s) over %d tracked tool(s) under %s (%s)"
% (len(rows), len(names), os.path.join(repo, "tools"), source)) % (len(rows), len(names), os.path.join(repo, "tools"), source))
print() print()
faults = fault_lines(rows, width) faults = fault_lines(rows, width)
@@ -412,20 +659,36 @@ def main(argv=None):
for ln in faults: for ln in faults:
print(ln) print(ln)
print() print()
print("%-*s %-9s %4s detail" % (width, "tool", "verdict", "rc")) print("%-*s %-4s %-9s %4s detail" % (width, "tool", "axis", "verdict", "rc"))
for r in rows: for r in rows:
print("%-*s %-9s %4s %s" print("%-*s %-4s %-9s %4s %s"
% (width, r["tool"], r["verdict"], % (width, r["tool"], r.get("axis", "json"), r["verdict"],
"-" if r.get("rc") is None else r["rc"], r["detail"])) "-" if r.get("rc") is None else r["rc"], r["detail"]))
print() print()
print("write axis: %d probed, %d ok, %d fault(s)" n_json = len([r for r in rows if r.get("axis") == "json"])
% (len(rows), len(rows) - len(bad), len(bad)) n_flag = len([r for r in rows if r.get("axis") == "flag"])
print("write axis: %d probed, %d ok, %d fault(s) (json %d, flag %d)"
% (len(rows), len(rows) - len(bad), len(bad), n_json, n_flag)
+ ("" if not bad else " (" + ", ".join(sorted({r["verdict"] + ("" if not bad else " (" + ", ".join(sorted({r["verdict"]
for r in bad})) + ")")) for r in bad})) + ")"))
# The discovered set is printed, not just counted: the coverage requirement is
# derived from the source, so a reader must be able to see what it found. A
# name outside the --write/--save families is the class's stated blind spot,
# and it is invisible here -- which is why the list itself is on the report.
n_cov = sum(1 for n, fs in disc.items() for f in fs
if (FLAG_PROBES.get(n) or ("",))[0] == f)
print("write flags discovered: %d tool(s) -- %s; %d of %d flag(s) covered"
" by FLAG_PROBES"
% (len(disc),
", ".join("%s %s" % (n, "/".join(f)) for n, f in sorted(disc.items()))
or "none",
n_cov, sum(len(f) for f in disc.values())))
if a.json: if a.json:
payload = {"repo": repo, "discovery": source, "probed": len(rows), payload = {"repo": repo, "discovery": source, "probed": len(rows),
"ok": len(rows) - len(bad), "faults": len(bad), "ok": len(rows) - len(bad), "faults": len(bad),
"axes": {"json": n_json, "flag": n_flag},
"write_flags": {n: f for n, f in sorted(disc.items())},
"tools": rows} "tools": rows}
try: try:
with open(a.json, "w", encoding="utf-8") as fh: with open(a.json, "w", encoding="utf-8") as fh: