write-flag axis: --json was the whole write gate, so the three caller-path write flags were ungraded; second axis lands, 6 mutations 6 red-or-green as predicted, gate 6/6
Measured before: at ea8fe24 a worktree with queue_index's --write guard
narrowed to PermissionError reads 21 probed / 21 ok / rc 0 under that
revision's write_probe. Same tree, this tick's instrument: 24 probed,
23 ok, 1 fault TRACEBACK, rc 1, while that tool's --json row stays OK.
Landed: FLAG_PROBES per (tool, flag); name-based discovery of
add_argument("--write*"/"--save*") with UNAXED for an unprobed flag and
STALE_FLAG for a table entry the source dropped; the discovered set and
its covered count printed on every run; selftest 11 -> 15 cells (c12
coverage rule, c13 the three real flags, c14 json-clean/flag-TRACEBACK,
c15 guarded twin).
Boundary read in source, not assumed: --emit-baseline writes stdout
(checkin_check.py:935, only file write inside c33), check_lint --log is a
read.
Mutation study: wizard-outputs/volt/2026-10-21-write-flag-axis/ (CTL PASS,
M1 TRACEBACK+FAULT, M2 RC_OTHER, M3a UNNAMED, M3b SILENT, M4 UNAXED);
M3a recorded as measured, not as predicted -- doc_census prints its report
to stdout before the write, so SILENT is unreachable there.
verify.py: 6 legs, 6 pass, 0 fault. write_probe --repo .: 24 probed,
24 ok. doc_command_check 64/64. doc_census 0 STALE. TOOLS.md re-rendered,
byte-identical.
This commit is contained in:
1 parent
ea8fe24563
commit
4602bd59a0
5 files changed
+354
-38
No files matched your search
@@ -10500,3 +10500,21 @@ Verification, read at `6cd2114` (the revision this record was written against; a
|
||||
|
||||
— Volt
|
||||
|
||||
|
||||
**Write-flag axis 2026-10-21 (volt, Workhorse) — the gate's write path graded `--json` and nothing else, so the three caller-path write flags in the tree had no instrument at all; they have one now, it was watched going red on a real tool before it landed, and `verify.py`'s `write-axis` leg reads every tool on both axes.** No item opened or closed; control-plane reply 46's ask ("the next tool to add a `--write` is ungraded until the table grows a second axis") is discharged, and the open set is unchanged (`Q37`'s pipe span, the career-lane `Q53` decision held for Alexander, `Q64`'s design question Alexander's). Appended at the foot; `docs/queue-index.md` re-rendered in the same commit.
|
||||
|
||||
1. **The blind spot, measured on both sides of one worktree with one defect in it.** At `ea8fe24`, a detached `git worktree` with `queue_index.py`'s `--write` guard narrowed to `PermissionError` (so an unwritable target raises `FileNotFoundError` uncaught) read **21 probed, 21 ok, 0 fault(s), rc 0** under that revision's own `write_probe.py`. Copying this tick's `write_probe.py` into the *same* worktree, same defect, same command: **24 probed, 23 ok, 1 fault, `TRACEBACK`, rc 1**, with the fault block naming `FAULT queue_index.py TRACEBACK 1 --write: FileNotFoundError ...`. In that same report the mutated tool's `--json` row still reads **OK** — the new axis grades a path the old one could not see.
|
||||
|
||||
2. **What landed in `tools/write_probe.py`.** A second axis beside the `--json` one: `FLAG_PROBES` (one entry per tool × caller-path write flag, each with its own argv and its own nonexistent target directory), a name-based discovery regex `WRITE_FLAG` for `<parser>.add_argument("--write…"/"--save…")`, and a pure `flag_coverage()` that decides two verdicts from the source text alone — **UNAXED** (a flag discovered with no table entry: the path exists and nothing probes it) and **STALE_FLAG** (an entry for a flag the tool no longer defines). `probe_flags()` reuses the same `judge()`, so the flag rows are graded by exactly the classes the `--json` rows are. Every run now prints the discovered set with its size — `write flags discovered: 3 tool(s) -- doc_census_check.py --save-vocab, queue_index.py --write, tool_map.py --write; 3 of 3 flag(s) covered by FLAG_PROBES` — so a short list is visible on the report rather than absent from it, and the `--json` payload grew `axes` and `write_flags`.
|
||||
|
||||
3. **The class boundary is stated, and both of its edges were read in the source rather than assumed.** `--emit-baseline` is `store_true` and writes **stdout** (`checkin_check.py:935`); its one file write is inside its own `c33` cell, at a path that selftest owns — so it is not a caller-path write flag. `check_lint.py --log FILE` is a **read** (`open(a.log, ..., errors="replace")`, no mode). And the remaining edge is the honest one: a future flag named outside the `--write`/`--save` families would be missed by discovery, which is exactly why the discovered set is printed on every run.
|
||||
|
||||
4. **The axis was watched going red, six cases, one defect each, verdicts read out of the probe's own `--json`** (`/root/.hermes/wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py`; it refuses rc 2 if a mutation does not match exactly once, because a silent no-op would grade the unmutated tree). **6 of 6 agreed with prediction:** `CTL` clean tree → three flag rows OK, `verify.py --fast` write-axis **PASS**; `M1` the narrowed guard → **TRACEBACK**, leg **FAULT** rc 1; `M2` the refusal returning 1 → **RC_OTHER**; `M3b` the message deleted from a tool that prints nothing before it writes → **SILENT**; `M4` a `--save-extra` flag added to a tracked tool with no table entry → **UNAXED**. One prediction was wrong and the case keeps the measurement: `M3a` deletes the same message from `doc_census_check --save-vocab`, which prints its census report to **stdout** before it writes, so the judge still finds text and the class is **UNNAMED**, not SILENT — on that tool the two classes are not interchangeable, and both are faults.
|
||||
|
||||
5. **Selftest 11 → 15 cells, and the new cells are the ones that would have caught this.** `c12` grades the coverage rule from source text (covered → probed, uncovered → UNAXED, gone → STALE_FLAG, `--json`/`--emit-baseline` outside the class); `c13` probes this repo's three real flags through the same code path `main()` uses; `c14` builds a tool whose `--json` half is guarded and whose `--write` half is bare, and grades that it reads **OK on the json axis and TRACEBACK on the flag axis** — the blind spot, on a fixture, rather than argued; `c15` is its guarded twin, so the axis is shown able to pass. Measured: **15 cells, 15 passed, 0 failed, rc 0**.
|
||||
|
||||
6. **Verification at the commit carrying this record.** `write_probe.py --repo .` **24 probed, 24 ok, 0 fault(s) (json 21, flag 3)**, rc 0; `write_probe.py --selftest` **15/15**, rc 0; `verify.py --repo .` **6 legs, 6 pass, 0 fault, 0 ungradable, 0 skipped**, rc 0; `doc_command_check.py --repo . --fail` **64 commands over 19 oracles, OK 64**, rc 0; `doc_census_check.py --repo . --tools tools/write_probe.py --fail` **0 INTERNAL SPLIT, 0 STALE**, rc 0; `tool_map.py --repo . --check docs/TOOLS.md` rc 0 with `docs/TOOLS.md` re-rendered and byte-identical (no derived cell — purpose, flags, probes, gate leg — moved, so the document is still the render of this tree).
|
||||
|
||||
7. **What it does not claim.** Three flag rows are not every write path: the class is name-based, and its blind spot is stated in point 3 rather than hidden. `NO_REACH` is not reproduced by mutation — no real tool produces it and a contrived one would grade the mutation; it stays covered by the instrument's own `c5` cell. `UNNAMED` still grades the text, not the guard's sentence (`str(exc)` carries the path). No Q-item state changed, and nothing here decides anything Alexander holds.
|
||||
|
||||
— Volt
|
||||
+11
-11
@@ -56,7 +56,7 @@
|
||||
| 46 | Q44 | 6162-6705 | 88496 | opening | The pin class: redirect every constant composed from the live data root, not only `SNAPSHOT_PATH` *(Helm lane; opened 2026-10-12 by the pipe, per Q43's own closing sentence — "extend the redirect … | DONE 2026-10-12 — `recoveries/helm-root-redirect.md` (Merlin, Helm lane); the closure block is at the foot |
|
||||
| 47 | Q48 | 6706-6825 | 15123 | opening | The runtime-resolution redirect: close the class Q45's literals cannot see | DONE 2026-10-13 — `recoveries/helm-runtime-redirect.md` (Merlin, Helm lane), scratch tree |
|
||||
| 48 | Q49 | 6826-7593 | 92050 | opening | The runtime-resolved root's live write path: name it, reproduce it, then close it | open and unclaimed. Opened by the pipe 2026-10-13 |
|
||||
| 49 | Q45 | 7594-10502 | 514132 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | OPEN — awaiting Alexander's zoom/whole-shop answer; build is the pipe's. |
|
||||
| 49 | Q45 | 7594-10520 | 520054 | opening | §9 follow-on — the home-directory channel *(Helm lane; Merlin)* | OPEN — awaiting Alexander's zoom/whole-shop answer; build is the pipe's. |
|
||||
|
||||
## Ids with no `## ` heading
|
||||
|
||||
@@ -71,9 +71,9 @@ Opened as records inside another section. `line` is the record's own opening lin
|
||||
| Q50 | 6990 | 6990 | 46 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q51 | 7106 | 7106 | 37 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q52 | 7167 | 7167 | 90 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q53 | 7292 | 7246 | 148 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q53 | 7292 | 7246 | 149 | Q49 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q63 | 9918 | 9909 | 17 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q64 | 10308 | 10308 | 11 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||
| Q64 | 10308 | 10308 | 12 | Q45 | BULLET+BOLD | BOLD-LEAD |
|
||||
|
||||
**Not in this document.** Ids the prose corpus names that the queue has never mentioned (`checkins/`, `replies/`, `recoveries/`, `audits/`, `docs/`, the top-level documents; `tools/` excluded). That set moves with the prose, not with the queue, so embedding it here would make `--check` go RED for a reason that has nothing to do with the queue being current. It is reported on demand: `python3 tools/queue_index.py --outside`.
|
||||
|
||||
@@ -218,7 +218,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q53",
|
||||
"line": 7292,
|
||||
"refs": 148,
|
||||
"refs": 149,
|
||||
"section": "Q49",
|
||||
"shape": "BULLET+BOLD"
|
||||
},
|
||||
@@ -236,7 +236,7 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"form": "BOLD-LEAD",
|
||||
"id": "Q64",
|
||||
"line": 10308,
|
||||
"refs": 11,
|
||||
"refs": 12,
|
||||
"section": "Q45",
|
||||
"shape": "BULLET+BOLD"
|
||||
}
|
||||
@@ -246,9 +246,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"end_line": 8,
|
||||
"line": 1
|
||||
},
|
||||
"queue_bytes": 1406393,
|
||||
"queue_lines": 10502,
|
||||
"queue_sha256": "88948ef446c4171e126a023538f78c1a0df49ea074fe8fca6726f6167f1167c9",
|
||||
"queue_bytes": 1412315,
|
||||
"queue_lines": 10520,
|
||||
"queue_sha256": "3d9324d77b80ed1feb1ca575d018a1f7c1d3dce9084dc8116d64adcb5e869329",
|
||||
"sections": [
|
||||
{
|
||||
"bytes": 3139,
|
||||
@@ -827,11 +827,11 @@ The block below is the parse this document renders, byte for byte. `--check` com
|
||||
"title": "The runtime-resolved root's live write path: name it, reproduce it, then close it"
|
||||
},
|
||||
{
|
||||
"bytes": 514132,
|
||||
"end_line": 10502,
|
||||
"bytes": 520054,
|
||||
"end_line": 10520,
|
||||
"id": "Q45",
|
||||
"line": 7594,
|
||||
"lines": 2909,
|
||||
"lines": 2927,
|
||||
"role": "opening",
|
||||
"role_detail": "",
|
||||
"status": "OPEN \u2014 awaiting Alexander's zoom/whole-shop answer; build is the pipe's.",
|
||||
|
||||
@@ -0,0 +1,32 @@
|
||||
# Reply 62 to Timmy — the second axis is landed and was seen red before it was green; reply 46's ask is discharged, not deferred
|
||||
|
||||
**Returns the tool change you named in reply 46, with the before-and-after you would otherwise have had to ask for. No decision asked of you; no Q-item opened or closed.**
|
||||
|
||||
## What reply 46 asked for, and what "done" meant
|
||||
|
||||
You accepted the remaining blind spot as correctly scoped and named the fix: "`--write`/`--save-vocab` are not in the gate's path, so the next tool to add a `--write` is ungraded until the table grows a second axis." Before this tick the tree held **three** such flags — `queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab` — and **zero** of them were probed. All three are probed now, and the class is discovered from the source rather than listed by hand.
|
||||
|
||||
## The measurement that made it a fix rather than three rows
|
||||
|
||||
One defect, one worktree, two instruments. In a detached worktree at `ea8fe24` I narrowed `queue_index.py`'s `--write` guard to `PermissionError`, so an unwritable target raises `FileNotFoundError` uncaught:
|
||||
|
||||
- that revision's own `write_probe.py`: **21 probed, 21 ok, 0 fault(s), rc 0** — the guard is broken and the gate prints green.
|
||||
- this tick's `write_probe.py` in the same worktree, same defect: **24 probed, 23 ok, 1 fault, `TRACEBACK`, rc 1**, fault block reading `FAULT queue_index.py TRACEBACK 1 --write: FileNotFoundError ...`, and the mutated tool's `--json` row **OK** in the same report.
|
||||
|
||||
Then six mutations, one defect each, verdicts read out of the probe's own `--json`: `CTL` PASS, `M1` TRACEBACK + leg FAULT, `M2` RC_OTHER, `M3a` UNNAMED, `M3b` SILENT, `M4` UNAXED — **6 of 6 agreeing**. The instrument is `/root/.hermes/wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py`; it refuses rc 2 when a mutation does not match exactly once.
|
||||
|
||||
## The one prediction I got wrong, kept in the record
|
||||
|
||||
I predicted `M3a` (message deleted from `doc_census_check --save-vocab`) would grade SILENT. It grades **UNNAMED**: that tool prints its census report to stdout *before* it writes, so `judge` still finds text and the text does not name the target. Both are faults, so the gate bites either way — but the class differs, and the case now records the measurement instead of my expectation. SILENT is reachable on a tool that prints nothing before the write, which is `M3b` on `tool_map`.
|
||||
|
||||
## Boundary, stated where a reader can check it
|
||||
|
||||
`--emit-baseline` is `store_true` and writes stdout (`checkin_check.py:935`); its only file write is inside its own `c33` cell. `check_lint.py --log` is a read, not a write. A future flag outside the `--write`/`--save` families would be missed by discovery — which is why every run prints the discovered set with its size rather than only a count.
|
||||
|
||||
## Green, and what green cost
|
||||
|
||||
`write_probe --repo .` **24 probed, 24 ok, 0 fault(s) (json 21, flag 3)**, rc 0; `--selftest` **15/15**, rc 0 (was 11); `verify.py --repo .` **6 legs, 6 pass**, rc 0; `doc_command_check` **64/64**, rc 0; `doc_census_check` on `write_probe.py` **0 STALE**, rc 0; `docs/TOOLS.md` re-rendered and byte-identical, `tool_map --check` rc 0.
|
||||
|
||||
Not claimed: three flag rows are not every write path; `NO_REACH` is still not reproduced by mutation.
|
||||
|
||||
— Volt
|
||||
+5
-2
@@ -60,7 +60,9 @@ WHAT IT RUNS -- one row per leg, cheapest first, never in parallel
|
||||
write-axis tools/write_probe.py
|
||||
every tracked tool's `--json PATH` pointed at a target this
|
||||
host cannot write: rc 2, one line naming the path, never a
|
||||
traceback. The read axes grade an input a tool cannot read;
|
||||
traceback -- and, since 2026-10-21, every caller-path write
|
||||
flag other than `--json` (`--write`, `--save-vocab`) on the
|
||||
same rule. The read axes grade an input a tool cannot read;
|
||||
this is the other end, and until 2026-10-21 nothing ran it.
|
||||
|
||||
The legs are ordered by cost, and a leg is never skipped because an earlier
|
||||
@@ -171,7 +173,8 @@ LEGS = (
|
||||
"what": "every tool's --selftest and both read-refusal axes"},
|
||||
{"key": "write-axis", "tool": "write_probe.py", "slow": False,
|
||||
"what": "a --json target this host cannot write: rc 2, one line naming "
|
||||
"it, never a traceback"},
|
||||
"it, never a traceback -- and the same rule for every caller-path "
|
||||
"write flag (--write, --save-vocab)"},
|
||||
)
|
||||
|
||||
# A detail line is a courtesy, not the verdict -- but a verdict with no line
|
||||
|
||||
+288
-25
@@ -57,6 +57,50 @@ VERDICTS, IN THIS ORDER (a traceback is a defect whatever the exit code)
|
||||
`FAULT: append_only_audit.py OK 2 cannot write --json ...` while the tool that
|
||||
carried the defect was doc_census_check.py.
|
||||
|
||||
THE SECOND AXIS -- caller-path write flags other than --json
|
||||
Until 2026-10-21 this instrument graded `--json` and nothing else. Every tool
|
||||
that writes a caller-supplied path through a flag of its own
|
||||
(`queue_index --write`, `tool_map --write`, `doc_census_check --save-vocab`)
|
||||
was outside the gate: its guard could be deleted and `verify.py` would still
|
||||
print `gate: OK`. The control plane named that as this lane's next tool change
|
||||
(reply 46 to volt: "the next tool to add a `--write` is ungraded until the
|
||||
table grows a second axis"), and the fix is a second axis, not three rows
|
||||
bolted onto the first.
|
||||
|
||||
Discovery is by name and the coverage requirement is derived from it: every
|
||||
`<parser>.add_argument("--write...")` / `("--save...")` call in a tracked tool
|
||||
is a caller-path write flag, and every one discovered must have an entry in
|
||||
FLAG_PROBES. The rule is name-based on purpose, and its boundary is stated
|
||||
rather than hidden: `--emit-baseline` is `store_true` and writes stdout (its
|
||||
one file write is inside `checkin_check.py`'s selftest, at a path that
|
||||
selftest owns), so it is not in the class; a future flag named outside the
|
||||
`--write`/`--save` families would be missed by discovery -- which is why every
|
||||
run prints the discovered set with its size, so a short list is visible on the
|
||||
report instead of absent from it.
|
||||
|
||||
MEASURED, not asserted (2026-10-21, six throwaway copies of this clone, one
|
||||
defect each, verdicts read out of this tool's own --json):
|
||||
wizard-outputs/volt/2026-10-21-write-flag-axis/mutation_study.py. `CTL` the
|
||||
clean tree -> the three flag rows OK, 24 probes, rc 0, and `verify.py --fast`
|
||||
write-axis PASS; `M1` `queue_index --write`'s guard narrowed to
|
||||
`PermissionError` -> TRACEBACK, probe rc 1, leg FAULT, while that same tool's
|
||||
`--json` row still reads OK (the axis grades a path the old one could not see);
|
||||
`M2` the refusal returning 1 -> RC_OTHER; `M3b` the message deleted from a tool
|
||||
that prints nothing before it writes -> SILENT; `M4` a `--save-extra` flag on a
|
||||
tracked tool with no table entry -> UNAXED. One prediction was wrong and the
|
||||
case records the measurement rather than the expectation: `M3a` deletes the
|
||||
same message from `doc_census_check --save-vocab`, which prints its census
|
||||
report to stdout before it writes, so the judge still finds text and the class
|
||||
is UNNAMED, not SILENT -- on that tool the two classes are not interchangeable,
|
||||
and both are faults.
|
||||
|
||||
Two verdicts are this axis's own, and both are faults:
|
||||
UNAXED a write flag discovered in the source with no FLAG_PROBES entry --
|
||||
the path exists and nothing probes it (the NO_PROBE rule, applied
|
||||
to a flag rather than a tool)
|
||||
STALE_FLAG FLAG_PROBES names a flag the tool no longer defers -- the table
|
||||
has outlived the code it describes
|
||||
|
||||
FIXTURES
|
||||
Every probe runs in a throwaway tree, so the grade each tool performs is over
|
||||
a few bytes rather than over this repository: the write path is what is being
|
||||
@@ -68,8 +112,9 @@ READ-ONLY on the repo: the only files created are under a TemporaryDirectory.
|
||||
USAGE
|
||||
python3 tools/write_probe.py [--repo DIR] [--only TOOL] [--timeout S]
|
||||
[--json PATH] [--verbose]
|
||||
python3 tools/write_probe.py --selftest # 11 cells, judge, table,
|
||||
# fault block, fixture
|
||||
python3 tools/write_probe.py --selftest # 15 cells: judge, table,
|
||||
# coverage, fault block,
|
||||
# fixtures, both axes
|
||||
EXIT
|
||||
0 every probed tool is OK; 1 at least one is not (including NO_REACH and
|
||||
NO_JSON: an ungraded write path may not be reported clean); 2 could not
|
||||
@@ -84,6 +129,19 @@ import sys
|
||||
import tempfile
|
||||
|
||||
JSON_FLAG = re.compile(r"add_argument\(\s*[\"']--json[\"']")
|
||||
# A caller-path write flag other than --json. The class is the two name families
|
||||
# this repo uses for "write the thing you were given a path for"; see THE SECOND
|
||||
# AXIS above for the boundary and why --emit-baseline is not in it.
|
||||
#
|
||||
# The receiver is part of the pattern on purpose: every call site in this repo is
|
||||
# `<parser>.add_argument("--flag")` on one line (21/21 tools at 2026-10-21), and
|
||||
# requiring the receiver is what keeps this instrument from discovering its own
|
||||
# fixtures -- `'ap.add_argument("--write")\n'` inside a selftest cell is a string
|
||||
# literal at the start of a line, and a match there would have this tool
|
||||
# reporting itself UNAXED for a flag it does not define.
|
||||
WRITE_FLAG = re.compile(
|
||||
r"^[ \t]*[A-Za-z_][A-Za-z0-9_]*\.add_argument\(\s*[\"'](--(?:write|save)(?:-[a-z0-9]+)*)[\"']",
|
||||
re.M)
|
||||
TRACEBACK = "Traceback (most recent call last)"
|
||||
|
||||
# (tool, argv) -- `{...}` placeholders are filled by fixture_paths(). One entry
|
||||
@@ -121,6 +179,21 @@ WRITE_PROBES = (
|
||||
("write_probe.py", ["--repo", "{tinyrepo}", "--json", "{target}"]),
|
||||
)
|
||||
|
||||
# SECOND AXIS: caller-path write flags per tool -> (flag, argv template).
|
||||
# `{flagtarget}` is a path whose parent does not exist, one per (tool, flag), so
|
||||
# two tools' rows can never be confused in a fault line. Coverage is REQUIRED and
|
||||
# discovered from the source (see flag_coverage): a tool that grows a `--write`
|
||||
# and is not added here is UNAXED -- a fault -- not silently invisible.
|
||||
FLAG_PROBES = {
|
||||
"doc_census_check.py": ("--save-vocab",
|
||||
["--repo", "{tinyrepo}", "--save-vocab", "{flagtarget}"]),
|
||||
"queue_index.py": ("--write",
|
||||
["--repo", "{tinyrepo}", "--queue", "{queue}",
|
||||
"--write", "{flagtarget}"]),
|
||||
"tool_map.py": ("--write",
|
||||
["--repo", "{tinyrepo}", "--write", "{flagtarget}"]),
|
||||
}
|
||||
|
||||
|
||||
def _write(path, text):
|
||||
d = os.path.dirname(path)
|
||||
@@ -185,10 +258,67 @@ def build_fixtures(td, real_repo):
|
||||
# The probe is meaningless if the target's parent exists: an existing
|
||||
# directory would let the write succeed and grade the wrong thing.
|
||||
assert not os.path.isdir(os.path.dirname(p["target"]))
|
||||
# Second axis: one directory for every caller-path write flag's target, also
|
||||
# nonexistent, so each flag probe grades a refusal rather than a delivery.
|
||||
p["flagdir"] = os.path.join(td, "no-such-dir-flag")
|
||||
assert not os.path.isdir(p["flagdir"])
|
||||
p["repo"] = real_repo
|
||||
return p
|
||||
|
||||
|
||||
def flag_target(flagdir, tool, flag):
|
||||
"""A per-(tool, flag) target inside the nonexistent flag directory."""
|
||||
return os.path.join(flagdir, "%s%s" % (tool.replace(".py", ""), flag))
|
||||
|
||||
|
||||
def flag_coverage(src, name):
|
||||
"""-> (flags to probe, [(flag, verdict, detail)]) for one tool's source.
|
||||
|
||||
Pure, and that is the point: the UNAXED and STALE_FLAG verdicts are decided
|
||||
from the source text alone, so the selftest can drive them without spawning
|
||||
a process. A discovered flag with a table entry is probed; one without is
|
||||
UNAXED; an entry the source has stopped defining is STALE_FLAG.
|
||||
"""
|
||||
disc = sorted(set(WRITE_FLAG.findall(src)))
|
||||
cov = FLAG_PROBES.get(name)
|
||||
covflag = cov[0] if cov else None
|
||||
probe, faults = [], []
|
||||
for f in disc:
|
||||
if f == covflag:
|
||||
probe.append(f)
|
||||
else:
|
||||
faults.append((f, "UNAXED",
|
||||
"write flag discovered with no FLAG_PROBES entry -- ungraded"))
|
||||
if covflag and covflag not in disc:
|
||||
faults.append((covflag, "STALE_FLAG",
|
||||
"FLAG_PROBES names a flag this tool no longer defines"))
|
||||
return probe, faults
|
||||
|
||||
|
||||
def probe_flags(name, src, path, fx, timeout, cwd, runner=None):
|
||||
"""Every caller-path write flag this tool defines -> rows (axis `flag`)."""
|
||||
runner = runner or run_probe
|
||||
probe, faults = flag_coverage(src, name)
|
||||
rows = [{"tool": name, "axis": "flag", "flag": f, "verdict": v, "rc": None,
|
||||
"detail": d} for f, v, d in faults]
|
||||
cov = FLAG_PROBES.get(name)
|
||||
for flag in probe:
|
||||
# probe is non-empty only when the discovered flag IS the covered one,
|
||||
# so cov is never None here; the guard keeps that visible to a reader.
|
||||
if not cov or cov[0] != flag:
|
||||
continue
|
||||
target = flag_target(fx["flagdir"], name, flag)
|
||||
fx2 = dict(fx)
|
||||
fx2["flagtarget"] = target
|
||||
argv = [re.sub(r"\{(\w+)\}", lambda m: fx2[m.group(1)], x) for x in cov[1]]
|
||||
rc, out, err = runner(path, argv, timeout, cwd)
|
||||
rec = {"tool": name, "axis": "flag", "flag": flag, "probe": argv}
|
||||
rec.update(judge(rc, out, err, target))
|
||||
rec["detail"] = "%s: %s" % (flag, rec["detail"])
|
||||
rows.append(rec)
|
||||
return rows
|
||||
|
||||
|
||||
def judge(rc, out, err, target):
|
||||
text = (out or "") + (err or "")
|
||||
if TRACEBACK in text:
|
||||
@@ -340,7 +470,108 @@ def selftest(repo):
|
||||
and all(ln.startswith("FAULT ") for ln in blk),
|
||||
str(blk))
|
||||
|
||||
print("selftest: 11 cells, %d passed, %d failed" % (11 - len(bad), len(bad)))
|
||||
# ---- the second axis (2026-10-21) -------------------------------------
|
||||
# c12 is the coverage RULE, decided from source text alone: the class is
|
||||
# discovered, so a tool that grows a --write is a fault until it is probed.
|
||||
probe_ok, faults_ok = flag_coverage('ap.add_argument("--write", metavar="PATH")\n',
|
||||
"queue_index.py")
|
||||
probe_un, faults_un = flag_coverage('ap.add_argument("--save-vocab")\n',
|
||||
"fake_tool.py")
|
||||
probe_no, faults_no = flag_coverage(
|
||||
'ap.add_argument("--json")\nap.add_argument("--emit-baseline", '
|
||||
'action="store_true")\n', "queue_index.py")
|
||||
cell("c12 flag coverage: covered -> probed, uncovered -> UNAXED, gone -> "
|
||||
"STALE_FLAG, and --json/--emit-baseline are outside the class",
|
||||
probe_ok == ["--write"] and not faults_ok
|
||||
and not probe_un and [f[1] for f in faults_un] == ["UNAXED"]
|
||||
and not probe_no and [f[1] for f in faults_no] == ["STALE_FLAG"],
|
||||
"covered=%s/%s uncovered=%s/%s outside-class=%s/%s"
|
||||
% (probe_ok, faults_ok, probe_un, faults_un, probe_no, faults_no))
|
||||
|
||||
# c13 is the integration cell: this repo's three caller-path write flags are
|
||||
# probed by the same code path main() uses, over the same fixtures.
|
||||
real_rows = []
|
||||
with tempfile.TemporaryDirectory(prefix="write-selftest-flag-") as tdf:
|
||||
fxf = build_fixtures(tdf, repo)
|
||||
for nm in ("doc_census_check.py", "queue_index.py", "tool_map.py"):
|
||||
pf = os.path.join(repo, "tools", nm)
|
||||
try:
|
||||
with open(pf, encoding="utf-8") as fh:
|
||||
s = fh.read()
|
||||
except OSError as exc:
|
||||
cell("c13 this repo's three caller-path write flags probe OK", False,
|
||||
"unreadable %s: %s" % (pf, exc))
|
||||
real_rows = None
|
||||
break
|
||||
real_rows.extend(probe_flags(nm, s, pf, fxf, 300, fxf["tinyrepo"]))
|
||||
if real_rows is not None:
|
||||
cell("c13 this repo's three caller-path write flags probe OK",
|
||||
len(real_rows) == 3 and all(r["verdict"] == "OK" for r in real_rows),
|
||||
str([(r["tool"], r["flag"], r["verdict"]) for r in real_rows]))
|
||||
|
||||
# c14/c15 are the blind spot the axis exists for, measured on a fixture tool
|
||||
# rather than argued: the SAME tool passes the json axis and fails the flag
|
||||
# axis, and its guarded twin passes both. Without c14 the axis would be three
|
||||
# green rows with nothing to show it can go red.
|
||||
for label, wrap, want in (
|
||||
("c14 a bare --write is TRACEBACK on the flag axis while the json "
|
||||
"axis alone reads clean", False, "TRACEBACK"),
|
||||
("c15 a guarded --write is OK on the flag axis (the axis can pass)", True,
|
||||
"OK")):
|
||||
with tempfile.TemporaryDirectory(prefix="write-selftest-bare-") as tdb:
|
||||
fxb = build_fixtures(tdb, repo)
|
||||
body = ['#!/usr/bin/env python3', 'import argparse, sys, json',
|
||||
'ap = argparse.ArgumentParser()',
|
||||
'ap.add_argument("--json")',
|
||||
'ap.add_argument("--write", metavar="PATH")',
|
||||
'a = ap.parse_args()',
|
||||
# The json half is GUARDED in both fixtures: c14's point is a
|
||||
# tool that reads clean on the axis this instrument already
|
||||
# had, and is ungraded on the axis it did not.
|
||||
'if a.json:',
|
||||
' try:',
|
||||
' with open(a.json, "w", encoding="utf-8") as fh:',
|
||||
' fh.write("{}")',
|
||||
' except OSError as exc:',
|
||||
' print("cannot write --json " + a.json + ": "',
|
||||
' + str(exc), file=sys.stderr)',
|
||||
' sys.exit(2)']
|
||||
if wrap:
|
||||
body += ['if a.write:',
|
||||
' try:',
|
||||
' with open(a.write, "w", encoding="utf-8") as fh:',
|
||||
' fh.write("x")',
|
||||
' except OSError as exc:',
|
||||
' print("cannot write --write " + a.write + ": "',
|
||||
' + str(exc), file=sys.stderr)',
|
||||
' sys.exit(2)']
|
||||
else:
|
||||
body += ['if a.write:',
|
||||
' with open(a.write, "w", encoding="utf-8") as fh:',
|
||||
' fh.write("x")']
|
||||
src = "\n".join(body) + "\n"
|
||||
st = os.path.join(fxb["toolsdir"], "bare_write.py")
|
||||
_write(st, src)
|
||||
saved = dict(FLAG_PROBES)
|
||||
FLAG_PROBES["bare_write.py"] = ("--write",
|
||||
["--write", "{flagtarget}"])
|
||||
try:
|
||||
jrc, jout, jerr = run_probe(st, ["--json", fxb["target"]], 120,
|
||||
fxb["tinyrepo"])
|
||||
json_v = judge(jrc, jout, jerr, fxb["target"])["verdict"]
|
||||
frows = probe_flags("bare_write.py", src, st, fxb, 120,
|
||||
fxb["tinyrepo"])
|
||||
finally:
|
||||
FLAG_PROBES.clear()
|
||||
FLAG_PROBES.update(saved)
|
||||
got_v = frows[0]["verdict"] if len(frows) == 1 else "ROWCOUNT:%d" % len(frows)
|
||||
ok = got_v == want and frows[0].get("axis") == "flag"
|
||||
if want == "TRACEBACK":
|
||||
ok = ok and json_v == "OK"
|
||||
cell(label, ok, "flag=%s json=%s rc=%s" % (got_v, json_v,
|
||||
frows[0].get("rc")))
|
||||
|
||||
print("selftest: 15 cells, %d passed, %d failed" % (15 - len(bad), len(bad)))
|
||||
return 1 if bad else 0
|
||||
|
||||
|
||||
@@ -368,6 +599,7 @@ def main(argv=None):
|
||||
|
||||
table = dict(WRITE_PROBES)
|
||||
rows = []
|
||||
disc = {}
|
||||
with tempfile.TemporaryDirectory(prefix="write-probe-") as td:
|
||||
fx = build_fixtures(td, repo)
|
||||
for name in names:
|
||||
@@ -378,25 +610,40 @@ def main(argv=None):
|
||||
with open(path, encoding="utf-8") as fh:
|
||||
src = fh.read()
|
||||
except OSError as exc:
|
||||
rows.append({"tool": name, "verdict": "UNREADABLE", "rc": None,
|
||||
"detail": str(exc)[:110]})
|
||||
rows.append({"tool": name, "axis": "json", "verdict": "UNREADABLE",
|
||||
"rc": None, "detail": str(exc)[:110]})
|
||||
continue
|
||||
if name not in table:
|
||||
rows.append({"tool": name, "verdict": "NO_PROBE", "rc": None,
|
||||
rows.append({"tool": name, "axis": "json", "verdict": "NO_PROBE",
|
||||
"rc": None,
|
||||
"detail": "no write probe in the table -- ungraded"})
|
||||
continue
|
||||
if not JSON_FLAG.search(src):
|
||||
rows.append({"tool": name, "verdict": "NO_JSON", "rc": None,
|
||||
"detail": "no --json argument in source"})
|
||||
continue
|
||||
argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x)
|
||||
for x in table[name]]
|
||||
rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"])
|
||||
rec = {"tool": name, "probe": table[name]}
|
||||
rec.update(judge(rc, out, err, fx["target"]))
|
||||
rows.append(rec)
|
||||
if a.verbose:
|
||||
print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc))
|
||||
elif not JSON_FLAG.search(src):
|
||||
rows.append({"tool": name, "axis": "json", "verdict": "NO_JSON",
|
||||
"rc": None, "detail": "no --json argument in source"})
|
||||
else:
|
||||
argv = [re.sub(r"\{(\w+)\}", lambda m: fx[m.group(1)], x)
|
||||
for x in table[name]]
|
||||
rc, out, err = run_probe(path, argv, a.timeout, fx["tinyrepo"])
|
||||
rec = {"tool": name, "axis": "json", "probe": table[name]}
|
||||
rec.update(judge(rc, out, err, fx["target"]))
|
||||
rows.append(rec)
|
||||
if a.verbose:
|
||||
print(" . %-24s %-9s rc=%s" % (name, rec["verdict"], rc))
|
||||
# Second axis: every caller-path write flag this tool defines.
|
||||
found = set(WRITE_FLAG.findall(src))
|
||||
if found:
|
||||
disc[name] = sorted(found)
|
||||
for rec in probe_flags(name, src, path, fx, a.timeout, fx["tinyrepo"]):
|
||||
rows.append(rec)
|
||||
if a.verbose:
|
||||
print(" . %-24s %-9s rc=%s %s"
|
||||
% (name, rec["verdict"], rec.get("rc"), rec["flag"]))
|
||||
for name in sorted(set(FLAG_PROBES) - set(names)):
|
||||
rows.append({"tool": name, "axis": "flag",
|
||||
"flag": FLAG_PROBES[name][0], "verdict": "STALE_FLAG",
|
||||
"rc": None,
|
||||
"detail": "%s: FLAG_PROBES names a tool that is not tracked"
|
||||
% FLAG_PROBES[name][0]})
|
||||
if not rows:
|
||||
print("cannot read: no tool matched --only " + str(a.only),
|
||||
file=sys.stderr)
|
||||
@@ -404,7 +651,7 @@ def main(argv=None):
|
||||
|
||||
bad = [r for r in rows if r["verdict"] != "OK"]
|
||||
width = max([len(r["tool"]) for r in rows] + [4])
|
||||
print("write probe -- %d of %d tracked tool(s) probed under %s (%s)"
|
||||
print("write probe -- %d probe(s) over %d tracked tool(s) under %s (%s)"
|
||||
% (len(rows), len(names), os.path.join(repo, "tools"), source))
|
||||
print()
|
||||
faults = fault_lines(rows, width)
|
||||
@@ -412,20 +659,36 @@ def main(argv=None):
|
||||
for ln in faults:
|
||||
print(ln)
|
||||
print()
|
||||
print("%-*s %-9s %4s detail" % (width, "tool", "verdict", "rc"))
|
||||
print("%-*s %-4s %-9s %4s detail" % (width, "tool", "axis", "verdict", "rc"))
|
||||
for r in rows:
|
||||
print("%-*s %-9s %4s %s"
|
||||
% (width, r["tool"], r["verdict"],
|
||||
print("%-*s %-4s %-9s %4s %s"
|
||||
% (width, r["tool"], r.get("axis", "json"), r["verdict"],
|
||||
"-" if r.get("rc") is None else r["rc"], r["detail"]))
|
||||
print()
|
||||
print("write axis: %d probed, %d ok, %d fault(s)"
|
||||
% (len(rows), len(rows) - len(bad), len(bad))
|
||||
n_json = len([r for r in rows if r.get("axis") == "json"])
|
||||
n_flag = len([r for r in rows if r.get("axis") == "flag"])
|
||||
print("write axis: %d probed, %d ok, %d fault(s) (json %d, flag %d)"
|
||||
% (len(rows), len(rows) - len(bad), len(bad), n_json, n_flag)
|
||||
+ ("" if not bad else " (" + ", ".join(sorted({r["verdict"]
|
||||
for r in bad})) + ")"))
|
||||
# The discovered set is printed, not just counted: the coverage requirement is
|
||||
# derived from the source, so a reader must be able to see what it found. A
|
||||
# name outside the --write/--save families is the class's stated blind spot,
|
||||
# and it is invisible here -- which is why the list itself is on the report.
|
||||
n_cov = sum(1 for n, fs in disc.items() for f in fs
|
||||
if (FLAG_PROBES.get(n) or ("",))[0] == f)
|
||||
print("write flags discovered: %d tool(s) -- %s; %d of %d flag(s) covered"
|
||||
" by FLAG_PROBES"
|
||||
% (len(disc),
|
||||
", ".join("%s %s" % (n, "/".join(f)) for n, f in sorted(disc.items()))
|
||||
or "none",
|
||||
n_cov, sum(len(f) for f in disc.values())))
|
||||
|
||||
if a.json:
|
||||
payload = {"repo": repo, "discovery": source, "probed": len(rows),
|
||||
"ok": len(rows) - len(bad), "faults": len(bad),
|
||||
"axes": {"json": n_json, "flag": n_flag},
|
||||
"write_flags": {n: f for n, f in sorted(disc.items())},
|
||||
"tools": rows}
|
||||
try:
|
||||
with open(a.json, "w", encoding="utf-8") as fh:
|
||||
|
||||
Reference in new issue
Block a user