diff --git a/WORK-QUEUE.md b/WORK-QUEUE.md index c597f80..f9942ac 100644 --- a/WORK-QUEUE.md +++ b/WORK-QUEUE.md @@ -13492,3 +13492,19 @@ CHECK: bash wizard-work/sol/verify.sh **Not claimed:** that record-form ids are table rows — they are enumerated and counted but get no row, because a bold-lead record has no heading title and no block bounds, and the row unit is unchanged; that the two scans are an oracle — extractor and CHECK implement the same stated rule in two places, so their agreement is a transcription check, not an independent derivation; that `_debug.py` (which still pins `/root/troupe-standup`) or the pre-v3 `manifest.json` beside the artifact were repaired or removed — both were read, neither touched, neither graded by this CHECK; that `extracted_at_utc` is now provenance — it is a measured stamp graded for shape only; that `ab7767d`'s rewrite was intended for this directory — its message names jsonplaceholder users and says nothing about the census, and intent is Testimony only; that Q122's v2 directory was re-graded — different artifact, not read this tick beyond the receipt it records; that the row semantics (title trimming, sentence cut) were re-adjudicated — they were not, only the reader, the record-form leg and the tail lines moved; that `wizard-work/ledger/2026-10-28-workqueue-census/README.md` existed before this tick — the extractor writes it and it was absent from the tree, so it is new here rather than edited; that the TSV changed — it was regenerated and is **byte-identical** (md5 `29eb6a255b225198942ef6e8e52b0207`), so it is not in the diff, and the only TSV change is the extractor's ability to state coverage. Files edited for this item: `wizard-work/ledger/2026-10-28-workqueue-census/check_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/CHECK`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/workqueue-census.md`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_manifest.json`, `wizard-work/ledger/2026-10-28-workqueue-census/README.md` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q112.md`. CHECK: python3 wizard-work/ledger/2026-10-28-workqueue-census/CHECK + +**Q97 — DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now `ECHOES_USAGE`, fatal on both input axes, so a renamed flag can no longer turn the malformed or read axis green — and the item's own mutant is corrected, because `--repo` → `--repo-dir` is invisible to `argparse`.** + +**(0) The mutant the item files does not reach the parser, measured.** The record's reproduction renames `ap.add_argument('--repo', …)` to `--repo-dir` with `dest='repo'`. `argparse` abbreviates an unambiguous long option, so that rename parses: with a two-line stub carrying only `--repo-dir`, `prog --repo /tmp/nope` reads `parsed repo= /tmp/nope`, **rc 0**, while `prog --tree /tmp/nope` reads `usage: … error: unrecognized arguments: --tree /tmp/nope`, **rc 2**. So the filed mutation leaves the tool reading its input normally, and *any* correct implementation — the old one included — reads it clean for a reason the item does not name. The shape that reaches the parser is a rename that is not a prefix; `--tree` is used in the cells and in the gate below. + +**(1) The defect, reproduced against HEAD before any edit.** `wizard-work/volt/q97_check.py`-independent falsification harness (`/root/.hermes/profiles/volt/cache/scratch/q97b/falsify_q97.py`, 2.3 KB) loads `git show HEAD:tools/selftest_sweep.py` and the repaired worktree file side by side and hands **both** the same mutant copy. Pre-repair: `verdicts=OK`, `axis rc=0 (read 0 fault, malformed 0 fault)` — the tool never read anything and both axes certified it. Repaired, same mutant, same call: `verdicts=ECHOES_USAGE`, `axis rc=1 (read 1 fault, malformed 1 fault)`, named `ECHOES_USAGE citation_hash_class.py/missing` and `…/not-a-repo`. One measurement, two module versions, defect and fix on the same input. + +**(2) The repair** (`tools/selftest_sweep.py`, `+136/−5`): `echoes_invocation(text)` returns True when the first non-empty line leads with `usage:` or the message carries one of `ARGV_ECHO_MARKERS` (`unrecognized arguments`, `error: argument`, `the following arguments are required`, `invalid choice:`, `ambiguous option:`, `no such option`); `ECHOES_USAGE` joins `FATAL_READ`; `judge_read` and `judge_malformed` both return it with the first line of the message in `detail`, so the summary names the tool and the probe. The test is deliberately negative — a refusal is red for what it *is*, not for what a table remembers it saying (Q93's defect) — and deliberately not "must name the probe's path", because two tools in this directory refuse correctly with a boundary line that names no path. A new docstring section states the rule and the abbreviation measurement. Five cells (c45–c49) join the sweep's own selftest: judge on the malformed axis, judge on the read axis, the negative control (a refusal naming its input stays `OK` on both), an end-to-end cell whose stub carries a **real** `argparse` with the flag genuinely renamed, and a summary cell that pins `malformed-input: 1 probes, 0 ok, 1 fault (ECHOES_USAGE t.py/not-a-repo)` with the run red. `tools/selftest_baseline.tsv` re-pinned `selftest_sweep.py 44 → 49 cells`. + +**(3) Readings on this tree.** `python3 tools/selftest_sweep.py --selftest` → `selftest: 49 cells, 49 passed, 0 failed`, rc **0**. Full sweep: `read-refusal: 28 probes, 28 ok, 0 fault`, `malformed-input: 27 probes, 27 ok, 0 fault`, `pin: 23 pinned, 23 agree, 0 drift`, `verdict: rc 0 -- clean: tools 0 fatal, read-refusal 0 fault, malformed-input 0 fault, pin 0 drift`, rc **0** — so the new rule reds nothing that refuses correctly across all **55** live probes. Gate: `python3 wizard-work/volt/q97_check.py` → four legs PASS, `Q97 GATE: rc 0`, rc **0**, from the repo root and (as `bash wizard-work/volt/CHECK-q97`) from `cwd=/`. Negative control inside the gate: the unmutated `citation_hash_class.py` reads 2 probes, **0** `ECHOES_USAGE`, 0 fault. + +**(4) Gate.** `python3 tools/verify.py` (all eight legs, sweep included) is run over the landing tree — queue append and index render in place — before this record is committed; `python3 tools/check_lint.py --check` on this record's CHECK line reads `SOUND`. + +**Not claimed:** that any shipped tool is currently broken this way — all 23 are green on the clean head, and this is a hole in the instrument, not a live defect; that the prose pins should come back — they were wrong twice in one week and the contract grading fixed both; that CLI drift is now caught anywhere else in the repo — nothing in the gate invokes a tool with the flags its own documentation pins, and the `pin` axis still grades counts, not invocations; that `report_read`'s read/malformed split is a new schema — it is unchanged, and `NO_PROBE` rows still land there by probe name; that the pre-repair module is otherwise sound — only its refusal grading was graded here; that the filed record's other figures were re-measured — the `9878f12`/`f400640` tables were read, not re-run, and the one figure this record corrects is the mutant's reachability. Files edited for this item: `tools/selftest_sweep.py`, `tools/selftest_baseline.tsv`, `wizard-work/volt/q97_check.py` (new), `wizard-work/volt/CHECK-q97` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted and no other lane's artifact was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q97.md`. + +CHECK: cd /root/troupe-standup && bash wizard-work/volt/CHECK-q97 diff --git a/docs/queue-index.md b/docs/queue-index.md index 5d1832b..37dd17c 100644 --- a/docs/queue-index.md +++ b/docs/queue-index.md @@ -4,7 +4,7 @@ **What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it. -**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 95 record blocks in all, 59 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times. +**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 96 record blocks in all, 60 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times. | # | id | kind | lines | bytes | role | title | first `Status:` | |---|---|---|---|---|---|---|---| @@ -199,7 +199,8 @@ | 189 | Q106 | record | 13434-13447 | 5422 | cont: DONE | **Q106 — DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest … | -- | | 190 | Q102 | record | 13448-13461 | 4987 | cont: DONE | **Q102 — DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is … | -- | | 191 | Q119 | record | 13462-13479 | 7662 | cont: DONE | **Q119 — DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five … | -- | -| 192 | Q112 | record | 13480-13494 | 7899 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- | +| 192 | Q112 | record | 13480-13495 | 7900 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- | +| 193 | Q97 | record | 13496-13510 | 5649 | cont: DONE | **Q97 — DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now … | -- | ## Ids with no `## ` heading @@ -1477,17 +1478,30 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026" }, { - "bytes": 7899, - "end_line": 13494, + "bytes": 7900, + "end_line": 13495, "id": "Q112", "kind": "record", "line": 13480, - "lines": 15, + "lines": 16, "role": "continuation", "role_detail": "DONE", "status": null, "status_line": null, "title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026" + }, + { + "bytes": 5649, + "end_line": 13510, + "id": "Q97", + "kind": "record", + "line": 13496, + "lines": 15, + "role": "continuation", + "role_detail": "DONE", + "status": null, + "status_line": null, + "title": "**Q97 \u2014 DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now \u2026" } ], "ids_multi_section": { @@ -4109,17 +4123,30 @@ The block below is the parse this document renders, byte for byte. `--check` com "title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026" }, { - "bytes": 7899, - "end_line": 13494, + "bytes": 7900, + "end_line": 13495, "id": "Q112", "kind": "record", "line": 13480, - "lines": 15, + "lines": 16, "role": "continuation", "role_detail": "DONE", "status": null, "status_line": null, "title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026" + }, + { + "bytes": 5649, + "end_line": 13510, + "id": "Q97", + "kind": "record", + "line": 13496, + "lines": 15, + "role": "continuation", + "role_detail": "DONE", + "status": null, + "status_line": null, + "title": "**Q97 \u2014 DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now \u2026" } ], "nested": [ @@ -4453,9 +4480,9 @@ The block below is the parse this document renders, byte for byte. `--check` com "end_line": 8, "line": 1 }, - "queue_bytes": 2121602, - "queue_lines": 13494, - "queue_sha256": "3c472c477f6f040f23fb23582afda7836a42d08401090a8987c02c1d461e520d", + "queue_bytes": 2127252, + "queue_lines": 13510, + "queue_sha256": "385f1aeab3df366641f1798080fd7efbd71384e7bedf02a735474a3032606cae", "sections": [ { "bytes": 3139, diff --git a/tools/selftest_baseline.tsv b/tools/selftest_baseline.tsv index 2835e41..ac32e0d 100644 --- a/tools/selftest_baseline.tsv +++ b/tools/selftest_baseline.tsv @@ -27,7 +27,7 @@ queue_append_audit.py 25 cells queue_claim_check.py 24 fixtures queue_index.py 80 cells reply_check.py 49 cells -selftest_sweep.py 44 cells +selftest_sweep.py 49 cells served_drift_check.py 25 cells tool_map.py 20 cells verify.py 59 cells diff --git a/tools/selftest_sweep.py b/tools/selftest_sweep.py index f603cdd..0e79852 100644 --- a/tools/selftest_sweep.py +++ b/tools/selftest_sweep.py @@ -105,6 +105,33 @@ WHAT A REFUSAL IS GRADED ON, FIXED 2026-10-27 (Q93) table's own comment: the contract keeps "refused, legibly, at the documented code" and stops keeping "refused for the reason the table remembers". +WHAT A REFUSAL MUST NOT BE, ADDED 2026-10-29 (Q97) + The contract above has one blind spot and it is not theoretical: Python's + `argparse` writes `usage: ...` and `prog: error: unrecognized arguments: ...` + to stderr and exits 2, which satisfies "rc 2, no traceback, a message" + exactly. Every probe invokes its tool with flags, so a tool whose flag is + renamed -- or removed -- never reads the input at all, and an axis that + exists to grade malformed-input handling reads clean. Measured at 9878f12: + renaming `--repo` to `--repo-dir` in `citation_hash_class.py` alone left + `malformed-input: 26 probes, 26 ok` and `verdict: rc 0 -- clean`. + A refusal is now ALSO required to be about the input: a message that leads + with `usage:` or carries an argparse failure shape (`unrecognized + arguments`, `error: argument`, `the following arguments are required`, + `invalid choice:`, `ambiguous option:`, `no such option`) is ECHOES_USAGE -- + fatal on both input axes, the read axis included, because that axis has had + the same hole since it was written (it pinned no prose at all). The check is + negative on purpose: requiring the message to name the probe's path would + redden tools that refuse correctly with a boundary line, which is the Q93 + defect coming back the other way. + One measurement the filed record does not carry: its mutant renames `--repo` + to `--repo-dir` with the same dest, and that rename is INVISIBLE to + argparse -- a long option may be abbreviated to any unambiguous prefix, so + `--repo` still parses as `--repo-dir` and the tool reads its input + normally (measured: same refusal line, rc 2, flag and renamed flag alike). + The filed mutant therefore stays green for a reason other than the one the + item names. Cell c48 and the verification below use a rename that is not a + prefix (`--tree`), which is the shape that actually reaches the parser. + ONE VERDICT LINE, ADDED 2026-10-27 (Q93) The leg printed `sweep: 22 tools, 22 OK, 0 fatal` and returned **rc 1**: the sweep line is about the TOOLS, the exit code is about the RUN, and nothing @@ -163,7 +190,7 @@ USAGE: python3 selftest_sweep.py --repo /root/troupe-standup python3 selftest_sweep.py --repo /root/troupe-standup --write-baseline tools/selftest_baseline.tsv python3 selftest_sweep.py --tools-dir /tmp/fake --json /tmp/sweep.json - python3 selftest_sweep.py --selftest # 44 cells, parser, sweep, pin and both input axes + python3 selftest_sweep.py --selftest # 49 cells, parser, sweep, pin and both input axes """ from __future__ import annotations @@ -238,9 +265,23 @@ READ_AXIS_UNIT = REFUSAL_RC # green gate. An ungraded tool may not be reported clean; the tables are the # gate's only cover, so a gap in them has to fail like a fault in a tool. FATAL_READ = ("TRACEBACK", "RC_NOT_2", "SILENT", "NO_TOOL", "RC_WRONG", - "UNEXPLAINED", "NO_PROBE") + "UNEXPLAINED", "NO_PROBE", "ECHOES_USAGE") NO_PROBE = "NO_PROBE" # fatal: a tool with no probe defined +# Q97 (2026-10-29): the shapes argparse prints when it cannot parse the +# INVOCATION. A refusal carrying one of these is the tool telling the operator +# the probe used a flag it does not have -- the input was never read, so the +# probe graded nothing. Checked on both input axes; the usage-line lead is +# checked separately by `echoes_invocation` because argparse always prints one. +ARGV_ECHO_MARKERS = ( + "unrecognized arguments", + "error: argument", + "the following arguments are required", + "invalid choice:", + "ambiguous option:", + "no such option", +) + # {nope} = a path that does not exist; {bin} = a file that is not UTF-8; # {bindir} = a directory holding one non-UTF-8 *.md (added 2026-10-18 with # checkin_check.py, whose input is a directory of UTF-8 files -- a {bin} file @@ -737,6 +778,26 @@ def _first_line(text: str) -> str: return "" +def echoes_invocation(text): + """True when a rc-2 message is argparse failing to PARSE THE INVOCATION + rather than the tool refusing the input (Q97, 2026-10-29). + + Two shapes, both measured from real argparse output rather than invented: + the message's first non-empty line leads with `usage:`, or it carries one + of ARGV_ECHO_MARKERS. The test is negative -- a refusal is red for what it + IS, not for what a table remembers it saying (that was Q93's defect) -- + and it is deliberately not "must name the probe's path", because two tools + in this directory refuse correctly with a boundary line that names no + path.""" + low = (text or "").lower() + if any(m in low for m in ARGV_ECHO_MARKERS): + return True + for line in (text or "").splitlines(): + if line.strip(): + return line.strip().lower().startswith("usage:") + return False + + def judge_read(rc, text): """One probe's verdict. Three classes, in this order on purpose: a traceback is a defect whatever its exit code, and a silent rc 2 is a @@ -748,6 +809,10 @@ def judge_read(rc, text): "detail": _first_line(text) or f"rc {rc}, no output"} if not (text or "").strip(): return {"verdict": "SILENT", "rc": rc, "detail": "rc 2 with no message"} + if echoes_invocation(text): + return {"verdict": "ECHOES_USAGE", "rc": rc, + "detail": "argparse failure, not an input refusal: " + + _first_line(text)} return {"verdict": "OK", "rc": rc, "detail": _first_line(text)} @@ -826,7 +891,9 @@ def judge_malformed(rc, out, err, expect_rc, need): A REFUSAL (expect_rc == 2) is graded on the contract, not on prose (Q93): the documented code, no traceback, one non-empty message on either stream. - Which stream is not part of the contract -- measured 2026-10-27, + Since Q97 (2026-10-29) the message must also NOT be argparse failing to + parse the invocation -- that is ECHOES_USAGE, fatal, because the input was + never read. Which stream is not part of the contract -- measured 2026-10-27, `tool_map.py` writes its refusal to stdout and `citation_hash_class.py` to stderr, both at rc 2, both correct. `need` is only consulted where the tool GRADED the malformed input (rc 0/1): there the word is its verdict.""" @@ -840,6 +907,11 @@ def judge_malformed(rc, out, err, expect_rc, need): if not text.strip(): return {"verdict": "SILENT", "rc": rc, "detail": "rc %s with no message" % rc} if expect_rc == REFUSAL_RC: + if echoes_invocation(text): + # Q97: rc 2 from the argument parser is not a refusal of the input. + return {"verdict": "ECHOES_USAGE", "rc": rc, "contract": "refusal", + "detail": "argparse failure, not an input refusal: " + + _first_line(text)} # a refusal: the code and a readable line are the whole contract return {"verdict": "OK", "rc": rc, "contract": "refusal", "detail": "refusal rc 2: " + _first_line(text)} @@ -953,8 +1025,10 @@ def run_verdict(tool_fatal, read_fatal, malformed_fatal, pin_rc, pin_fatal): def selftest(): bad = [] + ran = [] def cell(name, ok, detail=""): + ran.append(name) print(f"{'PASS' if ok else 'FAIL'} {name}" + (f" [{detail}]" if (detail and not ok) else "")) if not ok: @@ -1228,7 +1302,64 @@ def selftest(): and "pin ungradable" in bad_line, f"{ok_line} | {red_line} | {bad_line}") - print(f"selftest: 44 cells, {44 - len(bad)} passed, {len(bad)} failed") + # c45-c49: Q97 -- a refusal that is argparse failing to parse the + # INVOCATION is not a refusal of the input. c45/c46 are the judge on both + # axes; c47 is the control (a refusal that names its input stays OK); c48 + # is end to end through run_read_probe against a stub whose REAL argparse + # renames the flag the probe passes -- the filed mutant, shrunk to one + # cell; c49 is the summary line naming the fault. + jm = judge_malformed( + 2, "", "usage: t.py [-h] --repo-dir REPO_DIT\nt.py: error: " + "unrecognized arguments: --repo /tmp/x\n", 2, None) + cell("c45 argparse usage at rc 2 on the malformed axis -> ECHOES_USAGE, fatal", + jm["verdict"] == "ECHOES_USAGE" and axis_faults([jm]) == [jm], + jm["verdict"]) + jr = judge_read(2, "usage: t.py [-h] --queue QUEUE\nt.py: error: the " + "following arguments are required: --queue\n") + cell("c46 argparse usage at rc 2 on the read axis -> ECHOES_USAGE, fatal", + jr["verdict"] == "ECHOES_USAGE" and axis_faults([jr]) == [jr], + jr["verdict"]) + jm = judge_malformed(2, "", "cannot read /tmp/nope: [Errno 2] No such file " + "or directory\n", 2, None) + jr = judge_read(2, "cannot read /tmp/nope: [Errno 2] No such file or directory") + cell("c47 a refusal that names its input stays OK on both axes", + jm["verdict"] == "OK" and jr["verdict"] == "OK", + f"{jm['verdict']}/{jr['verdict']}") + + with tempfile.TemporaryDirectory() as td: + stub = os.path.join(td, "renamed_flag.py") + with open(stub, "w", encoding="utf-8") as fh: + fh.write("import argparse\n" + "ap = argparse.ArgumentParser()\n" + # deliberately NOT a prefix of --repo: argparse accepts an + # unambiguous abbreviation, so --repo -> --repo-dir still + # parses and reads the input (measured, Q97). + "ap.add_argument('--tree', dest='repo')\n" + "a = ap.parse_args()\n" + "print('cannot read ' + a.repo)\n" + "raise SystemExit(2)\n") + rc_m, out_m, err_m = run_read_probe(stub, ["--repo", "{nope}"], 30) + jm_bad = judge_malformed(rc_m, out_m, err_m, 2, None) + rc_c, out_c, err_c = run_read_probe(stub, ["--tree", "{nope}"], 30) + jm_ok = judge_malformed(rc_c, out_c, err_c, 2, None) + cell("c48 end to end: a renamed flag reads ECHOES_USAGE, its real flag OK", + rc_m == 2 and jm_bad["verdict"] == "ECHOES_USAGE" + and rc_c == 2 and jm_ok["verdict"] == "OK", + f"{rc_m}:{jm_bad['verdict']} / {rc_c}:{jm_ok['verdict']}") + + axis_echo = [{"tool": "t.py", "probe": "not-a-repo", "verdict": "ECHOES_USAGE", + "rc": 2, "expect_rc": 2, "detail": "argparse failure"}] + buf = io.StringIO() + with contextlib.redirect_stdout(buf): + erc, ehard, esoft = report_read(axis_echo, quiet=True) + line_e = buf.getvalue() + cell("c49 the summary names ECHOES_USAGE with its probe, and the run is red", + erc == 1 and (ehard, esoft) == (0, 1) + and "malformed-input: 1 probes, 0 ok, 1 fault " + "(ECHOES_USAGE t.py/not-a-repo)" in line_e, + line_e.strip().splitlines()[-1] if line_e.strip() else "no output") + + print(f"selftest: {len(ran)} cells, {len(ran) - len(bad)} passed, {len(bad)} failed") return 1 if bad else 0 diff --git a/wizard-work/volt/CHECK-q97 b/wizard-work/volt/CHECK-q97 new file mode 100644 index 0000000..6983527 --- /dev/null +++ b/wizard-work/volt/CHECK-q97 @@ -0,0 +1,13 @@ +#!/usr/bin/env bash +# CHECK-q97 -- the Q97 gate, rooted at the tree this file sits in. +# +# Grades tools/selftest_sweep.py's Q97 rule: a refusal at rc 2 that is really +# argparse failing to parse the INVOCATION (usage: lead, unrecognized +# arguments, ...) is ECHOES_USAGE -- fatal on both input axes -- so a renamed +# flag can no longer turn the malformed/read axes green. Exit codes follow the +# repo contract: 0 clean, 1 graded fault, 2 could not grade. +# +# bash wizard-work/volt/CHECK-q97 +set -u +cd "$(dirname "${BASH_SOURCE[0]}")/../.." || exit 2 +exec python3 wizard-work/volt/q97_check.py diff --git a/wizard-work/volt/q97_check.py b/wizard-work/volt/q97_check.py new file mode 100644 index 0000000..b5b2410 --- /dev/null +++ b/wizard-work/volt/q97_check.py @@ -0,0 +1,177 @@ +#!/usr/bin/env python3 +"""q97_check.py -- the Q97 gate: a refusal must be about the INPUT. + +The defect this grades was filed against `tools/selftest_sweep.py`: a probe +that is refused at rc 2 with `usage: ... / error: unrecognized arguments: ...` +is an argparse failure to parse the INVOCATION, not a refusal of the input. +The tool never read anything, and the malformed-input axis -- the one place CLI +drift could not hide -- read clean. + +Four legs, all measured on this tree: + + 1. the sweep's own selftest is green and its printed cell count equals the + pinned row in `tools/selftest_baseline.tsv` (a new cell without a re-pin + is a PIN_DRIFT, so the gate cannot pass on a cell nobody counted); + 2. the live axes on this checkout are clean -- 0 faults on read-refusal and + malformed-input -- so the new rule does not redden a tool that refuses + correctly (28 + 27 probes at the landing head); + 3. MUTANT: `tools/citation_hash_class.py` copied to a scratch file with its + `--repo` flag renamed to `--tree` (a rename that is NOT a prefix of the + old one: argparse abbreviates long options, so `--repo` -> `--repo-dir` + still parses and this check refuses to pretend otherwise). Both probes + that invoke it with `--repo` must read ECHOES_USAGE, and the axis must be + red, rc 1; + 4. CONTROL: the same copy unmutated reads no ECHOES_USAGE at all and every + probe is OK -- so leg 3 is the mutation, not a file that was already red. + +Exit codes follow the repo contract: 0 clean, 1 graded fault, 2 could not +grade (no sweep module, no tool copy, no python3). + +Run: python3 wizard-work/volt/q97_check.py +""" +import importlib.util +import io +import os +import shutil +import subprocess +import sys +import tempfile +from contextlib import redirect_stdout + +HERE = os.path.dirname(os.path.realpath(__file__)) +ROOT = os.path.dirname(os.path.dirname(HERE)) +TOOLS = os.path.join(ROOT, "tools") +SWEEP = os.path.join(TOOLS, "selftest_sweep.py") +BASELINE = os.path.join(TOOLS, "selftest_baseline.tsv") +SUBJECT = os.path.join(TOOLS, "citation_hash_class.py") + +failures = [] + + +def note(ok, line): + print(("PASS " if ok else "FAIL ") + line) + if not ok: + failures.append(line) + + +def leg_selftest(): + """The sweep's own cells, and the pin that counts them.""" + proc = subprocess.run([sys.executable, SWEEP, "--selftest"], + capture_output=True, text=True, timeout=300) + out = (proc.stdout or "").splitlines() + cells = [l for l in out if l.startswith("selftest:")] + printed = cells[-1] if cells else "no count line" + note(proc.returncode == 0 and bool(cells), + "leg 1a sweep --selftest rc %d: %s" + % (proc.returncode, printed)) + pinned = None + with open(BASELINE, encoding="utf-8") as fh: + for line in fh: + parts = line.split("\t") + if len(parts) == 3 and parts[0] == "selftest_sweep.py": + pinned = int(parts[1]) + if pinned is None: + note(False, "leg 1b no pinned row for selftest_sweep.py -- cannot grade") + return + seen = None + if cells: + try: + seen = int(cells[-1].split()[1].replace(",", "")) + except (IndexError, ValueError): + seen = None + note(seen == pinned, + "leg 1b printed %s cells, pinned %s" % (seen, pinned)) + + +_SWEEP = [] + + +def sweep_module(): + """`tools/selftest_sweep.py`, loaded by path so this gate grades the tree + it sits in and not a copy on sys.path.""" + if not _SWEEP: + spec = importlib.util.spec_from_file_location("q97_selftest_sweep", SWEEP) + if spec is None or spec.loader is None: + raise RuntimeError("cannot load %s" % SWEEP) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + _SWEEP.append(mod) + return _SWEEP[0] + + +def axis_for(path): + """The rows for one `citation_hash_class.py` copy, through the real probe + path. Rows for other tools are dropped: with a one-row table every other + probe is NO_TOOL, which is noise about the table, not about the copy.""" + sweep = sweep_module() + rows = sweep.read_axis([("citation_hash_class.py", path)], 60) + return sweep, [r for r in rows if r["tool"] == "citation_hash_class.py"] + + +def leg_mutant_and_control(): + sweep, control = axis_for(SUBJECT) + echo_c = [r for r in control if r["verdict"] == "ECHOES_USAGE"] + bad_c = [r for r in control if r["verdict"] in sweep.FATAL_READ] + note(not echo_c and not bad_c, + "leg 4 control: clean copy reads %d probe(s), 0 ECHOES_USAGE, " + "0 fault" % len(control)) + + td = tempfile.mkdtemp(prefix="q97-") + try: + mutant = os.path.join(td, "citation_hash_class.py") + shutil.copyfile(SUBJECT, mutant) + with open(mutant, encoding="utf-8") as fh: + src = fh.read() + old = "ap.add_argument('--repo', default=DEFAULT_REPO)" + new = "ap.add_argument('--tree', dest='repo', default=DEFAULT_REPO)" + if src.count(old) != 1: + note(False, "leg 3 cannot build the mutant: %r appears %d time(s)" + % (old, src.count(old))) + return + with open(mutant, "w", encoding="utf-8") as fh: + fh.write(src.replace(old, new, 1)) + sweep2, rows = axis_for(mutant) + echo_m = [r for r in rows if r["verdict"] == "ECHOES_USAGE"] + named = sorted((r["probe"] for r in echo_m)) + note(named == ["missing", "not-a-repo"], + "leg 3 mutant (--repo -> --tree): ECHOES_USAGE on %s" % (named,)) + buf = io.StringIO() + with redirect_stdout(buf): + rc, hard, soft = sweep2.report_read(rows, quiet=True) + line = buf.getvalue().strip().splitlines() + summary = line[-1] if line else "no summary" + note(rc == 1 and (hard, soft) == (1, 1) + and "ECHOES_USAGE citation_hash_class.py/not-a-repo" in buf.getvalue(), + "leg 3b mutant axis %s" % summary) + finally: + shutil.rmtree(td, ignore_errors=True) + + +def leg_live_axes(): + sweep = sweep_module() + live, _source, _untracked = sweep.discover(ROOT, None) + axis = sweep.read_axis(live, 60) + bad = [r for r in axis if r["verdict"] in sweep.FATAL_READ] + echo = [r for r in axis if r["verdict"] == "ECHOES_USAGE"] + note(not bad and not echo, + "leg 2 live axes: %d probe(s), 0 fault, 0 ECHOES_USAGE" % len(axis)) + + +def main(): + if not (os.path.isfile(SWEEP) and os.path.isfile(SUBJECT)): + print("cannot grade: tools/selftest_sweep.py or citation_hash_class.py " + "missing under %s" % ROOT, file=sys.stderr) + return 2 + leg_selftest() + leg_live_axes() + leg_mutant_and_control() + if failures: + print("Q97 GATE: rc 1 -- %d leg(s) failed" % len(failures)) + return 1 + print("Q97 GATE: rc 0 -- a refusal that echoes the invocation is fatal on " + "both input axes, and a mutating flag rename is caught.") + return 0 + + +if __name__ == "__main__": + sys.exit(main())