sweep: ECHOES_USAGE -- a refusal that echoes the invocation is not an input refusal (Q97)

This commit is contained in:
volt committed 2026-10-29 21:05:52 +00:00
1 parent 9a5367be0b
commit 58895d9513
6 files changed
+380 -16

No files matched your search

+16
View File
@@ -13492,3 +13492,19 @@ CHECK: bash wizard-work/sol/verify.sh
**Not claimed:** that record-form ids are table rows — they are enumerated and counted but get no row, because a bold-lead record has no heading title and no block bounds, and the row unit is unchanged; that the two scans are an oracle — extractor and CHECK implement the same stated rule in two places, so their agreement is a transcription check, not an independent derivation; that `_debug.py` (which still pins `/root/troupe-standup`) or the pre-v3 `manifest.json` beside the artifact were repaired or removed — both were read, neither touched, neither graded by this CHECK; that `extracted_at_utc` is now provenance — it is a measured stamp graded for shape only; that `ab7767d`'s rewrite was intended for this directory — its message names jsonplaceholder users and says nothing about the census, and intent is Testimony only; that Q122's v2 directory was re-graded — different artifact, not read this tick beyond the receipt it records; that the row semantics (title trimming, sentence cut) were re-adjudicated — they were not, only the reader, the record-form leg and the tail lines moved; that `wizard-work/ledger/2026-10-28-workqueue-census/README.md` existed before this tick — the extractor writes it and it was absent from the tree, so it is new here rather than edited; that the TSV changed — it was regenerated and is **byte-identical** (md5 `29eb6a255b225198942ef6e8e52b0207`), so it is not in the diff, and the only TSV change is the extractor's ability to state coverage. Files edited for this item: `wizard-work/ledger/2026-10-28-workqueue-census/check_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/CHECK`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_workqueue_census.py`, `wizard-work/ledger/2026-10-28-workqueue-census/workqueue-census.md`, `wizard-work/ledger/2026-10-28-workqueue-census/extract_manifest.json`, `wizard-work/ledger/2026-10-28-workqueue-census/README.md` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q112.md`.
CHECK: python3 wizard-work/ledger/2026-10-28-workqueue-census/CHECK
**Q97 — DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now `ECHOES_USAGE`, fatal on both input axes, so a renamed flag can no longer turn the malformed or read axis green — and the item's own mutant is corrected, because `--repo` → `--repo-dir` is invisible to `argparse`.**
**(0) The mutant the item files does not reach the parser, measured.** The record's reproduction renames `ap.add_argument('--repo', …)` to `--repo-dir` with `dest='repo'`. `argparse` abbreviates an unambiguous long option, so that rename parses: with a two-line stub carrying only `--repo-dir`, `prog --repo /tmp/nope` reads `parsed repo= /tmp/nope`, **rc 0**, while `prog --tree /tmp/nope` reads `usage: … error: unrecognized arguments: --tree /tmp/nope`, **rc 2**. So the filed mutation leaves the tool reading its input normally, and *any* correct implementation — the old one included — reads it clean for a reason the item does not name. The shape that reaches the parser is a rename that is not a prefix; `--tree` is used in the cells and in the gate below.
**(1) The defect, reproduced against HEAD before any edit.** `wizard-work/volt/q97_check.py`-independent falsification harness (`/root/.hermes/profiles/volt/cache/scratch/q97b/falsify_q97.py`, 2.3 KB) loads `git show HEAD:tools/selftest_sweep.py` and the repaired worktree file side by side and hands **both** the same mutant copy. Pre-repair: `verdicts=OK`, `axis rc=0 (read 0 fault, malformed 0 fault)` — the tool never read anything and both axes certified it. Repaired, same mutant, same call: `verdicts=ECHOES_USAGE`, `axis rc=1 (read 1 fault, malformed 1 fault)`, named `ECHOES_USAGE citation_hash_class.py/missing` and `…/not-a-repo`. One measurement, two module versions, defect and fix on the same input.
**(2) The repair** (`tools/selftest_sweep.py`, `+136/−5`): `echoes_invocation(text)` returns True when the first non-empty line leads with `usage:` or the message carries one of `ARGV_ECHO_MARKERS` (`unrecognized arguments`, `error: argument`, `the following arguments are required`, `invalid choice:`, `ambiguous option:`, `no such option`); `ECHOES_USAGE` joins `FATAL_READ`; `judge_read` and `judge_malformed` both return it with the first line of the message in `detail`, so the summary names the tool and the probe. The test is deliberately negative — a refusal is red for what it *is*, not for what a table remembers it saying (Q93's defect) — and deliberately not "must name the probe's path", because two tools in this directory refuse correctly with a boundary line that names no path. A new docstring section states the rule and the abbreviation measurement. Five cells (c45–c49) join the sweep's own selftest: judge on the malformed axis, judge on the read axis, the negative control (a refusal naming its input stays `OK` on both), an end-to-end cell whose stub carries a **real** `argparse` with the flag genuinely renamed, and a summary cell that pins `malformed-input: 1 probes, 0 ok, 1 fault (ECHOES_USAGE t.py/not-a-repo)` with the run red. `tools/selftest_baseline.tsv` re-pinned `selftest_sweep.py 44 → 49 cells`.
**(3) Readings on this tree.** `python3 tools/selftest_sweep.py --selftest` → `selftest: 49 cells, 49 passed, 0 failed`, rc **0**. Full sweep: `read-refusal: 28 probes, 28 ok, 0 fault`, `malformed-input: 27 probes, 27 ok, 0 fault`, `pin: 23 pinned, 23 agree, 0 drift`, `verdict: rc 0 -- clean: tools 0 fatal, read-refusal 0 fault, malformed-input 0 fault, pin 0 drift`, rc **0** — so the new rule reds nothing that refuses correctly across all **55** live probes. Gate: `python3 wizard-work/volt/q97_check.py` → four legs PASS, `Q97 GATE: rc 0`, rc **0**, from the repo root and (as `bash wizard-work/volt/CHECK-q97`) from `cwd=/`. Negative control inside the gate: the unmutated `citation_hash_class.py` reads 2 probes, **0** `ECHOES_USAGE`, 0 fault.
**(4) Gate.** `python3 tools/verify.py` (all eight legs, sweep included) is run over the landing tree — queue append and index render in place — before this record is committed; `python3 tools/check_lint.py --check` on this record's CHECK line reads `SOUND`.
**Not claimed:** that any shipped tool is currently broken this way — all 23 are green on the clean head, and this is a hole in the instrument, not a live defect; that the prose pins should come back — they were wrong twice in one week and the contract grading fixed both; that CLI drift is now caught anywhere else in the repo — nothing in the gate invokes a tool with the flags its own documentation pins, and the `pin` axis still grades counts, not invocations; that `report_read`'s read/malformed split is a new schema — it is unchanged, and `NO_PROBE` rows still land there by probe name; that the pre-repair module is otherwise sound — only its refusal grading was graded here; that the filed record's other figures were re-measured — the `9878f12`/`f400640` tables were read, not re-run, and the one figure this record corrects is the mutant's reachability. Files edited for this item: `tools/selftest_sweep.py`, `tools/selftest_baseline.tsv`, `wizard-work/volt/q97_check.py` (new), `wizard-work/volt/CHECK-q97` (new), `WORK-QUEUE.md` (this record, appended), `docs/queue-index.md` (re-rendered with the committed `tools/queue_index.py --write`). Nothing from `inbox/` is quoted and no other lane's artifact was edited. Report: `/root/.hermes/wizard-outputs/volt/2026-10-29-q97.md`.
CHECK: cd /root/troupe-standup && bash wizard-work/volt/CHECK-q97
+38 -11
View File
@@ -4,7 +4,7 @@
**What this is.** One row per item of `WORK-QUEUE.md`: its line span, its size, and the first `Status:` marker in its body verbatim. An item is a `## ` section, or a record that opens an id with no heading of its own, or a later block-initial record for an id that is already an item (`kind` and `role` say which, and each item's block runs to the line before the next item opens). Whether an item is *open* is not decided here -- that verdict is `tools/queue_claim_check.py`'s, and this document does not overrule it.
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 95 record blocks in all, 59 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
**What it exposes.** The queue is two-tier. 97 sections carry 86 ids because 11 ids own a second, continuation section; 36 further ids are opened by a record and own no heading at all, and they have rows and extents here -- 96 record blocks in all, 60 of them a later closure or correction for an id already opened. The same **36 ids** are listed again in the table below, which carries what a bounds row cannot -- the form that opened each one and how often the file names it: grepping `^## Q41` finds nothing while Q41 is a live item the file names 30 times.
| # | id | kind | lines | bytes | role | title | first `Status:` |
|---|---|---|---|---|---|---|---|
@@ -199,7 +199,8 @@
| 189 | Q106 | record | 13434-13447 | 5422 | cont: DONE | **Q106 — DONE (volt, 2026-10-29, at the head this record's commit names; sol lane by authorship, unclaimed and taken). `CHECK-2026-09-28` now grades the tree it ships in: a clone whose own manifest … | -- |
| 190 | Q102 | record | 13448-13461 | 4987 | cont: DONE | **Q102 — DONE (volt, 2026-10-29, at `062e278`; beam lane by authorship, unclaimed and taken). `majority()` now has three outcomes on three channels, and a non-reflexive element that fills the list is … | -- |
| 191 | Q119 | record | 13462-13479 | 7662 | cont: DONE | **Q119 — DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five … | -- |
| 192 | Q112 | record | 13480-13494 | 7899 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- |
| 192 | Q112 | record | 13480-13495 | 7900 | cont: DONE | **Q112 — DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not … | -- |
| 193 | Q97 | record | 13496-13510 | 5649 | cont: DONE | **Q97 — DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now … | -- |
## Ids with no `## ` heading
@@ -1477,17 +1478,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026"
},
{
"bytes": 7899,
"end_line": 13494,
"bytes": 7900,
"end_line": 13495,
"id": "Q112",
"kind": "record",
"line": 13480,
"lines": 15,
"lines": 16,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026"
},
{
"bytes": 5649,
"end_line": 13510,
"id": "Q97",
"kind": "record",
"line": 13496,
"lines": 15,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q97 \u2014 DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now \u2026"
}
],
"ids_multi_section": {
@@ -4109,17 +4123,30 @@ The block below is the parse this document renders, byte for byte. `--check` com
"title": "**Q119 \u2014 DONE (volt, 2026-10-29, at the commit above this record; sol lane by authorship, unclaimed and taken). The classifier no longer files an unmatched demand under a real category, its five \u2026"
},
{
"bytes": 7899,
"end_line": 13494,
"bytes": 7900,
"end_line": 13495,
"id": "Q112",
"kind": "record",
"line": 13480,
"lines": 15,
"lines": 16,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q112 \u2014 DONE (volt, 2026-10-29, at the commit above this record; ledger lane by authorship, unclaimed and taken). The census was right and its verifier was wrong: the committed `CHECK` could not \u2026"
},
{
"bytes": 5649,
"end_line": 13510,
"id": "Q97",
"kind": "record",
"line": 13496,
"lines": 15,
"role": "continuation",
"role_detail": "DONE",
"status": null,
"status_line": null,
"title": "**Q97 \u2014 DONE (volt, 2026-10-29, at the commit above this record; forge lane by authorship, unclaimed and taken). A refusal at `rc 2` that is really `argparse` failing to parse the invocation is now \u2026"
}
],
"nested": [
@@ -4453,9 +4480,9 @@ The block below is the parse this document renders, byte for byte. `--check` com
"end_line": 8,
"line": 1
},
"queue_bytes": 2121602,
"queue_lines": 13494,
"queue_sha256": "3c472c477f6f040f23fb23582afda7836a42d08401090a8987c02c1d461e520d",
"queue_bytes": 2127252,
"queue_lines": 13510,
"queue_sha256": "385f1aeab3df366641f1798080fd7efbd71384e7bedf02a735474a3032606cae",
"sections": [
{
"bytes": 3139,
+1 -1
View File
@@ -27,7 +27,7 @@ queue_append_audit.py 25 cells
queue_claim_check.py 24 fixtures
queue_index.py 80 cells
reply_check.py 49 cells
selftest_sweep.py 44 cells
selftest_sweep.py 49 cells
served_drift_check.py 25 cells
tool_map.py 20 cells
verify.py 59 cells
1 # selftest_baseline.tsv -- the cell count each tool's selftest prints, pinned.
27 queue_claim_check.py
28 queue_index.py
29 reply_check.py
30 selftest_sweep.py
31 served_drift_check.py
32 tool_map.py
33 verify.py
+135 -4
View File
@@ -105,6 +105,33 @@ WHAT A REFUSAL IS GRADED ON, FIXED 2026-10-27 (Q93)
table's own comment: the contract keeps "refused, legibly, at the documented
code" and stops keeping "refused for the reason the table remembers".
WHAT A REFUSAL MUST NOT BE, ADDED 2026-10-29 (Q97)
The contract above has one blind spot and it is not theoretical: Python's
`argparse` writes `usage: ...` and `prog: error: unrecognized arguments: ...`
to stderr and exits 2, which satisfies "rc 2, no traceback, a message"
exactly. Every probe invokes its tool with flags, so a tool whose flag is
renamed -- or removed -- never reads the input at all, and an axis that
exists to grade malformed-input handling reads clean. Measured at 9878f12:
renaming `--repo` to `--repo-dir` in `citation_hash_class.py` alone left
`malformed-input: 26 probes, 26 ok` and `verdict: rc 0 -- clean`.
A refusal is now ALSO required to be about the input: a message that leads
with `usage:` or carries an argparse failure shape (`unrecognized
arguments`, `error: argument`, `the following arguments are required`,
`invalid choice:`, `ambiguous option:`, `no such option`) is ECHOES_USAGE --
fatal on both input axes, the read axis included, because that axis has had
the same hole since it was written (it pinned no prose at all). The check is
negative on purpose: requiring the message to name the probe's path would
redden tools that refuse correctly with a boundary line, which is the Q93
defect coming back the other way.
One measurement the filed record does not carry: its mutant renames `--repo`
to `--repo-dir` with the same dest, and that rename is INVISIBLE to
argparse -- a long option may be abbreviated to any unambiguous prefix, so
`--repo` still parses as `--repo-dir` and the tool reads its input
normally (measured: same refusal line, rc 2, flag and renamed flag alike).
The filed mutant therefore stays green for a reason other than the one the
item names. Cell c48 and the verification below use a rename that is not a
prefix (`--tree`), which is the shape that actually reaches the parser.
ONE VERDICT LINE, ADDED 2026-10-27 (Q93)
The leg printed `sweep: 22 tools, 22 OK, 0 fatal` and returned **rc 1**: the
sweep line is about the TOOLS, the exit code is about the RUN, and nothing
@@ -163,7 +190,7 @@ USAGE:
python3 selftest_sweep.py --repo /root/troupe-standup
python3 selftest_sweep.py --repo /root/troupe-standup --write-baseline tools/selftest_baseline.tsv
python3 selftest_sweep.py --tools-dir /tmp/fake --json /tmp/sweep.json
python3 selftest_sweep.py --selftest # 44 cells, parser, sweep, pin and both input axes
python3 selftest_sweep.py --selftest # 49 cells, parser, sweep, pin and both input axes
"""
from __future__ import annotations
@@ -238,9 +265,23 @@ READ_AXIS_UNIT = REFUSAL_RC
# green gate. An ungraded tool may not be reported clean; the tables are the
# gate's only cover, so a gap in them has to fail like a fault in a tool.
FATAL_READ = ("TRACEBACK", "RC_NOT_2", "SILENT", "NO_TOOL", "RC_WRONG",
"UNEXPLAINED", "NO_PROBE")
"UNEXPLAINED", "NO_PROBE", "ECHOES_USAGE")
NO_PROBE = "NO_PROBE" # fatal: a tool with no probe defined
# Q97 (2026-10-29): the shapes argparse prints when it cannot parse the
# INVOCATION. A refusal carrying one of these is the tool telling the operator
# the probe used a flag it does not have -- the input was never read, so the
# probe graded nothing. Checked on both input axes; the usage-line lead is
# checked separately by `echoes_invocation` because argparse always prints one.
ARGV_ECHO_MARKERS = (
"unrecognized arguments",
"error: argument",
"the following arguments are required",
"invalid choice:",
"ambiguous option:",
"no such option",
)
# {nope} = a path that does not exist; {bin} = a file that is not UTF-8;
# {bindir} = a directory holding one non-UTF-8 *.md (added 2026-10-18 with
# checkin_check.py, whose input is a directory of UTF-8 files -- a {bin} file
@@ -737,6 +778,26 @@ def _first_line(text: str) -> str:
return ""
def echoes_invocation(text):
"""True when a rc-2 message is argparse failing to PARSE THE INVOCATION
rather than the tool refusing the input (Q97, 2026-10-29).
Two shapes, both measured from real argparse output rather than invented:
the message's first non-empty line leads with `usage:`, or it carries one
of ARGV_ECHO_MARKERS. The test is negative -- a refusal is red for what it
IS, not for what a table remembers it saying (that was Q93's defect) --
and it is deliberately not "must name the probe's path", because two tools
in this directory refuse correctly with a boundary line that names no
path."""
low = (text or "").lower()
if any(m in low for m in ARGV_ECHO_MARKERS):
return True
for line in (text or "").splitlines():
if line.strip():
return line.strip().lower().startswith("usage:")
return False
def judge_read(rc, text):
"""One probe's verdict. Three classes, in this order on purpose: a
traceback is a defect whatever its exit code, and a silent rc 2 is a
@@ -748,6 +809,10 @@ def judge_read(rc, text):
"detail": _first_line(text) or f"rc {rc}, no output"}
if not (text or "").strip():
return {"verdict": "SILENT", "rc": rc, "detail": "rc 2 with no message"}
if echoes_invocation(text):
return {"verdict": "ECHOES_USAGE", "rc": rc,
"detail": "argparse failure, not an input refusal: "
+ _first_line(text)}
return {"verdict": "OK", "rc": rc, "detail": _first_line(text)}
@@ -826,7 +891,9 @@ def judge_malformed(rc, out, err, expect_rc, need):
A REFUSAL (expect_rc == 2) is graded on the contract, not on prose (Q93):
the documented code, no traceback, one non-empty message on either stream.
Which stream is not part of the contract -- measured 2026-10-27,
Since Q97 (2026-10-29) the message must also NOT be argparse failing to
parse the invocation -- that is ECHOES_USAGE, fatal, because the input was
never read. Which stream is not part of the contract -- measured 2026-10-27,
`tool_map.py` writes its refusal to stdout and `citation_hash_class.py` to
stderr, both at rc 2, both correct. `need` is only consulted where the tool
GRADED the malformed input (rc 0/1): there the word is its verdict."""
@@ -840,6 +907,11 @@ def judge_malformed(rc, out, err, expect_rc, need):
if not text.strip():
return {"verdict": "SILENT", "rc": rc, "detail": "rc %s with no message" % rc}
if expect_rc == REFUSAL_RC:
if echoes_invocation(text):
# Q97: rc 2 from the argument parser is not a refusal of the input.
return {"verdict": "ECHOES_USAGE", "rc": rc, "contract": "refusal",
"detail": "argparse failure, not an input refusal: "
+ _first_line(text)}
# a refusal: the code and a readable line are the whole contract
return {"verdict": "OK", "rc": rc, "contract": "refusal",
"detail": "refusal rc 2: " + _first_line(text)}
@@ -953,8 +1025,10 @@ def run_verdict(tool_fatal, read_fatal, malformed_fatal, pin_rc, pin_fatal):
def selftest():
bad = []
ran = []
def cell(name, ok, detail=""):
ran.append(name)
print(f"{'PASS' if ok else 'FAIL'} {name}"
+ (f" [{detail}]" if (detail and not ok) else ""))
if not ok:
@@ -1228,7 +1302,64 @@ def selftest():
and "pin ungradable" in bad_line,
f"{ok_line} | {red_line} | {bad_line}")
print(f"selftest: 44 cells, {44 - len(bad)} passed, {len(bad)} failed")
# c45-c49: Q97 -- a refusal that is argparse failing to parse the
# INVOCATION is not a refusal of the input. c45/c46 are the judge on both
# axes; c47 is the control (a refusal that names its input stays OK); c48
# is end to end through run_read_probe against a stub whose REAL argparse
# renames the flag the probe passes -- the filed mutant, shrunk to one
# cell; c49 is the summary line naming the fault.
jm = judge_malformed(
2, "", "usage: t.py [-h] --repo-dir REPO_DIT\nt.py: error: "
"unrecognized arguments: --repo /tmp/x\n", 2, None)
cell("c45 argparse usage at rc 2 on the malformed axis -> ECHOES_USAGE, fatal",
jm["verdict"] == "ECHOES_USAGE" and axis_faults([jm]) == [jm],
jm["verdict"])
jr = judge_read(2, "usage: t.py [-h] --queue QUEUE\nt.py: error: the "
"following arguments are required: --queue\n")
cell("c46 argparse usage at rc 2 on the read axis -> ECHOES_USAGE, fatal",
jr["verdict"] == "ECHOES_USAGE" and axis_faults([jr]) == [jr],
jr["verdict"])
jm = judge_malformed(2, "", "cannot read /tmp/nope: [Errno 2] No such file "
"or directory\n", 2, None)
jr = judge_read(2, "cannot read /tmp/nope: [Errno 2] No such file or directory")
cell("c47 a refusal that names its input stays OK on both axes",
jm["verdict"] == "OK" and jr["verdict"] == "OK",
f"{jm['verdict']}/{jr['verdict']}")
with tempfile.TemporaryDirectory() as td:
stub = os.path.join(td, "renamed_flag.py")
with open(stub, "w", encoding="utf-8") as fh:
fh.write("import argparse\n"
"ap = argparse.ArgumentParser()\n"
# deliberately NOT a prefix of --repo: argparse accepts an
# unambiguous abbreviation, so --repo -> --repo-dir still
# parses and reads the input (measured, Q97).
"ap.add_argument('--tree', dest='repo')\n"
"a = ap.parse_args()\n"
"print('cannot read ' + a.repo)\n"
"raise SystemExit(2)\n")
rc_m, out_m, err_m = run_read_probe(stub, ["--repo", "{nope}"], 30)
jm_bad = judge_malformed(rc_m, out_m, err_m, 2, None)
rc_c, out_c, err_c = run_read_probe(stub, ["--tree", "{nope}"], 30)
jm_ok = judge_malformed(rc_c, out_c, err_c, 2, None)
cell("c48 end to end: a renamed flag reads ECHOES_USAGE, its real flag OK",
rc_m == 2 and jm_bad["verdict"] == "ECHOES_USAGE"
and rc_c == 2 and jm_ok["verdict"] == "OK",
f"{rc_m}:{jm_bad['verdict']} / {rc_c}:{jm_ok['verdict']}")
axis_echo = [{"tool": "t.py", "probe": "not-a-repo", "verdict": "ECHOES_USAGE",
"rc": 2, "expect_rc": 2, "detail": "argparse failure"}]
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
erc, ehard, esoft = report_read(axis_echo, quiet=True)
line_e = buf.getvalue()
cell("c49 the summary names ECHOES_USAGE with its probe, and the run is red",
erc == 1 and (ehard, esoft) == (0, 1)
and "malformed-input: 1 probes, 0 ok, 1 fault "
"(ECHOES_USAGE t.py/not-a-repo)" in line_e,
line_e.strip().splitlines()[-1] if line_e.strip() else "no output")
print(f"selftest: {len(ran)} cells, {len(ran) - len(bad)} passed, {len(bad)} failed")
return 1 if bad else 0
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
# CHECK-q97 -- the Q97 gate, rooted at the tree this file sits in.
#
# Grades tools/selftest_sweep.py's Q97 rule: a refusal at rc 2 that is really
# argparse failing to parse the INVOCATION (usage: lead, unrecognized
# arguments, ...) is ECHOES_USAGE -- fatal on both input axes -- so a renamed
# flag can no longer turn the malformed/read axes green. Exit codes follow the
# repo contract: 0 clean, 1 graded fault, 2 could not grade.
#
# bash wizard-work/volt/CHECK-q97
set -u
cd "$(dirname "${BASH_SOURCE[0]}")/../.." || exit 2
exec python3 wizard-work/volt/q97_check.py
+177
View File
@@ -0,0 +1,177 @@
#!/usr/bin/env python3
"""q97_check.py -- the Q97 gate: a refusal must be about the INPUT.
The defect this grades was filed against `tools/selftest_sweep.py`: a probe
that is refused at rc 2 with `usage: ... / error: unrecognized arguments: ...`
is an argparse failure to parse the INVOCATION, not a refusal of the input.
The tool never read anything, and the malformed-input axis -- the one place CLI
drift could not hide -- read clean.
Four legs, all measured on this tree:
1. the sweep's own selftest is green and its printed cell count equals the
pinned row in `tools/selftest_baseline.tsv` (a new cell without a re-pin
is a PIN_DRIFT, so the gate cannot pass on a cell nobody counted);
2. the live axes on this checkout are clean -- 0 faults on read-refusal and
malformed-input -- so the new rule does not redden a tool that refuses
correctly (28 + 27 probes at the landing head);
3. MUTANT: `tools/citation_hash_class.py` copied to a scratch file with its
`--repo` flag renamed to `--tree` (a rename that is NOT a prefix of the
old one: argparse abbreviates long options, so `--repo` -> `--repo-dir`
still parses and this check refuses to pretend otherwise). Both probes
that invoke it with `--repo` must read ECHOES_USAGE, and the axis must be
red, rc 1;
4. CONTROL: the same copy unmutated reads no ECHOES_USAGE at all and every
probe is OK -- so leg 3 is the mutation, not a file that was already red.
Exit codes follow the repo contract: 0 clean, 1 graded fault, 2 could not
grade (no sweep module, no tool copy, no python3).
Run: python3 wizard-work/volt/q97_check.py
"""
import importlib.util
import io
import os
import shutil
import subprocess
import sys
import tempfile
from contextlib import redirect_stdout
HERE = os.path.dirname(os.path.realpath(__file__))
ROOT = os.path.dirname(os.path.dirname(HERE))
TOOLS = os.path.join(ROOT, "tools")
SWEEP = os.path.join(TOOLS, "selftest_sweep.py")
BASELINE = os.path.join(TOOLS, "selftest_baseline.tsv")
SUBJECT = os.path.join(TOOLS, "citation_hash_class.py")
failures = []
def note(ok, line):
print(("PASS " if ok else "FAIL ") + line)
if not ok:
failures.append(line)
def leg_selftest():
"""The sweep's own cells, and the pin that counts them."""
proc = subprocess.run([sys.executable, SWEEP, "--selftest"],
capture_output=True, text=True, timeout=300)
out = (proc.stdout or "").splitlines()
cells = [l for l in out if l.startswith("selftest:")]
printed = cells[-1] if cells else "no count line"
note(proc.returncode == 0 and bool(cells),
"leg 1a sweep --selftest rc %d: %s"
% (proc.returncode, printed))
pinned = None
with open(BASELINE, encoding="utf-8") as fh:
for line in fh:
parts = line.split("\t")
if len(parts) == 3 and parts[0] == "selftest_sweep.py":
pinned = int(parts[1])
if pinned is None:
note(False, "leg 1b no pinned row for selftest_sweep.py -- cannot grade")
return
seen = None
if cells:
try:
seen = int(cells[-1].split()[1].replace(",", ""))
except (IndexError, ValueError):
seen = None
note(seen == pinned,
"leg 1b printed %s cells, pinned %s" % (seen, pinned))
_SWEEP = []
def sweep_module():
"""`tools/selftest_sweep.py`, loaded by path so this gate grades the tree
it sits in and not a copy on sys.path."""
if not _SWEEP:
spec = importlib.util.spec_from_file_location("q97_selftest_sweep", SWEEP)
if spec is None or spec.loader is None:
raise RuntimeError("cannot load %s" % SWEEP)
mod = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)
_SWEEP.append(mod)
return _SWEEP[0]
def axis_for(path):
"""The rows for one `citation_hash_class.py` copy, through the real probe
path. Rows for other tools are dropped: with a one-row table every other
probe is NO_TOOL, which is noise about the table, not about the copy."""
sweep = sweep_module()
rows = sweep.read_axis([("citation_hash_class.py", path)], 60)
return sweep, [r for r in rows if r["tool"] == "citation_hash_class.py"]
def leg_mutant_and_control():
sweep, control = axis_for(SUBJECT)
echo_c = [r for r in control if r["verdict"] == "ECHOES_USAGE"]
bad_c = [r for r in control if r["verdict"] in sweep.FATAL_READ]
note(not echo_c and not bad_c,
"leg 4 control: clean copy reads %d probe(s), 0 ECHOES_USAGE, "
"0 fault" % len(control))
td = tempfile.mkdtemp(prefix="q97-")
try:
mutant = os.path.join(td, "citation_hash_class.py")
shutil.copyfile(SUBJECT, mutant)
with open(mutant, encoding="utf-8") as fh:
src = fh.read()
old = "ap.add_argument('--repo', default=DEFAULT_REPO)"
new = "ap.add_argument('--tree', dest='repo', default=DEFAULT_REPO)"
if src.count(old) != 1:
note(False, "leg 3 cannot build the mutant: %r appears %d time(s)"
% (old, src.count(old)))
return
with open(mutant, "w", encoding="utf-8") as fh:
fh.write(src.replace(old, new, 1))
sweep2, rows = axis_for(mutant)
echo_m = [r for r in rows if r["verdict"] == "ECHOES_USAGE"]
named = sorted((r["probe"] for r in echo_m))
note(named == ["missing", "not-a-repo"],
"leg 3 mutant (--repo -> --tree): ECHOES_USAGE on %s" % (named,))
buf = io.StringIO()
with redirect_stdout(buf):
rc, hard, soft = sweep2.report_read(rows, quiet=True)
line = buf.getvalue().strip().splitlines()
summary = line[-1] if line else "no summary"
note(rc == 1 and (hard, soft) == (1, 1)
and "ECHOES_USAGE citation_hash_class.py/not-a-repo" in buf.getvalue(),
"leg 3b mutant axis %s" % summary)
finally:
shutil.rmtree(td, ignore_errors=True)
def leg_live_axes():
sweep = sweep_module()
live, _source, _untracked = sweep.discover(ROOT, None)
axis = sweep.read_axis(live, 60)
bad = [r for r in axis if r["verdict"] in sweep.FATAL_READ]
echo = [r for r in axis if r["verdict"] == "ECHOES_USAGE"]
note(not bad and not echo,
"leg 2 live axes: %d probe(s), 0 fault, 0 ECHOES_USAGE" % len(axis))
def main():
if not (os.path.isfile(SWEEP) and os.path.isfile(SUBJECT)):
print("cannot grade: tools/selftest_sweep.py or citation_hash_class.py "
"missing under %s" % ROOT, file=sys.stderr)
return 2
leg_selftest()
leg_live_axes()
leg_mutant_and_control()
if failures:
print("Q97 GATE: rc 1 -- %d leg(s) failed" % len(failures))
return 1
print("Q97 GATE: rc 0 -- a refusal that echoes the invocation is fatal on "
"both input axes, and a mutating flag rename is caught.")
return 0
if __name__ == "__main__":
sys.exit(main())